Sounds like your Postgres instance is offline. Any logs from that container?
this post was submitted on 18 Jun 2023
10 points (100.0% liked)
Selfhosted
40113 readers
974 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
The postgres container appears to be running.
sudo docker ps output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e305af2b91c8 dessalines/lemmy-ui:0.17.4 "docker-entrypoint.s…" 27 minutes ago Up 27 minutes 127.0.0.1:12074->1234/tcp lemmyname_lemmy-ui_1
3978157ef390 dessalines/lemmy:0.17.4 "/app/lemmy" 27 minutes ago Restarting (101) 34 seconds ago lemmyname_lemmy_1
626d7792534b asonix/pictrs:0.3.1 "/sbin/tini -- /usr/…" 27 minutes ago Up 27 minutes 6669/tcp, 127.0.0.1:21166->8080/tcp lemmyname_pictrs_1
65eab82adee9 mwader/postfix-relay "/root/run" 27 minutes ago Up 27 minutes 25/tcp lemmyname_postfix_1
2ab8c789fdcd postgres:15-alpine "docker-entrypoint.s…" 27 minutes ago Up 27 minutes 5432/tcp lemmyname_postgres_1
Container logs: ...
PostgreSQL init process complete; ready for start up.
2023-06-18 01:57:16.112 UTC [1] LOG: starting PostgreSQL 15.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
2023-06-18 01:57:16.112 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-06-18 01:57:16.112 UTC [1] LOG: listening on IPv6 address "::", port 5432
2023-06-18 01:57:16.113 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-18 01:57:16.117 UTC [53] LOG: database system was shut down at 2023-06-18 01:57:16 UTC
2023-06-18 01:57:16.123 UTC [1] LOG: database system is ready to accept connections
2023-06-18 02:02:16.215 UTC [51] LOG: checkpoint starting: time
2023-06-18 02:02:20.329 UTC [51] LOG: checkpoint complete: wrote 44 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.110 s, sync=0.003 s, total=4.115 s; sync files=12, longest=0.002 s, average=0.001 s; distance=252 kB, estimate=252 kB
Can you log into it using your password? If so, then the network between the two isn’t working properly and you’ll need to check your Docker network config.
Yes, I can log into the psql shell on the container and connect to the database with no issue. I guess it is a network issue, I'll look into it further tomorrow
I just installed using a different OS (Debian) and it works. Not sure why, but I'll take it. Posting from my new instance!
Is there a postgresql docker container running?
Yes, I posted the docker status info and logs from the container in my reply to Veraticus
suggest you check postgres is running, accepting connections and returning results on the container first:
docker exec -it lemmyname_postgres_1 /bin/bash
assuming you can connect to the docker container and see a prompt, does this command put you at a sql prompt?
psql -U lemmy -d lemmy
if so, let's confirm that a sql command works.
select name from person; (will likely be an empty set if you haven't set up an admin user yet).
Does all this work OK?
docker exec -it lemmyname_postgres_1 /bin/bash
returns the bash prompt as expected: 2ab8c789fdcd:/#
psql -U lemmy -d lemmy successfully opens psql:
psql (15.3)
Type "help" for help.
lemmy=#
ran select name from person; from the psql shell, it does look empty (haven't created any users yet):
ERROR: relation "person" does not exist
LINE 1: select name from person;
it should return an empty set. It looks like the database is running but the lemmy database tables were not installed. Can you get back to the sql prompt and try
\dt
if the database was installed correctly, you should see something like this:
lemmy-# \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------------+-------+-------
public | __diesel_schema_migrations | table | lemmy
public | activity | table | lemmy . .
. .
public | person | table | lemmy
. . .
can you check that the tables are actually there? My guess is that the tables in the database weren't created. assuming you don't see any tables when you do this, again at the sql prompt, please try
\c lemmy
and see if it connects? if it does, it will say You are now connected to database "lemmy" as user "lemmy".
Let me know what the result of that is and we'll try to figure out what's gone wrong with the database setup because it looks like that's your problem - the database was not populated when it was created.
\c lemmy does return "You are now connected to database "lemmy" as user "lemmy""
Running \dt after connecting returns "Did not find any relations", so it looks like you were right and the database didn't get populated.
ok thanks - I'm not very familiar with the lemmy installation process so I will need to check if possibly the schema objects in the database aren't created until the admin user is first set up, which is another possibility, but they definitely aren't there right now - I'll set up a fresh install to check in a few minutes.
Switching gears a bit - when you say you can't access the web UI, can you elaborate on that? The web server should be running. You are going to https://your.url, presumably and then what do you actually see in the browser? It should show an initial setup screen - do you not see anything or is there some kind of error message? When the ansible script runs to create the instance does it throw any errors at all or everything looks happy in the output, including creation of your ssl certs?
No errors of any kind in the ansible script. nginx is running, and it's at https://lemmy.name - I get some gateway timeout errors once in a while, but that's all. When I do get error pages, they are served via https at least, so I don't think there's anything wrong with that part.
I assume this is because the lemmy backend container is continually restarting (as I detailed more in my original post and other replies here) due to database connection errors
Ok so I ran a clean install and confirmed that the database objects exist. So that's definitely the problem. Your database is munted. An interesting thing happened when I re-ran the playbook - I noticed it didn't rebuild the postgres container. Very suspicious! Let's try blowing it away completely and re-running your script:
docker stop lemmyname_postgres_1
docker rm lemmyname_postgres_1
now re run the ansible playbook and let's see if that works.
Ran those commands successfully, then the playbook again. Playbook has zero errors or warnings.
The postgres container was remade.. with the same broken database. Went into it with psql like before, did \c lemmy, then \dt - same error - did not find any relations.
weird. it's like it failed to install the postgres public schema properly. I am not at all familiar with ansible but I see you can set verbosity. Do you think it would be worth trying that?
According to link below you can preface your playbook command with ANSIBLE_DEBUG=true ANSIBLE_VERBOSITY=4
https://www.shellhacks.com/ansible-enable-debug-increase-verbosity/
This alternative installer might be a way around this issue: https://github.com/ubergeek77/Lemmy-Easy-Deploy
Wiped the VPS clean, new 22.04 server install.
Deployed using that installer.
Exact same issue. The lemmy container cannot reach the database and continually restarts. I'll post some logs, but they're practically identical to what I posted before:
lemmy container logs:
`thread 'main' panicked at 'Error connecting to postgres://lemmy:PasswordRedacted@postgres:5432/lemmy', crates/db_schema/src/utils.rs:161:56
sudo docker ps output:
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
9eaa62ea6f99 caddy:latest "caddy run --config …" 12 minutes ago Up 12 minutes
0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 443/udp, 2019/tcp lemmy-easy-deploy-proxy-1
a10009ed128d dessalines/lemmy-ui:0.17.4 "docker-entrypoint.s…" 12 minutes ago Up 12 minutes
lemmy-easy-deploy-lemmy-ui-1
a166b1486d51 dessalines/lemmy:0.17.4 "/app/lemmy" 12 minutes ago Restarting (101) 41 seconds ago lemmy-easy-deploy-lemmy-1
08329653260b asonix/pictrs:0.3.1 "/sbin/tini -- /usr/…" 12 minutes ago Up 12 minutes
6669/tcp, 8080/tcp lemmy-easy-deploy-pictrs-1
51af8f312511 postgres:15-alpine "docker-entrypoint.s…" 12 minutes ago Up 12 minutes
lemmy-easy-deploy-postgres-1
well, that rules out your local system and ansible, at least. It seems like there's an inability for the docker containers to reach postgres definitely at the install and probably at runtime - can we check this?
first let's make sure the postgres container is on the network and has an address:
sudo docker exec -it <yourinstance>_postgres_1 /bin/sh
once you're on the container
ifconfig
which should return on the second line: inet addr:172.18.0.4 (or similar ipV4 address)
exit to return to the host and let's test connecting from the host :
nc -zv 172.18.0.4 5432 (use IP from the DB container)
Ubuntu host should respond something like : Connection to 172.18.0.4 5432 port [tcp/postgresql] succeeded!
now let's try the containers
sudo docker exec -it <yourinstance>_lemmy-ui_1 /bin/sh
now from that container test connectivity to postgres:
nc -zv 172.18.0.4 5432 (use IP from the DB container)
now the other
sudo docker exec -it <yourinstance>_lemmy_1 /bin/sh
on the docker containers the nc command should return something like
172.18.0.4 (172.18.0.4:5432) open
is that working?