Lemmy Administration

I’m using the Lemmy ansible installation method. I’ve been trying to add sendgrid to the postfix section of the config.hjson file on my local machine. But where do I add the API key and username? I used port 587 but nothing works. Can anyone help walk me through how to integrate sendgrid into Lemmy-Ansible? Thanks!!

the email section of config.hjson looks like this, did I do this right?

  email: {
    smtp_server: "smtp.sendgrid.net:587"
    smtp_from_address: "noreply@{{ domain }}"
    tls_type: "tls"
  }

I was able to find the server location on my VPS under srv/lemmy/domain, so I can edit the lemmy.hjson file there if need be.

Out of the 2G of swap assigned it used to sit at ~250M. It is now being utilised close to 100%.

https://github.com/LemmyNet/lemmy/issues/2943#issuecomment-1581485335

https://github.com/LemmyNet/lemmy/pull/2055

curl -X PUT http://localhost:1236/api/v3/community/hide \   
-H "Content-Type: application/json" \
-d \
'{"community_id":3,"hidden":true,"reason":"controversal","auth":"Foo"}'

I haven't tried this, but maybe someone will find it useful and test it out. You could probably also easily do it in the database instead of using the API call.

I put the middle finger emoji under their newest piracy announcement and they banned me from Lemmy World for a week. I can argue that it was ambiguous. It could have been against Lemmy for letting pirates back in, or it could have been against the pirates. So damn quick to judge.

cross-posted from: https://lemmy.ml/post/4489142

Originally asked in #lemmy:matrix.org

1 The Idea

I've been thinking about writing a website to monitor Lemmy instances, much in the same vein as lemmy-status.org, to help people like me, who are interested in the operational health of their favourite servers, have a better understanding of patterns and be notified when things go wrong.

I thought I'd share my thoughts w/ you and ask for your feedback before going down any potential rabbit hole.

1.1 Public-facing monitoring solution external to a cluster

I don't wish to add any more complexity to a Lemmy setup. Rather I'm thinking about a solution which is totally unknown to a Lemmy server AND is publicly available.

I'm sure one could get quite a decent monitoring solution which is internal to the cluster using Prometheus+Grafana but that is not the aim of this.

1.2 A set of key endpoints

In the past there've been situations where a particular server's web UI would be a 404 or 503 while the mobile clients kept happily working.

I'd like to query a server for the following major functionalities (and the RTT rate):

• web/mobile home feed
• web/mobile create post/comment
• web/mobile search

1.3 Presenting stats visually via graphs

I'd like to be able to look at the results in a visual way, preferably as graphs.

1.4 History

I think it'd be quite cool (and helpful?) to retain the history of monitoring data for a certain period of time to be able to do some basic meaningful query over the rates.

1.5 Notification

I'd like to be able to receive some sort of a notification when my favourite instance becomes slow or becomes unavailable and when it comes back online or goes back to "normal."

2 Questions

❓ Are you folks aware if someone has already done something similar?

❓ I'm not very familiar w/ Rust (I wrote only a couple of small toy projects w/ it.) Where can I find a list of API endpoints a Lemmy server publicly exposes?

❓ If there's no such list, which endpoints do you think would work in my case?

I'm looking into migrating (part of) my ever-growing pictrs data to something cheaper than the VPS disk it's currently on. Ideally I'd like to use minio's Object Tiering to migrate object to/from cheaper storage.

Anybody using Backblaze's cloud storage product? What other options are out there and what potential pitfalls should I be aware of?

Woke up in the morning and my selfhosted Lemmy server was basically braindead. I installed it in a Proxmox lxct container using the ansible playbook. It spams my logs with this:

target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=8206186d-eaf9-486d-99ad-d9c5def188c8", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.001260Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::community::announce::receive
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:46
lemmyohaaxyz-lemmy-1 |    1: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=dee66940-4048-4424-88c8-51cb58851eb0
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=dee66940-4048-4424-88c8-51cb58851eb0", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:08.442900Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum AnnouncableActivities
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::community::announce::receive
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:46
lemmyohaaxyz-lemmy-1 |    1: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum AnnouncableActivities, context: SpanTrace [{ target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 46 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ba977ad6-03ea-46b3-a7d8-c0eb05355358", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:10.626474Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::comment::verify
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/comment.rs:127
lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::fetcher::post_or_comment::verify
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/fetcher/post_or_comment.rs:68
lemmyohaaxyz-lemmy-1 |    2: lemmy_apub::activities::voting::vote::verify
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
lemmyohaaxyz-lemmy-1 |    3: lemmy_apub::activities::community::announce::receive
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
lemmyohaaxyz-lemmy-1 |    4: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::objects::comment", name: "verify", file: "crates/apub/src/objects/comment.rs", line: 127 }, { target: "lemmy_apub::fetcher::post_or_comment", name: "verify", file: "crates/apub/src/fetcher/post_or_comment.rs", line: 68 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0aeb2c01-23ac-492c-9693-3a9bb171a900", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:16:14.796561Z  WARN lemmy_server::root_span_builder: data did not match any variant of untagged enum PageOrNote
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::activities::voting::vote::verify
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::activities::community::announce::receive
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
lemmyohaaxyz-lemmy-1 |    2: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=552c4fb1-c65c-4220-8919-430243d720cd
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: data did not match any variant of untagged enum PageOrNote, context: SpanTrace [{ target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=552c4fb1-c65c-4220-8919-430243d720cd", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:27:00.779726Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:30:09.026861Z  WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::person::read_from_id
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/person.rs:66
lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::fetcher::user_or_community::read_from_id
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/fetcher/user_or_community.rs:47
lemmyohaaxyz-lemmy-1 |    2: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "read_from_id", file: "crates/apub/src/objects/person.rs", line: 66 }, { target: "lemmy_apub::fetcher::user_or_community", name: "read_from_id", file: "crates/apub/src/fetcher/user_or_community.rs", line: 47 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=0a4c7741-ce73-4108-a323-e7f5cc5b92ae", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:35:46.568991Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:37.481187Z  WARN activitypub_federation::activity_queue: Queueing activity https://lemmy.ohaa.xyz/activities/announce/df1ccbd5-3971-49e1-96b4-df6e73f39420 to https://pcglinks.com/inbox for retry after connection failure: Request error: error sending request for url (https://pcglinks.com/inbox): operation timed out.  Sleeping for 216000s and trying again
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:42:51.370663Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:46:36.393798Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T05:53:44.661098Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for hot ranks update: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T06:00:21.684837Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for captcha cleanup: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T06:10:18.172432Z  WARN lemmy_server::root_span_builder: Timeout occurred while waiting for a slot to become available
lemmyohaaxyz-lemmy-1 |    0: lemmy_apub::objects::person::from_json
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/objects/person.rs:134
lemmyohaaxyz-lemmy-1 |    1: lemmy_apub::activities::verify_person_in_community
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/mod.rs:62
lemmyohaaxyz-lemmy-1 |    2: lemmy_apub::activities::voting::vote::verify
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/voting/vote.rs:57
lemmyohaaxyz-lemmy-1 |    3: lemmy_apub::activities::community::announce::receive
lemmyohaaxyz-lemmy-1 |              at crates/apub/src/activities/community/announce.rs:144
lemmyohaaxyz-lemmy-1 |    4: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_apub::objects::person", name: "from_json", file: "crates/apub/src/objects/person.rs", line: 134 }, { target: "lemmy_apub::activities", name: "verify_person_in_community", file: "crates/apub/src/activities/mod.rs", line: 62 }, { target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 144 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=b7159d57-eaae-4992-8d37-0ab87f9f3adb", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | 2023-08-22T06:16:50.839271Z ERROR lemmy_server::scheduled_tasks: Failed to establish db connection for active counts update: could not translate host name "postgres" to address: Try again
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | 2023-08-22T06:26:51.471879Z  WARN lemmy_server::root_span_builder: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | Caused by:
lemmyohaaxyz-lemmy-1 |     Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
lemmyohaaxyz-lemmy-1 |    0: lemmy_server::root_span_builder::HTTP request
lemmyohaaxyz-lemmy-1 |            with http.method=POST http.scheme="https" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind="server" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770
lemmyohaaxyz-lemmy-1 |              at src/root_span_builder.rs:16
lemmyohaaxyz-lemmy-1 | LemmyError { message: None, inner: Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT
lemmyohaaxyz-lemmy-1 | 
lemmyohaaxyz-lemmy-1 | Caused by:
lemmyohaaxyz-lemmy-1 |     Http Signature is expired, checked Date header, checked at Tue, 22 Aug 2023 06:26:51 GMT, expired at Tue, 22 Aug 2023 06:16:26 GMT, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=57b8b802-fc7e-4f14-87ac-89de2b0d7770", file: "src/root_span_builder.rs", line: 16 }] }
lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 5' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=ab4b5ccf-f8b9-4acd-9e4b-3c7966a17fd5", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8
lemmyohaaxyz-lemmy-1 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
lemmyohaaxyz-lemmy-1 | thread 'actix-server worker 3' panicked at 'read local site data: LemmyError { message: None, inner: Timeout occurred while waiting for a slot to become available, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.ohaa.xyz http.target=/inbox otel.kind=\"server\" request_id=4e4ba6d1-d716-4d48-93de-e3d7772243d7", file: "src/root_span_builder.rs", line: 16 }] }', crates/apub/src/lib.rs:45:8
lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command
lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz
lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536
lemmyohaaxyz-lemmy-1 | connection error: db error: FATAL: terminating connection due to administrator command
lemmyohaaxyz-lemmy-1 | federation enabled, host is lemmy.ohaa.xyz
lemmyohaaxyz-lemmy-1 | Starting http server at 0.0.0.0:8536

This post is intended as a central place that admins can reference regarding the XSS incident from this morning.

What happened?

A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.

Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.

Am I vulnerable?

If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.

I had custom emojis on my instance, what should I do?

This should be enough to mitigate now:

1. Remove custom emoji

DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;

2. Rotate your JWT secret (invalidates all current login sessions)

-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();

3. Restart Lemmy server

If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)

Legal

If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402

More context:

https://github.com/LemmyNet/lemmy-ui/issues/1895

https://github.com/LemmyNet/lemmy-ui/pull/1897