I want to say that with 0.18 the definition of federation_workers has changed massively due to the improved queue. As in, whatever is good in 0.17 is not necessarily good for 0.18.

On 0.18, it probably makes sense to have it around 100 to 10'000. Setting it to 0 is also be an option (unlimited, that's the default). Anything much higher is probably a bad idea.

On 0.18, retry tasks are also split into a separate queue which should improve things in general.

0 might have perf issues since every federation task is one task with the same scheduling priority as any other async task (like ui / user api requests). So if 10k federation tasks are running and 100 api requests are running then tokio will schedule the api requests with probability 100 / (10k+100) (if everything is cpu-limited). (I think, not 100% sure how tokio scheduling works)

Holy fuck 200k workers!? I'm not familiar with lemmy internals but I've literally never seen any program run anything close to well at levels that high. Want some help from someone who is a DevOps engineer by day? I think I remember you said you were a psql dba professionally so maybe my experience could help out?

Ideally, we can fix this in the software eventually (most likely it has already been improved a lot in 0.18.1 - we'll find out for sure when lemmy.world upgrades), but for now it really does seem that defederating offline servers will massively improve the success rate of federated posts and comments reaching other instances.

Admins: FYI on lemmy logs:

Example of a federation message success (HTTP Response 200):

INFO actix_web::middleware::logger: 12.34.56.78 'POST /inbox HTTP/1.1' 200 0 '-' 'Lemmy/0.17.4; +https://remote.lemmy' 0.145673

and failure (HTTP Response 400):

INFO actix_web::middleware::logger: 12.34.56.78 'POST /inbox HTTP/1.1' 400 65 '-' 'Lemmy/0.17.4; +https://remote.lemmy' 0.145673

These are usually accompanied soon with a more verbose reason like Http Signature is expired.

Lemmy is far better at logging INCOMING stuff than it is OUTGOING. You can grep for activity_queue to get a sense if there are issues. This is not good:

Target server https://lemmy.lemmy/inbox rejected https://lemmy.local/activities/announce/9852ff01-c768-484b-a38da-da021cd1333, aborting

Also there are stats indicating "pile-ups":

Activity queue stats: pending: 1, running: 1, retries: 706, dead: 0, complete: 12

It's only been a few minutes but I'm seeing non timing out federation in my nginx access log. Hopefully it keeps working.
Also at least on my instance, lemmy.ml has completely broken, I'm not getting anything from it at all anymore. it dropped out at 13:52:22 and besides a couple few messages it's been silence since then. It seems to be working on lemmy.world so I'm not sure what's causing that.

Thanks for your work and sharing results!

I think that kbin and lemmy are going to ultimately have to record per-instance response time and back off on a given instance. Like, if another instance is failing or overloaded, it's going to have to reduce the frequency with which it attempts to communicate with that instance, to avoid having a ton of workers tied up trying to communicate with that instance.

Posted this last night, but reposting for visibility:

To those experiencing federation issues with communities that aren't local, make sure to properly set your language in our profile! I thought my off-instance communities were having extremely slow federation, but the issue was I didn't have English as one of my profile languages.

So as of right now, https://lemmy.ca still seems bugged.

• https://lemmy.world/c/[email protected]
• https://lemmy.world/c/[email protected]

These two are 0 comments here on lemmy.world, while comments clearly exist over at lemmy.ca.

• https://lemmy.world/c/[email protected]

The opposite here: I made a test post at [email protected], so lemmy.world thinks there is +1 comment. But the true instance at http://lemmy.ca/c/microcontroller sees 0 comments, so my comment fails to traverse the federation to lemmy.ca.

So both imports and exports to these communities on lemmy.ca seem bugged.

From what I've seen, there is a 10-second hard-coded timeout for HTTP, seems too low for the kind of load going on. Especially if the server is opening tons of connects to the same peer server.

My feeling is that the update has improved the situation. For example, when I filter by Hot, significantly more recent posts appear than before.

EDIT: nevermind, I figured out my issue https://github.com/LemmyNet/lemmy-ui/issues/1862