We are aware of an issue with some clients not being able to sync (receiving 502 errors, etc..). It does not appear to affect all clients and the web view is still working. We will update this post as we know more and when we have resolved it.
Updates:
09-Sept 11:19 PM UTC We attempted to roll back from Lemmy v0.18.4-beta.8 to v0.18.4 (as well as even earlier beta versions) but the service failed to start. We've reverted back to v0.18.4-beta.8 for now and will continue investigating the cause of the sync issues.
10-Sept 3:30 AM UTC And... we are back, where we started. If you hadn't noticed, we were down for almost 2.5 hours while we tried to rollback the database migrations introduced in the beta releases so we could go back to the stable v0.18.4. When that failed, we tried to restore from the backup we took just prior to the rollback attempt. When that failed, we used Digital Ocean's restore to point in time and created a new DB cluster and switched to it. So... we are back on v0.18.4-beta.8 again, with the expectation that 3rd party clients are still broken. Unfortunately, we are not sure what we can do about this as we believe it is something in Lemmy that is breaking it. The mitigation we have put in place to avoid this going forward is that we no longer use the "latest" tag when loading a docker image and now hard lock to a specific version. We will update that version in a more intentional manner going forward.
10-Sept 4:15 AM UTC After some additional digging through the Lemmy diff for beta.5 to beta.8 (and then looking through the PRs that were merged to create this diff), the issue is indeed with the Lemmy code and is even called out in one of the PRs,. What this means is that the 3rd party clients that are not able to access the site will need to update their code to fix things on their side (or else they will be broken when v0.19 is released and all other instances update to it).
16-Sept 11:23 PM UTC Both Sync and Mlem have now released new versions that resolve connectivity issues. If there are any remaining clients we suggest you reach out directly to them to ask them. We'll be unpinning this post in a few hours
27-Sept 10:30 PM UTC We have made the decision to update to the RC (release candidate) for v19 as the beta we were previously on was missing some much-missed admin tools that we need to run the site with. This update appears to have affected 3rd parties again (I have tested Mlem and Memmy and get errors about authentication, however they still seem to load?). There also appears to be an issue with the home page loading that we are currently looking into how to resolve. As before, we don't anticipate any way to roll back. To steal a line from my daughter's favorite movie: Keep Moving Forward
28-Sept 12:03 AM UTC We found an issue with the markdown parsing that was causing the homepage sidebar to throw errors. We've resolved the issue and will update the core developers of Lemmy.
03-Oct 1:28 AM UTC We have implemented a workaround that should have fixed all apps. Please let us know if you are still having issues with an app (we may not be able to add a workaround for it, but we would like to at least track the ones not working)
