About Lemmit

177 readers
1 users here now

About the lemmit.online service and its software.


Got questions, complaints, suggestions? This is the place.

founded 1 year ago
MODERATORS
1
10
Community cleanup (lemmit.online)
submitted 10 months ago* (last edited 10 months ago) by [email protected] to c/[email protected]
 
 

Since its inception, the Lemmit instance has been controversial. That might be an understatement, but let's roll with it for now. One of the major issues people have with the bot, is the cross-posting of "interactive" Reddit posts, ie posts where the value lies in interacting with the OP, like AskMen, AskWomen, and AmITheAsshole. Personally, I fully agree with that viewpoint, but I didn't feel like interfering with supply and demand - in the sense that AmITheAsshole, for some reason, is the most subscribed community on this server.

That might change though. Earlier this week, I disabled the posibility to request new subreddits. This weekend I will follow that up by disabling the scraping of so-called interactive communities. So in order to facilitate that, I created a list of all the communities on this server (posted separately in [email protected]), and I will check each of them to see if they should be disabled. The goal is to keep a list of "content only" (or at least "content mostly") communities, where the value lies in the link that's provided or in the body of the self-post - not in the comment section. I'm sure this is going to be a disappointment to some people, but I do agree with the sentiment that this is better for Lemmy as a whole.

Edit: It is done. All 816 communities have been checked and and 110 of those have been purged from updates. I am sure some mistakes were made - that some communities have been disabled or left intact when they shouldn't have. If that's the case, reach out to me, and I'll fix it.

2
15
List of communities. (lemmit.online)
submitted 10 months ago* (last edited 10 months ago) by [email protected] to c/[email protected]
 
 

Since it's impossible to see all the communities on here without logging in (mandatory NSFW filter), and I'm the only one with an account on here, here's a list of them. This list will be a snapshot, so the subscriber count will not be up to date, but I'm sure you'll figure it out.

Ident NSFW Subscribers
1000ccplus NSFW 14
2000ccplus NSFW 8
2007scape 13
2137 41
2meirl4meirl 29
2westerneurope4u 254
3dprinting 90
80sdesign 64
90sdesign 54
AskReddit 222
Boise 67
Erotica NSFW 153
IdiotsInCars 140
InternetIsBeautiful 73
MuseumOfReddit 71
PerfectlyCutMeows 54
ProgrammerHumor 275
SFWdeepfakes 57
Superstonk 295
Ultralight 85
abandonedporn 242
aboringdystopia 56
acmilan 26
adviceanimals 124
afterplayio 20
akaimpc 29
aleague 34
algarve 34
alternateangles 59
altgirls NSFW 362
amateurroomporn 11
amazonposition NSFW 62
anal NSFW 344
anal_yiff NSFW 50
analog 18
analogcommunity 20
androiddev 43
ani_bm 38
animalsbeingderps 15
anime 157
animemes 127
anormaldayinrussia 63
antimeme 15
antiwork 153
apksapps 45
apple 58
applevisionpro 35
appnotes 31
arizona 40
arizonatrail 51
armenia 8
arsenalwfc 30
artificial 10
artisanvideos 34
artsakh 7
asiansgonewild30plus NSFW 56
asiantraps NSFW 90
assettocorsa 8
atbge 6
audioproductiondeals 26
australia 40
autism 12
ayaneo 47
backyardfarmers 5
badscificovers 30
banano 33
bangmybully NSFW 32
bapcsalescanada 43
baseball 75
battlebots 54
battlestations 153
bbw NSFW 132
bbw_banging NSFW 18
bbwbutthole NSFW 50
bbwgw NSFW 33
bbwhardcore NSFW 75
bbwmilf NSFW 184
bbwtits NSFW 57
bdsm NSFW 70
bdsm_smiles NSFW 36
beamng 47
beautifulfemales 120
beefcurtains NSFW 70
beer 43
bestoflegaladvice 109
bestofredditorupdates 151
bicycletouring 42
bigareolalover NSFW 45
bigboobsgonewild NSFW 17
bigdickgirl NSFW 47
biggerbabes NSFW 34
bigtittygothgf NSFW 50
bikeporn 42
biketouring 43
biology 29
bioniclelego 22
blockedandreported 29
blunderyears 68
blursedimages 29
bmw 31
bobstavern 4
bois NSFW 129
bonnebouffe 9
boxing 38
brasil 47
brownchickswhitedicks NSFW 142
buildapcsales 156
burdurland 3
bustynaturals NSFW 275
buttsandbarefeet NSFW 198
cantonese 27
carpentry 5
cars 48
castiron 34
casualpt 39
celebhub 40
celebnsfw NSFW 199
celebnudedebut NSFW 55
celebritylegs 15
celebs 109
celebswithpetitetits 215
cfb 50
cfs 47
chastitycouples NSFW 81
chickflixxx NSFW 73
chloe 46
chubby NSFW 78
chubby_hentai NSFW 55
cirkeltrek 50
citiesskylines 122
citypop 53
codzombies 29
collapse 29
combatfootage 94
comedycemetery 7
comedyheaven 7
comedyhomicide 11
comedynecromancy 7
comedynecrophilia 10
comics 162
comicstriphistory 56
comicwalls 42
completeanarchy 4
condomtobarebackmfmt NSFW 34
coolguides 144
copypasta NSFW 17
corgigifs 7
coronavirus 51
couplesgonewildplus NSFW 33
cozyplaces 5
crackwatch 57
crboxes 34
creampie NSFW 46
cremposting 7
crossstitch 58
cruelcaptions NSFW 18
cryptocurrency 81
cuckold NSFW 158
cuckoldpregnancy NSFW 57
cuckoldstories2 NSFW 26
cumcoveredfucking NSFW 99
cumdumpsters NSFW 184
cumshotgifs NSFW 201
cumsluts NSFW 629
curatedtumblr 31
cursedcomments 65
daintywilderfans NSFW 41
dallas 48
dankmemes 28
daresgonewild NSFW 76
datahoarder 76
dataisbeautiful 364
dccomics 66
deadmau5 18
decor 3
deeprockgalactic 74
depthhub 33
destiny2 71
deusex 39
dewalt 8
dinosaure 5
discordvideos 8
diwhy 90
diwhynot 37
documentaries 69
dota2 73
dross NSFW 7
drugscirclejerk 59
edmonton 18
electricians 5
emogirlsfuck NSFW 402
enculerlesvoitures 10
engineeringporn 172
engorgedveinybreasts NSFW 127
entertainment 51
enterthegungeon 3
epicgamespc 55
erotichypnosis NSFW 66
eroticliterature NSFW 64
eu4 8
eu_nvr 23
everything_gripe 31
exmormon 58
extrafabulouscomics 24
extramile NSFW 213
eyebleach 90
facepalm 37
factorio 87
fairytaleasfuck 11
fakealbumcovers 36
fatwomenlove NSFW 42
fedora 91
feedthebeast 94
femboy 39
femboymemes 20
femboys NSFW 149
femboys4real NSFW 30
feral_yiff NSFW 41
feralpokeporn NSFW 49
fighters 6
fixedgearbicycle 52
flagporn 5
flatchests NSFW 88
food 124
forbiddensnacks 11
forgottenbookmarks 36
formula1 30
formuladank 133
fortcollins 39
fortyfivefiftyfive NSFW 247
fosscad 70
fossdroid 137
foundryvtt 32
framework 17
freegamefindings 118
frogbutt NSFW 115
functionalprint 103
funny 342
furry_irl 165
furryhits 30
futadomworld NSFW 7
fuzzypeeks NSFW 28
gadgets 53
gamedeals 212
gamedealsfree 82
games 164
gamingleaksandrumours 44
gardening 10
garfieldminusgarfield 42
gay_irl 10
gaytwerking NSFW 16
generative 31
genshin_impact 53
gentlemanboners 293
geocaching 37
germanshepherds 28
gfur NSFW 98
gfurcomics NSFW 16
ghosts 30
ghoststories 28
ginger NSFW 156
girlsfarting NSFW 8
girlsfinishingthejob NSFW 553
girlsjoy NSFW 105
girlsjustwanttobefuck NSFW 121
girlsmasturbating NSFW 239
girlswholovetobefuck NSFW 178
giscardpunk 9
globaloffensive 53
gme_meltdown 6
gmecanada 39
gnome 5
gocommitdie 7
godot 13
godpussy NSFW 401
gonemildplus NSFW 36
gonewild NSFW 508
gonewild30plus NSFW 424
gonewildaudio NSFW 254
gonewildchubby NSFW 89
gonewildcolor NSFW 40
gonewildhairy NSFW 29
gonewildplus NSFW 36
gonewildstories NSFW 278
goodanimemes 127
googleplaydeals 119
gooned NSFW 168
gothstyle 3
granturismo 33
gravelcycling 47
greenhouses 6
grimdank 70
grime 8
grool NSFW 120
guildwars 10
guildwars2 65
guiltygear 49
gunners 50
hackernews 18
hardware 53
haunted 29
havanese 31
hayastan 7
hearthstone 41
hentai NSFW 31
hentaihumiliation NSFW 75
hfy 58
highstrangeness 86
hmmm 5
hobbydrama 18
hockey 63
holdmybeer 7
hololive 57
holup 34
homeassistant 430
homedecorating 4
homelab 146
homelabsales 23
homestead 7
homesteading 8
honkaistarrail 22
horror 73
hotwifetexts NSFW 38
hoyas 40
humiliationcaptions NSFW 34
humongousaurustits NSFW 23
hyruleengineering 67
icecreamery 42
ich_iel 5
idm 5
imaginarybestof 77
imaginarywarhammer 72
impregnation NSFW 18
imsorryjon 26
imthemaincharacter 88
inceltear 48
incest_captions NSFW 104
incestsexstories NSFW 75
incorgnito 7
india 46
indonesia 6
interestingasfuck NSFW 40
interiordesign 4
iphone 68
itookapicture 314
jailbreak 38
japanese_adult_video NSFW 57
japantravel 22
jav NSFW 131
java 19
javdreams NSFW 49
jerkofftoceleb NSFW 58
karengillan 3
keep_track 30
kgbtr 35
kiernanshipka 3
kinkycaptions NSFW 23
koreannsfw NSFW 56
kpopfap NSFW 49
labiadangling NSFW 59
ladyladyboners 119
lasercutting 41
latvia 3
learncantonese 28
learnjapanese 17
legalcatadvice 32
leopardsatemyface 136
lesbianpov NSFW 69
lesbians NSFW 140
lifehacks 53
lifeprotips 138
link_dies 33
linustechtips 118
linux_gaming 225
liverpoolfc 32
livestreamfail 80
loveforboozecruisers 39
luxembourg 42
machinelearning 8
machineporn 40
mademesmile 93
magictcg 5
malelivingspace 60
maliciouscompliance 130
manga 79
mapporn 98
marchagainstnazis 87
margotrobbie 6
mathgifs 33
maybemaybemaybe 21
mazda3 36
mcmansionhell 54
me_irl 17
mealtimevideos 13
meirl 21
meme 14
memes 52
merdasdoolx 37
microsoft 43
middleeasternhotties NSFW 38
midjourney 80
mildlyinteresting 155
minecraft 5
misterfpga 7
mmgirls NSFW 81
modcoord 80
mommymilkersnsfw NSFW 152
monero 40
monstermusume 32
mortalkombat 52
mousereview 4
movieposterporn 63
movies 110
mpcusers 27
mre 32
musicthemetime 36
mylittlepony 7
mylittleredacted 10
nadinejansen NSFW 4
nanocurrency 42
natalee NSFW 11
nativeamericangirls2 NSFW 26
nba 57
neovim 14
newcastleupontyne 33
newzealand 83
nexdock 46
nextcloud 40
nfa 63
nfl 75
nicechips 34
nightofthefullmoon 25
nirvannatheband 6
nixos 42
noncredibledefense NSFW 10
nordictrackandroid 46
northkoreapics 27
nosleep 187
nostalgia 16
notkenm 36
nsfw_caption NSFW 105
nsfw_gif NSFW 308
nsfw_japan NSFW 85
nsfwcelebs NSFW 210
nsfwcosplay NSFW 148
nsfwcyoa NSFW 8
nudecelebsonly NSFW 102
nukedmemes 8
nyc 40
offgrid 11
offgridcabins 6
oilporn NSFW 106
okbuddyhololive 34
okbuddyphd 13
okbuddyretard 90
okdraudzindauni 3
onebag 14
onepiece 43
onepunchman 110
onguardforthee NSFW 53
onmww NSFW 63
onoffcelebs NSFW 84
opensource 77
opensourceapps 43
orgasms NSFW 116
osr 11
outmanga 10
owlhousemystery 5
paradoxplaza 5
paranormal 40
paranormalencounters 18
pastaemportugues 19
pathofexile 15
paymoneywubby 61
pcmasterrace 475
peloton 76
permaculture 9
perth 55
philadelphia 41
pillowtalkaudio NSFW 18
piracy 101
piracyarchive 42
plotterart 20
plumbing 5
plussizedhotwives2 NSFW 41
polandball 6
politicalcompassmemes 23
politicalhumor 55
polska 44
portugal 73
portugalcaralho 61
preggohentai NSFW 59
presscumference NSFW 67
prettygirls 144
prettygirlsuglyfaces 37
programminglanguages 43
projectceleste 36
pronebone NSFW 24
publicfreakout 106
pussywallet NSFW 56
quebec 59
quilting 46
raining 44
rance 13
randonneuring 23
rareinsults 36
rarepuppers 21
realcivilengineer 41
redlettermedia 46
redneckengineering 83
retroussetits NSFW 146
rg35xx 23
riae_ NSFW 10
riaesuicide NSFW 15
rickandmorty 17
ringfitadventure 2
roms 50
roomporn 5
rpclipsgta 44
rpi 39
rule34 NSFW 331
running 8
rust 36
sailing 5
salmacian 5
samsungdex 51
save3rdpartyapps 6
sbcgaming 24
scams NSFW 8
science 107
sdforall 36
sdnsfw NSFW 286
selfhosted 157
seltinsweety NSFW 48
sffpc 61
sfwredheads 92
sharktits NSFW 24
shecame NSFW 47
shefuckshim NSFW 382
shegothands 48
shibbysays NSFW 47
shitposting 16
shorthairchicks NSFW 137
shorthairedwaifus 52
simpsonsshitposting 110
singularity 140
sissyinspiration NSFW 54
sissyperfection NSFW 79
skamtebord 8
skiing 28
slimthick NSFW 55
slink NSFW 16
slutoon NSFW 31
sluttyconfessions NSFW 92
smallboobs NSFW 208
soccer 64
solardiy 5
solaropposites 12
solidworks 23
sonicporn NSFW 20
sonicthehedgehog 65
space 101
spanking NSFW 21
specializedtools 61
speedrun 61
spicykittens 29
squaredcircle 96
stablediffusion 95
starcraft 49
stationeers 37
steamdeals 192
steamdeck 266
stevenuniverse 54
stihl 5
stolendogbeds 24
strapon NSFW 144
strava 37
stuffers NSFW 8
submechanophobia 54
suctiondildos NSFW 53
surface 39
sweatypalms 12
sweden 16
swiftui 13
swtor 51
taboocaptions NSFW 60
talesfromretail 34
talesfromtechsupport 53
tasker 54
tech 74
technicallythetruth 70
technology 163
techsupportgore 16
telegrambots 38
television 148
tentai NSFW 106
teslamotors 46
tf_irl 45
thalassophobia 49
thanksihateit 8
thedeprogram 95
thegrandtour 34
thenetherlands 57
theowlhouse 7
therewasanattempt 95
therian 42
thick NSFW 110
thisismylifemeow NSFW 15
thisismylifenow 38
thomastheplankengine 43
threesome NSFW 192
throatpies NSFW 38
tifu 412
tihi 8
tiktokchallenge NSFW 25
tiktokthots NSFW 164
tili 2
tinyawoos 7
titanfall 32
todayilearned 786
transformation NSFW 51
transmedical 35
traps NSFW 148
truefmk NSFW 31
truescarystories 25
truespotify 15
truscum 52
tucson 56
turkeyjerky 5
tvplus 38
twinks NSFW 135
twisthearthstone 4
u_icky_peach NSFW 10
ufo 42
ufos 130
ukraine 131
ukrainewarvideoreport 40
ukrainianconflict 203
unbgbbiivchidctiicbg 117
undertoys NSFW 25
unexpected 76
unfilteredcaptions NSFW 19
unixporn 74
unixsocks 40
upliftingnews 107
vegetablegardening 34
videos 35
vim 7
visionpro 43
vore NSFW 6
vxjunkies 23
vyos 26
wagnervsrussia 16
wallstreetbets 8
walmart 12
washingtondc 38
watchitfortheplot NSFW 120
weightlifting 47
weightroom 23
wetpussys NSFW 512
wholesomeyuri 86
whoopsgoesthecondom NSFW 24
widaczabory 7
wildhearthstone 3
windsynth 27
woodworking 13
wordsonanimegirls 7
worldnews 593
wow 51
wowservers 32
wroclaw 28
wrx 45
wtf 51
xsome NSFW 210
yiff NSFW 159
yiffbondage NSFW 32
yiffcomics NSFW 42
youseeingthisshit 44
youtubedrama 36
yuri NSFW 48
yuri_jp 42
zeldass NSFW 67
zillowgonewild 27
ziplyfiber 25
3
18
submitted 1 year ago* (last edited 10 months ago) by [email protected] to c/[email protected]
 
 

In the short time since this instance and bot launched, I've been seeing the same questions resurface multiple times. This is totally understandable, since the concept of a Fediverse is still new to most (myself included), and this server is not like the others.

Q: What is Lemmit?

A: Lemmit is a Lemmy instance specifically designed for archiving Reddit content. Users can request new subreddits to be included in the archiving process by posting in the [email protected] community. It is powered by an open source python bot, which periodically checks the request list, adds new requests to the queue, and continuously monitors the Hot feed of those subs for new posts to cross-post here.

Q: Does it synchronize comments?

A: No, that would be impossible. Considering there are thousands of posts already on Lemmit, many of them having at least several hundred comments on Reddit, often buried in deep layers, it simply wouldn't be feasible to index those for more than a few posts, let alone keep them up to date.

Unfortunately, this means that archiving certain subreddits, such as Ask Historians/Men/Women/Hyperintelligentshadesofthecolourblue-type subs, is going to be rather pointless.

Q: Can it send comments back to Reddit?

A: No, it cannot. The purpose is to help bootstrap the Lemmy platform, not to serve as a bridge between the two networks. Also, see the answer about synchronizing comments.

Q: Can I request any subreddit?

A: ~~Technically, yes. However, as the list of subs grows, the time it takes to update all of them will also increase. I do not have strict guidelines in place for this, so I'm relying on your common sense (hoooo boy). At some point, I will probably have to either stop accepting new requests or disable scraping for very low-traffic communities.~~ Nope. Automatic requests for new subreddits are closed. You can send me a PM if you think a specific subreddit would be a valuable addition to the Lemmyverse, but you'll have to make a strong case.

Q: Does this use the API? Will it keep working after July 1st?

A: Nope, it uses a combination of the public feed and scraping old.reddit.com. So, as long as those are still available, it will continue working. And even if they close those sources, there will probably be new ways to achieve the same effect. "Content, eh, finds a way."

Q: This is spam, can you stop?

A: First of all, I apologise for the inconvenience. All you have to do is block @[email protected], and none of its posts will ever show up on your instance. If you you don't want anyone else on your server to be exposed to this bot/instance, you should convince your admin to defederate from lemmit.online. Since there are no other users on here, there will be no harm done.

Obviously I could stop, because running this server and software is only ever going to cost me time and money. But for the reasons listed above, I still think this server is a useful addition to the lemmyverse at this time. But I'm looking forward to the day where I can turn the bot off because it's no longer needed.

Q: What started this?

A: Okay, nobody asked this, but I'm going to tell you anyway. After Reddit made it clear that they are effectively killing third-party apps and implementing plenty of other anti-end user decisions, I realized that I would either have to accept not being able to access my time-wasting content or have to do so in a rather uncomfortable way (either through the official app or old.reddit.com for as long as they'll allow it to exist).

Being a stubborn developer, naturally, I chose option C: Have my own Reddit. With blackjack, and hookers. This way, I would still be able to access my beloved content without being beholden to Reddit's mood swings and abusive relationship tendencies.

Besides that, I also know that Content is King. So I'm order to counter the network effect (No users because no content, No content because no users), I figured it would be better to have some inorganic content to bootstrap the adoption of Lemmy.

Q: Are NSFW subreddits allowed?

A: Absolutely. Like I said: Blackjack and hookers.

Q: My request isn't picked up by the bot!

A: That isn't a question. ~~But yeah, the process isn't flawless yet. I'm trying to iron out all the bugs as I encounter them. In the meantime, feel free to re-request the subreddit by making a second post. No harm done.~~ New subreddit requests are closed for the time being.

Q: No new posts are showing up at all on Lemmit

A: If no posts are appearing on the Lemmit Frontpage (sorted by NEW), it's possible that the bot has crashed or is stuck on something. Since no software is flawless, this sometimes happens. I usually fix this as soon as I'm aware, and I'm happy to say that these kinds of fatal errors are becoming less and less frequent. However, they may still occur, and as a human with needs of sleep and other responsibilities, I'm not always able to fix them immediately.

Q: Posts aren't showing up on my instance, what's up?

A: Due to the spammy nature of the bot, some server admins choose to block this server, and that is completely understandable. So first of all, make sure to check the instances link in the footer of your home server. If Lemmit is the Blocked Instances list, you're out of luck.

When you have verified that Lemmit is not blocked on your instance, try unsubscribing, waiting a little, and then re-subscribing. That tends to fix things.

4
9
Migration complete (lemmit.online)
submitted 9 months ago* (last edited 9 months ago) by [email protected] to c/[email protected]
 
 

The good news: The migration is complete, and I've even managed to update the version to 0.18.5 (was stuck on 0.18.4-beta8 for the longest time).

~~The sad news: Cloudflare is having some issues, so nobody is able to access the new server at this time. Oh well. It'll probably be fixed Saturday morning, and I'll turn the bot back on.~~

Migration complete, and the bot has caught up on the 24 hour gap that it was offline. It only took like 12 hours this time, while in the past it was closer to taking an entire day. It probably helped that the new VM is dual core, even though the bot itself only ever makes 1 request at a time, so I didn't expect this much of an improvement.

5
 
 

The server is becoming a tad bit too big for the VM it's running on, so I'll be moving it this weekend. Until it's back up and running, I have paused the bot.

6
 
 

Hi,

I don't post this to be malicious or rude, but simply out of concern.

I believe this bot is killing community growth on Lemmy. I keep coming across would-be-interesting communities only to find a wall of bot posts with zero comments, zero votes, zero engagement. And I walk away disappointed instead of getting into a new community. As, I'm sure, may other users are doing.

Sure, you say, "just block the bot" if I don't like it. But that doesn't stop this thing from stifling any real engagement and growth in communities. Surely if someone can "just...", you can "just" go back to reddit if you want to read reddit content that badly.

I admire the engineering you put into making this thing work. It's impressive, and honestly very cool. But I really think it's actively disengaging users, when Lemmy has enough of a hurdle to overcome in growing new communities.

Thats just my 2 cents. I'm not sure it will mean much, but I felt I had to share it. Again, no ill intent against what you've accomplished in creating this. Best wishes.

7
 
 

Hi, today I just learned about RDX and I realized that it solved my biggest gripe with Lemmit. That being that a lot of the video content I find on Lemmit is hosted on Reddit, and clicking through to it gives Reddit a pageview and puts me back on a site I've sworn never to go back to.

RDX is doing something similar to Lemmit, but rather than publishing reddit content to Lemmy, RDX is just making a whole site with all the reddit content. That makes it the perfect place for Lemmit to direct to for links to content which is hosted on reddit.

Previously I was just downvoting any lemmit content that directed me to reddit, but now on desktop, I'm able to use a redirect plugin to redirect all reddit links to RDX, which is great.

Unfortunately this doesn't work on mobile, because the embedded safari client used in most mobile apps will not run the redirect plugin I have installed and configured in mobile safari.

And then I got to thinking, shouldn't this just be the default behavior of Lemmit anyway? It wouldn't require a lot of coding I don't think, and if I were to hazard a guess, I would think that anyone who's moved off Reddit but still accesses it's content via Lemmit would much prefer to not be redirected back to reddit to view the content.

Could we make rdx redirection the default behavior of Lemmit please?

8
 
 

Some subreddits change their 'title' month-to-month or seasonally with a theme. It would be good if the title was periodically retrieved and the title of the community here updated to reflect that.

9
 
 

I've noticed that new posts aren't coming through from the subreddits, and newly requested subreddits aren't getting the automated response either. Seems to have stopped about 3 days ago.

Apologies if this is known/planned, but seeing as I couldn't see any announcement about it I figured I ought to report it.

10
 
 

Hi, thank you for this bot, it's great!

[email protected] is flagged as nsfw even though the actual reddit ( http://reddit.com/r/formula1/ ) isn't can this be fixed?

Thank you!

11
 
 

As discussed here, I have implemented a minimum level of upvotes that a post needs to have on reddit, as well as a minimum ratio of upvotes to downvotes.

Right now I have those configured to require at least 5 upvotes, and more upvotes than downvotes (0.51). At first glance this already seems to be great improvement. There might be some tweaking later.

As a side note I have now switched from using the reddit RSS feed, to using the JSON feed. This was required in order to get easy access to the upvote/ratio properties. So there might be some new and interesting new bugs introduced because of that. It's a brave new world.

Needless to say, the first thing I'll do after releasing this, is plop down on the couch with a beer, and hope this doesn't crash. Fingers crossed!

12
 
 

Some of posts content were cut off, one example : https://lemmit.online/post/132266

Is there any way to fix that?

13
 
 

I'd like to hear some feedback on this, or approach vectors.

Right now the bot is rather spammy. I was hoping that by using Reddits HOT feed, it would return have some level of quality control (I know, right?). Unfortunately, it seems that in most cases, it will just return anything that's new. The downside of this is that a lot of garbage gets through, and the bot spends a lot of time scraping the underlying page to get the details.

I propose to only archive reddit posts that have a karma score of 5 or higher. In case of subs that hide the karma scores of posts for a certain time, they'd have to be at least 2 hours old, so that the Reddit moderators can weed out garbage on our behalf.

Do you folks have any thoughts on this?

Secondly, I want to put sticky comments on each community, with links to native Lemmy communities that cover the same subject. For this I would need some kind of API, or a master list of... oh, I see sub.rehab has just the thing I need. So expect that somewhere this week :).

14
 
 

So I replied to a comment in a thread the bot posted over in one of the television communities, but I noticed tonight when I was viewing that community in Memmy all of the threads the bot was posting appeared to be empty. Just the title, and the info about where the original post came from and who posted it, but otherwise, pretty useless. I wondered if it was a mistake from the bot or the client I was using, so I tried it over on wefwef, and I saw the links there.

So I guess I don’t know if there’s a bug in how Memmy displays the content, or a bug with how the bot posts, or if it’s just an inconsistency or whatever, but I thought someone might want to know to make some changes.

15
 
 

See you on the other side!


So the update is done, but the bot was offline for 6 hours, and needed to catch up.

Unfortunately, another update slipped through, which switched the default feed from www.reddit.com to old.reddit.com, which has the side effect of changing all the urls in the posts as well. On one hand this is great, because new reddit sucks. On the other hand, this is terrible, because for every post the bot encounters, it checks if it already exists on lemmit... based on the url.

So for every post the bot encountered, it went like "old.reddit.com/r/blabla/123? Haven't seen that one yet, there's an www.reddit.com/r/blabla/123, but that must be something completely different, let's post it again!"

This also meant that the bot took over a minute and a half to update each community because it takes a couple of second per post. When I went to bed last night, I figured it was just posting a lot of content because it had so much catching up to do. But this morning I figured something was off because it still hadn't caught up.

Anyway, the fix is out now. Sorry for all the duplicates. I need coffee now.

16
 
 

ChatGPT, write a post for the stuff that I have in my head and want to get out as an update.

Hmm. No brain implant yet. Guess I'll have to write this the hard way.

Syncing update

It has been an eventful week. I successfully deployed the initial version of smarter content syncing, and have made some adjustments to algorithm since then. Most notably, communities with only 1 subscriber (the bot) will no longer receive updates, and communities with fewer than 5 subscribers or with a low posting frequency will only be updated twice a day. Furthermore, for the highest update priority (every 10 minutes), a community must have a minimum of 50 subscribers. Implementation details can be found in the decide_interval() method over here.

Being a developer is fun

Meanwhile... Damnit, bot is stuck again.

2023-07-08 10:13:39,945 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:48 ago, interval 120 minutes
2023-07-08 10:13:40,653 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:45,324 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
2023-07-08 10:13:46,333 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:54 ago, interval 120 minutes
2023-07-08 10:13:48,581 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:51,227 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
...

1 bugfix and deployment later:

2023-07-08 10:46:42,836 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  3:03:51 ago, interval 120 minutes
2023-07-08 10:46:43,573 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:46:48,327 - utils.syncer - ERROR - Couldn't find post on https://old.reddit.com/r/BustyNaturals/comments/14told8/latina_bodies_are_the_best/, skipping.

Defederation

Meanwhile, the folks at https://lemmy.world reached out to me to tell me they're defederating Lemmit. They are not fond of high volume of posts made by the bot, and the fact that there are now (quick check) 462 communities on this server all being moderated by a single person. They have already received a couple of complaints about spam, and it didn't help that some requests for NSFW subreddits were not marked as NSFW. Occasionally, those subreddits had explicit thumbnails that appeared in the 'All feed' without warning.

I had a good talk with the LemmyWorld admin, wherein they explained their point of view, and I explained mine. I understand their decision to disassociate with Lemmit, and appreciate their attempt to contact me. Other instances like Beehaw, and some smaller ones have also reached the same decision.

This does mean that you will no longer be able to get new community updates on those servers. So make sure to check the blocked instances list on your home server if you were subscribed to Lemmit. At the same time I have removed all the subscriptions of users from those servers, in order to not affect the sync priority mentioned above. This does mean, that if LemmyWorld, Beehaw, etc ever decide to connect to Lemmit again (however unlikely), you will need to un- and re-subscribe from there.

Meanwhile, I've added a feature in the bot that will remove request posts for NSFW subreddits, if the post itself is not marked for NSFW. This should prevent explicit thumbnails showing up where they are not wanted.

Server growth

Last night I got an alert from my server monitoring that the disk is 80% full. Unfortunately, the disk is only 60 GB, so that doesn't leave much room for expansion. On the bright side, a good chunk of that is from Lemmys very verbose logging (like, 4 GB a day, which gets cleaned up daily), so it should last throughout the weekend if I tune that down. Furthermore, most of the storage growth is from from pictrs, the image upload part of Lemmy, and that can utilize an S3 bucket, rather than using the VM's storage like it is now. Using an S3 bucket offers a cost-efficient solution for expanding storage. Initial estimates indicate a monthly cost of around $5 for 1000 GB of storage, which should be sufficient for a while *fingers crossed*.

In the early days of Lemmit (literally, as the server is less than a month old) image uploads were limited to a default setting, which was something around 40 megabytes. That did add up quickly (thanks to half-minute porn gifs), and so I had to limit the max filesize to 1 MB, and later 0.5 MB. Once the server has switched to S3 storage, I can probably up that limit a little, although not too much.

Finally, Lemmy v0.18.1 has been released, and it contains even more performance boosts compared to v0.18.0, so if there's time left this weekend (and I can verify the Lemmit Bot is compatible), I will probably perform the upgrade.

17
 
 

In particular, posts to NSFW videos hosted on v.redd.it don't work on the www version. The links take you to the comments page, which blocks NSFW content, and nags you to go to the app.

old.reddit.com links just work without logging in.

18
 
 

See the bot in action here!

My instance running Leddit

Click here for a more detailed explanation about the bot's purpose

This bot is intended to be self-hosted. Unfortunately, I can't operate a public instance that takes subreddit requests because of how long syncing comments takes. For comparison, Lemmit takes 21 minutes to sync all of the subreddits on this instance using the old system, but Leddit takes the same amount of time to sync 3 subreddits with around 500k subscribers each once an hour. Smart syncing is planned, but it won't decrease the amount of time taken to sync big and active subreddits.

If you need help setting up an instance, feel free to ask questions in this thread or on the Leddit instance's community.

19
 
 

I want to follow many of the NSFW subreddit-communities from this account and probably request more, but the communities page doesn't show NSFW when not logged in (making an account on lemmit itself just to see the list seems like overkill and could be confusing about what's logged in where, if it's even allowed)

20
6
submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]
 
 

Okay, this one took me a bit longer than I planned (mostly due to sql fun and trying to use integers as minutes, WEEEE!).

Backdrop: Last week I disabled the mirroring of a couple of subreddits to the database, because they were initially requested but the nobody subscribed to them. At the same time, the bot was just crawling in a loop, starting at todayilearned, ending at latestsubreddit. As more subreddits were requested, this loop took longer and longer (21 minutes before I rolled out this update). This wasn't sustainable.

So here's the new situation. The more popular a community is, the more often it will be updated. In this case popular means a mixture between number of subscribers and the amount of posts it receives per day (Link to relevant snippet of source code).

In short, the most popular subs will be synced every 10 minutes, the next tier ever 30 minutes, 120 minutes and the content with either no posts per day or no subscribers (other than the bot), will only be synced every 12 hours. I hope this will hit a good distribution of updates vs popularity, but it will most likely be refined at some point in the future.

Speaking of distribution, we now have over 300 communities on this server 🥳, and their update intervals are spread out as such:

  • Every 10 minutes: 22
  • Every 30 minutes: 39
  • Every 60 minutes: 55
  • Every 120 minutes: 143
  • Every 720 minutes: 44

With this update running live (I started typing after I deployed it, and it has now gotten through the backlog of 'abandoned' subs), I'm going to step back from feature development for a few days. Any bugs that cause the bot to crash will of course continue to be addressed.

Have a blast!

21
 
 

Just as a disclaimer: I'm not complaining, it's great that all this exists at all :)

I'm just checking if this may be a bug or not: if I compare https://www.reddit.com/r/DotA2/new/ and https://lemmit.online/c/dota2?dataType=Post&page=1&sort=New then the bot is definitely skipping some posts.

For example, between "NothingToSay is called 'responsibility god' in CN Dota2 community." and "TIP: Medusa doesn’t reduce magic damage to her mana when BKB is up" there are 3 other posts on reddit directly, which are missing on lemmit.online

I'm just hoping this can be fixed since this bot makes populating the "real" dota2 community I moderate much easier, but some posts I want to cross-post are missing so I need to do shit manually.

22
 
 

Have you considered doing something similar for Mastodon, to allow interacting with toots within Lemmy UI? I know the opposite is possible and Kbin also has some kind of integration, but that doesn't seem to fully work at the moment either.

23
 
 

For example, it'd be nice for anyone that looks at/finds https://lemmit.online/c/dota2 to also find that https://discuss.tchncs.de/c/dota2 or https://lemmy.world/c/dota2 is an actual community that corresponds to that with user content, not bot content.

I'm sure there's lots of equivalents for other communities as well where that would make sense.

24
 
 

Before was running on the cheapest model (1 core / 1GB mem / 30GB storage) at $12/month. The machine was running pretty low on memory, causing it to start swapping, which in turn caused the cpu to get too busy, and everything to slow down.

Now it has a whopping 2GB of memory, and things seem to have calmed down - cpu is back to around 10-15% usage, and swap is down to 0. Happy times all around.

Because of the amount of subs being archived, it now takes about 15 minutes between updates for each sub (was 18 before I updated the VM).

I'm planning to build some kind of scoring system, based on the amount of posts per subreddit (per day?), and amount of subscribers on the lemmy community. That way communities with little subscribers or that don't see many posts per day, will only be updated once per hour.

At the same time, I feel that subs like AskReddit, OutOfTheLoop and other "question-based" subreddits shouldn't be archived by Lemmit. In my opinion those kind of posts are useless without those answers, but please let me know if you disagree.

25
 
 

[email protected] doesnt match up with r/perchance. its last post was 5d ago.

view more: next ›