Hi,
Many of you have run into this issue lately where the second page of your feed loads so slow that it can even timeout and not load at all.
If anyone can test again to confirm my last fix actually works for everyone and not just me.
I’ll explain my findings in this post once fixed.
Update
The first red herring
I was initially chasing an issue in the UI, the symptoms looked like something we ran into in the past related to dead community icons timing out sequentially and making the page take forever to load. This one was probably fixed in lemmy-ui a while ago.
The second red herring
I noticed this morning that my “mod/new” view was taking 30s to load page 2.
I spent some time messing around, but that’s another issue (that is still unfixed), possibly related with how that specific query is implemented.
Anyway
The actual problem
It turns out this specific issue was completely different, this time it was a database issue.
In the default postgres settings, autovacuum’s analyze trigger is 50 + 0.1 × reltuples, so a table needs ~10% of its rows inserted, updated, or deleted before it re-analyzes. On a multi-million-row table that’s an enormous absolute number, and because the threshold scales with table size, the wait gets longer as the table grows. Append-heavy tables like post_read and comment_like basically sit at 2–4% of threshold forever. That degrades slowly and maybe eventually self-corrects, maybe.
What made it an actual issue… I’m not 100% sure… probably it was a stats reset (pg_upgrade doesn’t carry activity counters).
The long-ass countdown restarted from scratch, while pg_stats still held pre-upgrade values.
Basically, the psql query planner was thrown off by estimates from older table versions.
Why some people’s feed and not everyone’s?
Good damned question.
I think the planner was suboptimal for everyone and everything, but that some specific sorts and filters multiplied it to a catastrophic level for some users’ feeds.
Active/Hot are likely more expensive than New, people with specific language filters, hide read posts, etc.
I’m not exactly sure.
The temporary fix…
In the process of pulling my hair out, I eventually ran these psql queries:
ANALYZE post;
ANALYZE person;
ANALYZE local_user_language;
ANALYZE community_block;
ANALYZE instance_block;
ANALYZE person_block;
ANALYZE post_hide;
ANALYZE post_read;
ANALYZE community_follower;
1-2 seconds each, bingo, symptoms went away immediately.
The longer term fixes
Adjust autovacuum thresholds and/or just periodically analyze such tables.
Go live in the woods.
Thanks for fixing the bug. However, I found it genuinely useful for the page to not load in full… to stop doom scrolling!
Do you think pagination can be introduced? As an option or setting? Or is this something to address to jerboa that I’m using?
The web UI has pagination by default.
Not sure about Jerboa, I haven’t used it in a while
Great to hear. I noticed that looking at a specific community didn’t have the issue and that was my workaround. Glad it’s resolved.
deleted by creator
Thank you!
Go live in the woods
Take me with you please
(Thanks for the fix tho!)
HEYO! I can finally browse more than one page of All for the first time in a long while. Much appreciated!
Glad to hear, as it was essentially unusable before.
I’ve updated the post with more info.
Thanks!
Lets goooooooo thank you so much!
Why some people’s feed and not everyone’s? Good damned question.
It’s because some of us are prettier. 💅
No, but honestly it ran fine on my laptop browser but not my phone app. The phone app dynamically loads so I think at about page two was where it was running into the load error you described. My desktop lemmy is paged instead of dynamic so it was no issue there.
Interesting, I’ve been having the opposite issue where it runs on my app but not my browser.
Gah!
I guess I’m just prettier when I’m on my phone? ✨ ✨ ✨
I was having similar issues, but only when on the all feed
This is awesome.
Now can someone smarter than me explain or forward this to the people who run lemmy.wolrld.
Is this an issue you’re having with your lemmy.world account? And if so, on which feed and which sort type?
Yes. I thought it was the same problem that was happening here. All for feed type, and it doesn’t matter what I pick as the sort type.
I wish I could explain better but I’m not entirely sure how all this works. Like for example, shit just works is a different instance I think. But it is using the same software? so the fix should work? on this page https://sh.itjust.works/c/main I can hit next at the bottom and it loads a second page, but it does not work on lemmyworld.
The problem might very well be the same.
Probably, if you browse lemmy.world without being logged in, it’ll work fine, at least that how it was with us.
I’ll reach out to them and point them here.
I’m on jerboa 0.0.88_fdroid version and it’s working. Thank you so much :)
Confirmed, page 2+ now loading. Thanks!
Scrolling all feed is restored for me using Voyager, and it feels like the paging is loading much quicker too. Tremendous work, thanks bud!
Was wondering what that was… Works on my end!
Just dropping in to say that I don’t understand most of that but it is working now so thank you <3
Thanks!
