Like some of you have already noticed, the instance has been down for the last week and a half. I documented the problem and the process of bringing it back online in a thread on our Discord, so if you are curious about that I’ll redirect you there.

The TL;DR is that the instance ran out of disk space, so the database crashed. No database, no Lemmy.

I solved it by moving our 30GB of images to a separate, much cheaper storage (we moved from 3,00€ / month down to just 0,02€ / month for the image storage!), freeing up a bunch of space for the database. This should keep us going for a while and allow us to scale much better in the future.

The new host

The new image host we are using, is an object storage located on a separate machnie from the Lemmy server (while previously images resided on the same server as the database and the instance). Because of this, you are likely to experience some milliseconds of delay with the loading of new images, because some back and forth between Lemmy and the image server needs to happen before you can see it (Lemmy downloads the image, sends it to the image storage, image storage returns a link to the image to Lemmy, THEN you can see the image. It takes a while).

Next moves

While I’m on a roll with the Lemmy updates, later today I plan on updating our instance to version 0.18.5 of Lemmy. This should give us even more stability and better uptime in the future, but might temporarily break user flairs in !pcm@lemmy.basedcount.com.

I am terribly sorry for the prolonged downtimes and I really appreciated all the people who joined our Discord server asking if they could somehow help or simply showing care for our work on the instance.
Please remember that this is mostly a solo project of mine, where I am left handling both the server admin side as well as the community facing one. It’s a lot for one guy to deal with, I’m sure you’ll understand.

EDIT: I’ve succefully updated the instance, everything seems to be working fine. Let me know if something feels odd or buggy.