Hi guys!
This weekend we will move lemmy.today over to using object storage for images. We will be serving images from Amazon S3 in the Oregon region (western USA).
The way lemmy software is designed right now, it caches every image federated from other instances. So even if we are small instance, we still have to store a lot of federated images locally on our disk. This leads to disk space running out quickly and we previously had to delete images because of this.
When we delete images, it removes not only those cached images but also user profile icons and banners, as well as community icons and banners. This is why we have some missing images under Communities right now, and also why users have lost their profile pics.
Its been very embarrassing to have to do this, and now we will move to object storage to prevent this from happening in the future. Its much cheaper compared to ordinary disk space and gives better performance for users, so its a win-win. We just need to do a one-time migration over to it.
**Estimated downtime hours: **
Oregon time: Sunday 3 am - 6 am
CET: Sunday 12 pm to 15 pm.
If you have any questions, you know what to do. :)
EDIT: Looks like it went well and images are now served from S3 instead of filling up our disks. :) The url to the images still looks like they are served by the instance, but thats by design appearently. In the background, they are fetched from S3.
Please fill free to re-upload any banners, user avatars or community pictures you had in place before that may have been broken by the disk cleaning before.
- When you do, you have to create a new picture with a new name for Lemmy to actually replace the image. Otherwise it wont work - ive tried myself. :)
Out of curiosity what is this instance running on now? AWS with EBS and you’re shifting storage to S3?
BTW Oregon region is my favorite, I have several instances based there for clients. Maybe not as many features as East but far better reliability in my experience.
The instance itself is actually running in Hetzners server hall in Oregon, and it’s been extreamly reliable. Not a single issue with them since we started this instance. :)
Now we will store images on s3, which means all image loadings will be from s3 to you users directly, and that means the instance doesn’t even have to send you any images anymore - just the link to them on s3. So that will reduce load on the instance and help to make it even faster.
I’m very curious myself how this will be after the migration. Excited for it. :)
Finally, Yes!!! So the object storage will be for those thumbnails and cache media things AND the other (core) content storage of the instance will be in the disk space, right? :O
And it’ll allow our instance to be faster :D I wonder if all the instances were doing that?
I hope the upcomming lemmy updates have this issue resolved too until we’re still fine :D
Yeah every image will be stored on s3 so I think we will see even faster speeds. Hope so!
Very excited for it! Thanks for the improvements to infrastructure, manager!
Hopefully all goes well. :)
Nice work!
Well done. Seems to be working great this morning!
Yes it’s really nice, and all my worries about disk space running out is gone. Now we enjoy Lemmy. :)
Wish you the best hope everything works out
Thank you. We are already up and migration is complete. It was faster than I expected!
Just looking around a bit now to verify that there is no weirdness… :)
