Do not ever push to production without testing things first. I went and moved us to the beta branch because this commit caught my eye and I wanted us to have the emoji picker fixed: https://github.com/LemmyNet/lemmy-ui/commit/ae4c37ed4450b194719859413983d6ec651e9609

The beta branch was on dockerhub so I thought that it was at least minimally tested. I was sorely mistaken. Lemmy (the backend) itself would load, but lemmy-ui (the actual website that renders everything) would keep crashing when attempting to load. I couldn’t roll back because the database was migrated to this beta version and it couldn’t migrate back to the old version when attempting to launch said old version.

I had no choice but to restore from backup. We’ve lost a whole day’s worth of posts (anything after 3AM CST.) I’m really really sorry… blobcatsadpleading

I was just so excited to be able to unveil this, I didn’t take my time with actually testing it.

  • Spiffy Diffy@burggit.moeEnglish
    5·
    2 years ago

    It’s time to speedrun “repost everything and re-setup new community any% glitchless”! Here’s hoping for a PB! ablobcathyper

  • Burger@burggit.moeOPM
    4·
    2 years ago

    Also, to add: Thank fuck for Borg. It’s one of the few backup solutions that hasn’t corrupted itself just from doing incremental backups.

    • Burger@burggit.moeOPM
      2·
      2 years ago

      Right. I thought I was restoring the most recent backup, but it turned out that the order for the archive list was in ascending order. I picked the top one thinking it was the latest. So that’s why you saw 8 days when Burggit was back up for a split second.

      My server automatically takes a backup once daily in the early AM, and keeps (I think? I’d need to check) 7 on hand. Any older than that, and they’re pruned.

      • CookieJarObserver@burggit.moeEnglish
        3·
        2 years ago

        You should pick one day a week that is stored longer, if there is a repetitive issue it might corrupt over more than a few days.

        Man i thought i did something wrong when the posts from the last week where gone

          • CookieJarObserver@burggit.moe
            1·
            2 years ago

            Yeah had a problem once where a backup (made one once a day and keep 14) was overwritten by a corrupted data set and i basically lost most data, would be sad if that happens here if its avoidable👍