Yesterday, you probably saw this informal post by one of our head admins (Chris Remington). This post lamented some of the difficulties we’re running into with the site at this point, and what the future might hold for us. This is a more formal post about those difficulties and the way we currently see things.

Up front: we aren’t confident in the continued use of Lemmy. We are working through how best to make the website live up to the vision of our documents—and simply put, the vast majority of the limitations we’re running into are Lemmy’s at this point. An increasing amount of our time is spent trying to work around or against the software to achieve what we want rather than productively building this community. That leaves us with serious questions about our long-term ability to stay on this platform, especially with the lingering prospect of not having the people needed to navigate backend stuff.

Long-time users will no doubt be aware of our advocacy for moderator tools that we think the platform needs (and particularly that we need). Our belief in the importance and necessity of those tools has only hardened with time. Progress of those tools, however—and even organizing work on them—has been pretty much nonexistent outside of our efforts from what we can see.[1] In the three months since we started seriously pushing the ideas we’d like to see, we’re not aware of any of them being seriously considered—much less taken up or on the way to being incorporated into Lemmy.

In fact: even within the framework of Lemmy’s almost nonexistent roadmap and entirely nonexistent timetable on which to expect features it has been made clear to us that improving federation or moderation on the platform are not big priorities.[2] We have implicitly been told that if this part of the software is to improve we will need to organize that from scratch. And we have tried that to be clear. Our proposal is (and has been) paying people bounties for their labor toward implementing these features, in line with paying all labor done on our behalf—but we’ve received mixed messages from the top on whether this would be acceptable. (Unclear guidance and general lack of communication is symptomatic of a lot of our relation with the Lemmy devs in the past few months.)

Things aren’t much better on the non-moderator side of things. The problems with databases are almost too numerous to talk about and even Lemmy’s most ardent supporters recognize this as the biggest issue with the software currently. A complete rewrite is likely the only solution. Technical issues with the codebase are also extensive; we’ve made numerous changes on our side because of that. Many of the things we’re running into have been reported up the chain of command but continue to languish entirely unacknowledged. In some cases bugs, feature requests, and other requests to Lemmy devs have explicitly been blown off—and this is behavior that others have also run into with respect to the project. Only very recently have we seen any overtures at regular communication—and this communication has not hinted at any change in priorities.

All of what was just described has been difficult to get a handle on—and having fewer users, less activity, and more moderators has not done a whole lot to ease that. We honestly find that the more we dig and the more we work to straighten out issues that pop up, the more pop out and the more it feels like Lemmy is structurally unsound for our purposes. (One such example of what we’re working with is provided in the next section.)

In summary: we believe we can either continue to fight the software in basically every way possible, or we can prioritize building the community our documents preach. It is our shared belief that we cannot, in the long-term, do both; in any case, we’re not interested in constantly having to fight for basic priorities—ones we consider extremely beneficial to the health of the overall Lemmy network—or having to unilaterally organize and recruit for their addition to the software. We are hobbyists trying to make a cool space first and foremost, and it’s already a job enough to run the site. We cannot also be surrogates for fixing the software we use.

PenguinCoder: A brief sketch of the technical perspective

I’ve said a few words about this topic already, here and here. Other Beehaw admins have also brought some concerns to the Lemmy devs. Those issues still exist. To be clear: this is a volunteer operation and Lemmy is their software; they have a right to pick and choose what goes into it and what to put a priority on. But we have an obligation to keep users safe and secure, and their priorities increasingly stifle our own.

In the case of this happening for open source projects, the consensus is to make your own fork. But:

The problem with forking Lemmy is in starting from all the bad that is inherently there, and trying to make it better. That is way more work than starting fresh with more developers. IE, not using Rust for a web app and UI, better database queries from the start, better logging/functions from the start; not adding on bandaids. A fork of Lemmy will have all of Lemmy’s problems but now you’re responsible for them instead.

We don’t need a fork, we need a solution.

To give just one painful example of where an upstream solution is sorely needed: the federation, blocking, and/or removal of problem images.

  1. You post an image to Beehaw.
  2. Beehaw sends your content out to every other server it’s federated with
  3. Federated server accepts it (beehaw.org is on their allowlist), or rejects it (beehaw.org is on their denylist)
  4. If the server accepts it, a copy of your post or comment including the images are now on that receiving server as well as on the server you posted it to. Federation at work.
  5. Mod on beehaw.org sees your post doesn’t follow the rules. Removes it from beehaw.org. The other instances Beehaw pushed this content to, do not get that notice to remove it. The copy of your content on Beehaw was removed. The copy of your content on other servers was not removed.
  6. The receiving federated instance needs to manually remove/delete the content from their own server
  7. For a text post or comment that’s removed, this can be done via the admin/mod tools on that instance
  8. For a post or comment including a thumbnail, uploaded images, etc; that media content is not removed. It’s not tracked where in Lemmy that content was used at. Admin removal of media commences. This requires backend command line and database access, and takes about a dozen steps per image; sometimes more.

There are dozens of issues—some bigger, some smaller—like this that we have encountered and have either needed to patch ourselves or have reported up the chain without success.

Alternatives and the way forward

If possible the best solution here is to stay on Lemmy—but this is going to require the status quo changing, and we’re unsure of how realistic that is. If we stay on Lemmy, it is probable that we will have to do so by making use of a whitelist.

For the unfamiliar, we currently use a blacklist—by default, we federate with all current and newly-created nodes of the Fediverse unless we explicitly exclude them from interacting with our site. A switch to a whitelist would invert this dynamic: we would not federate with anybody unless we explicitly choose to do so. This has some benefits—maintaining federation in some form; staying on Lemmy; generally causing less entropy than other alternatives, etc. But the drawbacks are also obvious: nearly everything described in this post will continue, blacklist or whitelist, because a huge part of the problem is Lemmy.

Because of that we have discussed almost every conceivable alternative there is to Lemmy. We are interested in the thoughts of this community on platforms you have all used and what our eventual choice is going to be, but we are planning on having more surveys in the future to collect this feedback. We ask that you do not suggest anything to us at this time, and comments with suggestions in this thread will be removed.

As for alternatives we’re seriously considering right now: they’re basically all FOSS; would preserve most aspects of the current experience while giving us less to worry about on the backside of things (and/or lowering the bar for code participation); are pretty much all more mature and feature-rich than Lemmy; and generally seem to avoid the issues we’re talking about at length here. Downsides are varied but the main commonality is lack of federation; entropy in moving; questions of how sustainable they are with our current mod team; and more cosmetic things like customization and modification.

We’re currently investigating the most promising of them in greater depth—but we don’t want to list something and then have to strike it, hence the vagueness. If we make a jump, that will be an informed jump. In any case logistics mean that the timetable here is on the order of months. Don’t expect immediate changes. As things develop, we’ll engage the community on what the path forward is and how to make it as smooth as possible.


  1. Other administrators have probably vocally pushed for these things, but we’re not aware of any public examples we can point to of this taking place. Their advocacy has not produced results that we’re aware of in any case, which is what matters. ↩︎

  2. Perhaps best illustrated by the recent Lemmy dev AMA. We’ll also emphasize that Beehaw’s admin team is not alone in the belief that Lemmy devs do not take mod tools or federation issues particularly seriously. ↩︎

  • Gaywallet (they/it)@beehaw.orgM
    link
    fedilink
    arrow-up
    98
    ·
    edit-2
    1 year ago

    A few high level notes about this post, given some of the discussions and behavior in the informal chat post by Chris the other day:

    • We understand this is perhaps the biggest crossroads we’ve hit yet, and a seriously big issue. It’s understandable that you might have strong emotions about the Fediverse as a whole, or the action we are taking as an instance. If you are not from our instance and you come into this thread with a short hostile comment about how we aren’t respecting your views or that we should never have joined the Fediverse in the first place, your comments will be removed and you will be banned.
    • Any suggestions for what we should do, that involve actual effort or time, such as finding developers to fix the problems we’ve had should be accompanied with an explanation of how you’re going to be helping. We’ve lodged countless github tickets. We’ve done our due diligence, so please treat this post with good faith.
    • Similarly doing nothing more than asking for more details on the technical problems we are struggling with, without a firm grasp of the existing issues with Lemmy or the history of conversations and efforts we’ve put in is not good faith either. We’re not interested in people trying to pull a gotcha moment on us or to make us chase our tails explaining the numerous problems with the platform. If you’re offering your effort or expertise to fix the platform you’re welcome to let us know, but until you’ve either submitted merge requests or put in significant effort (Odo alone has put in hundreds of hours trying to document, open tickets, and code to fix problems) we simply may not have the time to explain everything to you.
    • I want to reiterate the final paragraph here in case you missed it - we are not looking to make any changes in the short term. We expect it would be at the minimum several months before we made any decisions on possible solutions to the problems we’ve laid out here.
    • Finally, I want to say that I absolutely adore this community and what we’ve all managed to build here and that personally, I really care about all of you. I wish we weren’t here and I wish this wasn’t a problem we are facing. But we are, so please do not hesitate to share your feelings 💜
    • potterman28wxcv@beehaw.org
      link
      fedilink
      arrow-up
      18
      ·
      1 year ago

      Similarly doing nothing more than asking for more details on the technical problems we are struggling with, without a firm grasp of the existing issues with Lemmy or the history of conversations and efforts we’ve put in is not good faith either. We’re not interested in people trying to pull a gotcha moment on us or to make us chase our tails explaining the numerous problems with the platform

      This is understandable but leaving platform is a big decision and the technical reasons are not really clear. Or at least they are not really crystal clear from the posts I have read. As end users we don’t really have much of a choice except to trust you.

      Personally one example I have is the lack of moderation tools. I have read numerous times that it was a problem. But I do not know what it means practically speaking - what is missing exactly.

      You do not have to explain it and I am not asking it of you. But I just want to say that I feel like there are details that sound to be very relevant to your future decision but are yet undisclosed. Or maybe I just missed them

      Thanks for all the work into making Beehaw what it is today. I joined during the Reddit exile and I’m happy to have found this community. I hope it continues to thrive

      • Lionir [he/him]@beehaw.orgM
        link
        fedilink
        English
        arrow-up
        36
        ·
        1 year ago

        Some of the moderation issues that we’ve talked about in the past are linked in the OP post. I will say that it has only gotten worse over time. I cannot think of a single moderation feature which actually fully works. That is how bad I think things are. They’re all broken in subtle ways. Yes, even reports are broken.

        • potterman28wxcv@beehaw.org
          link
          fedilink
          arrow-up
          4
          ·
          1 year ago

          Given that it’s still a newish project I do not find it abnormal to have broken features though I understand that it must be frustrating to have to deal with issues like that.

          If a benevolent user were to work on fixing such bugs perhaps the problem would get solved ? Maybe once the new contributors catch up, one of them will make development in that direction ? Or do you believe there is really no hope for the project in that regard ?

          • Hisnitch@beehaw.org
            link
            fedilink
            arrow-up
            14
            ·
            edit-2
            1 year ago

            Well, under any other circumstances, I would absolutely agree. Many users for other projects do, and have, volunteered their time, money, and blood (both in a metaphorical and a somewhat literal sense) so that a project can survive and live. However, that’s just not the case here.

            More then enough end users have tried to make the switch to become engineers, developers, bug hunters, the full works. They have offered up their time, their blood, their money, and their minds trying to help Lemmy get off the problems it currently sits on. There isn’t a bereft amount of people, both experienced and inexperienced, unwilling to help, in fact, there’s probably too many spoons in the pot.

            What I’m trying to say is that, it’s not problem with the helpers, it’s a problem with those who need the help, because it appears that those who need the help don’t really want it in the first place at all.

            • potterman28wxcv@beehaw.org
              link
              fedilink
              arrow-up
              10
              ·
              edit-2
              1 year ago

              Sorry if I missed it, but do you have a specific example where the proposed help was denied by the maintainers? A case where they clearly acted against it such as a merge request denial, prematurely closing the issue or explicitly telling contributors to not contribute on that?

              I see there is an issue here: https://github.com/LemmyNet/lemmy/issues/3275

              This issue was too large so lionir asked it to be split and linked two related issues like https://github.com/LemmyNet/lemmy/issues/3662

              So far I am not seeing anything like maintainers refusing the help - but perhaps this happened in private side channels or something like that?

              • afbpinheiro@beehaw.org
                link
                fedilink
                arrow-up
                4
                ·
                1 year ago

                I might have missed it too, but I don’t see what’s the core issue. I understand that the lack of moderation tools can be very frustrating, but Lemmy is open source and anyone can do a PR with improvements.

                If the devs discard them for no reason or are rude I would understand the “they don’t want any help” argument, but I saw no example.

                Creating another social media project is just splitting efforts and I’m not sure if the amount of work is not being underestimated by a big margin.

          • acastcandream@beehaw.org
            link
            fedilink
            English
            arrow-up
            3
            ·
            1 year ago

            It may be new, but there has been a huge surge in users and the jank has gone from “annoying” to “liability” now.

      • acastcandream@beehaw.org
        link
        fedilink
        English
        arrow-up
        4
        ·
        1 year ago

        I feel the technical reasons are pretty clear. The media example alone is a great one. If someone drops illegal media onto one instance now, every admin has to take a ton of steps to scrub it from their instances. Through no fault of their own!

        We hear every instance people say “we need moderation tools.“ Well, we don’t have them. I’m not mad at the developers, it has been a crazy few months and it’s not like any of these people are getting paid to do this. But it doesn’t matter what the intention is. If the tools aren’t there and are needed, eventually something has to give. There is too much burden on the admins of instances as it is. I’m not even talking about the financial side either!

    • m-p{3}@lemmy.ca
      link
      fedilink
      arrow-up
      15
      ·
      1 year ago

      I’m sad to see how dire the situation is on the moderation and admin side of things, but I just wanted to say best of luck and wish you the best no matter what you decide.

      The communities in this instance are occupying a good portion of my subscriptions (quality content most of the time) and I’ll definitely miss them if you end up leaving the fediverse.

    • acastcandream@beehaw.org
      link
      fedilink
      English
      arrow-up
      6
      ·
      edit-2
      1 year ago

      I think ultimately y’all need to do what allows you to create the community you (we all really) want, even if that means ending the instance and moving somewhere else. It is not a moral imperative to use Lemmy and this was always a risk.

      Lemmy has been a great experiment and there was a lot of momentum with the modest exodus that happened from Reddit, but if development is not going the way it needs to go, then we can’t stay here. Simple as that.

      I was unaware of the Herculean effort it took in order to remove unwanted media from other instances. That is not just an inconvenience, that is a potentially huge liability. It’s also one more reason why I don’t host my own instance (despite the fact that I probably know enough to get it started, but probably too little to stay out of trouble lol). 

      All of this is to say I support whatever decision you make. As long as Beehaw continues to exist I want to be a part of it, and I trust y’all to steer the ship accordingly.