Hi all. I’ve been having some problems keeping fedia.io running - at the moment, either the message workers or the php web server processes are dying after an hour or so and I have to restart everything. I have been working with the mbin team and installed some updates that we hoped would fix the problems, but no luck. I am going to work on a cron job to automatically restart things once an hour. The down side, is that you’ll likely see some error 500’s if you happen to hit it when the processes are restarting, but it should happen quickly and refreshing the page should make it work again.

  • jerry@fedia.ioOPM
    0·
    2 months ago

    I moved fedia.io away from fastly. I have a nagging feeling it has something to do with fastly. Can you let me know if you continue to see this?

      • melroy@kbin.melroy.org
        0·
        2 months ago

        For now try Firefox or a fork: Floorp, LibreWolf, etc. I heard that works better… I know this isn’t the solution, but that is the best workaround atm.

        • jerry@fedia.ioOPM
          0·
          2 months ago

          Most interesting: the problem had only been happening on MS Edge on my laptop. I have been using safari on my phone without issue. Just a bit ago, i refreshed the page and now every time I revisit the site, I have to log back in, just like on Edge. It’s like the old session expired and the new ones aren’t sticking. I’ll try FF on my phone.

          Note: even in the time I started typing this reply to when I hit the “add comment” button, I got logged out

          • melroy@kbin.melroy.org
            0·
            2 months ago

            Note: even in the time I started typing this reply to when I hit the “add comment” button, I got logged out

            That is really bad indeed. And the only error you see on the server side is only “Invalid CSRF token”?

            • jerry@fedia.ioOPM
              0·
              2 months ago

              ok - I just had it happen again while looking at logs. interestingly, there was NOT a CSRF log when that happened. There were a bunch of other errors, but enough that I could look through all of them and see that they were all related to activitypub issues - signaturevalidator and the like

              • melroy@kbin.melroy.org
                0·
                2 months ago

                I really hope it’s not a session issue with Valkey or something (I don’t think so…). We are now just going deep into this issue I think. Both sessions & csrf. Since I notice already some weird config issues with csrf forms

                • melroy@kbin.melroy.org
                  0·
                  2 months ago

                  FYI. Reading: https://symfony.com/doc/7.2/security/csrf.html#installation

                  The tokens used for CSRF protection are meant to be different for every user and they are stored in the session. That’s why a session is started automatically as soon as you render a form with CSRF protection.

                  Moreover, this means that you cannot fully cache pages that include CSRF protected forms. As an alternative, you can:

                  • Embed the form inside an uncached ESI fragment and cache the rest of the page contents;
                  • Cache the entire page and load the form via an uncached AJAX request;
                  • Cache the entire page and use hinclude.js to load the CSRF token with an uncached AJAX request and replace the form field value with it.
            • jerry@fedia.ioOPM
              0·
              2 months ago

              I have so many errors in prod.log that it’s hard to tell for certain, but when I try to filter out those that are associated with failed federation events, that seems to be when I’m left with. I am trying again to see if I can confirm

    • melroy@kbin.melroy.org
      0·
      2 months ago

      I found:

      [2024-09-12T20:42:54.414611+02:00] request.ERROR: Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\BadRequestHttpException: "Invalid CSRF token" at AbstractController.php line 39 {"exception":"[object] (Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException(code: 0): Invalid CSRF token at /var/www/kbin.melroy.org/html/src/Controller/AbstractController.php:39)
[stacktrace]
#0 /var/www/kbin.melroy.org/html/src/Controller/FavouriteController.php(24): App\\Controller\\AbstractController->validateCsrf()
#1 /var/www/kbin.melroy.org/html/vendor/symfony/http-kernel/HttpKernel.php(183): App\\Controller\\FavouriteController->__invoke()
#2 /var/www/kbin.melroy.org/html/vendor/symfony/http-kernel/HttpKernel.php(76): Symfony\\Component\\HttpKernel\\HttpKernel->handleRaw()
#3 /var/www/kbin.melroy.org/html/vendor/symfony/http-kernel/Kernel.php(182): Symfony\\Component\\HttpKernel\\HttpKernel->handle()
#4 /var/www/kbin.melroy.org/html/vendor/symfony/runtime/Runner/Symfony/HttpKernelRunner.php(35): Symfony\\Component\\HttpKernel\\Kernel->handle()
#5 /var/www/kbin.melroy.org/html/vendor/autoload_runtime.php(29): Symfony\\Component\\Runtime\\Runner\\Symfony\\HttpKernelRunner->run()
#6 /var/www/kbin.melroy.org/html/public/index.php(7): require_once('...')
#7 {main}
"} []

      And you found:

      {"message":"Uncaught PHP Exception Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException: \"Invalid CSRF token\" at AbstractController.php line 39","context":{"exception":{"class":"Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException","message":"Invalid CSRF token","code":0,"file":"/var/www/mbin/src/Controller/AbstractController.php:39"}},"level":400,"level_name":"ERROR","channel":"request","datetime":"2024-09-12T18:54:45.620576+00:00","extra":{}}
{"message":"Uncaught PHP Exception Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException: \"Invalid CSRF token\" at AbstractController.php line 39","context":{"exception":{"class":"Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException","message":"Invalid CSRF token","code":0,"file":"/var/www/mbin/src/Controller/AbstractController.php:39"}},"level":400,"level_name":"ERROR","channel":"request","datetime":"2024-09-12T18:54:45.803347+00:00","extra":{}}

      Not sure yet what the root-cause is. But it’s on our radar now.

      • tiredofsametab@fedia.io
        0·
        2 months ago

        Y’all are great. Feel free to ask if you need me to try anything. I haven’t touched PHP in years, but I am a software engineer, so feel free to be as technical as you’d like.

        • melroy@kbin.melroy.org
          0·
          2 months ago

          We can definitively use more developers. We are currently with only two: me and bentigorlich (recently debounced left as well as e-five). I also didn’t use Symfony (the PHP framework behind it), but I now also got those skills in place… So no worries, we are happy to help you. You can join us at Matrix, so it’s easier to chat and discuss: Mbin Matrix space. I hope to see you there!

          • melroy@kbin.melroy.org
            0·
            2 months ago

            Sorry you also went through this: -> kbin.social (died) -> kbin.run (died) -> fedia. Kbin.run was the instance of debounced, mentioned earlier…

            • jerry@fedia.ioOPM
              0·
              2 months ago

              This annoys me about the fediverse - people take a chance on coming here and then repeatedly get left in the dark when their instance is shut down. That’s why I was so very happy when you and others helped me get fedia.io back to healthy.