Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Such as?
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
This is bullshit. Just take this as an example. I found it with one quick search and there are plenty more. Perhaps we should broaden our horizons a little rather than entrusting everything to some corpos.
My dude, I think you’re not super familiar with these technologies.
The most basic form of a content delivery network is a set of globally distributed servers that replicate content from a source of truth and a network to direct traffic to the closest server with a valid replica. So the cost here is servers.
With Lemmy, this problem is solved by eliminating the need for individuals to own many servers and a lack of need for trust between servers. The effort and cost is distributed among individual humans, making it manageable.
Now, if you’re familiar with blockchain, you probably perked up when you heard “lack of need for trust.” That’s what the blockchain was built for! Perfect fit, right? Ehh, not so much.
There’s two problems: acting as a proxy for content requires trust, and some single service needs to direct clients to the right local server. If I can arbitrarily join some network of serving content, I can always tell other servers in the network that I’m serving what they ask… and then serve ads. There’s no (reasonable and fast) way for the network to verify that I’m serving the correct content to every client. There’s no way to avoid the need for trust. Additionally, DNS, which directs you from mysite.com to 120.1.2.1, isn’t intelligent. It can’t direct clients to a geographically (or route-efficient, fucking ISPs) local IP. The best it can do is pick a random one from the pool. So when you go to lemmy.world, DNS can’t pick the correct server for you. So some set of servers needs to do the logic to select which local server to actually get content from. Those servers need to be central for the whole content delivery network.
This company you linked is just another company using “blockchain” to get investment money. If you read through their page to get a cursory understanding of how things work, an easy question comes up: what is the purpose of
mediatokens? Sure, maybe you can buy CDN time with it, but when you pay that token to someone providing compute… what do they do with that token? It’s worthless, just like crypto currency. Fucking scams. All that said, blockchain is a super, super interesting technology. There’s just very, very few suitable applications of it.I’ve worked in IT for about 12 years now. Everything from infrastructure monitoring to data analysis to data engineering to DevOps to backend engineering to product management. I’ve worked with systems serving tens of users and tens of millions of users. Happy to answer any questions. I love this shit.
If someone could figure out a trustless, decentralized way to implement a CDN, I’d eat that up in a second, but with my current understanding of the internet and available technologies, I don’t see a way it can work. At least, not with making every web page take >3s to load, which would absolutely kill websites.
Two things:
Isn’t there always trust issues though? Also, could SSL passthrough help in that?
Instead of CDN for protection, couldn’t a local WAF help solve this too?
Great breakdown
I could a agree with the first part and it does not contradict with the idea of a distributed network for content saving. Think about it this way. Instead of one big local server farm you have multiple small local servers which together form a global network. Now we come to the blockchain. As you pointed out you get these tokens for the CDN time the storage or more generally the server operation costs. Of course the blockchain these tokens are hosted on (Solana) do have to be trustworthy (which in this case they may not be. I don’t like solana that much either). But does that mean that this could not be achieved? It seems logical to me that with a distributed storage and computing network something like this could be achieved very efficiently and cheaply. Heck I’m using a decentralized VPN right now that works with the same principles I mentioned. Or take the Helium network for example? Don’t you see the potential there? Like with all technology these things have to mature but with my understanding they are pretty much doable.
Sure, its doable, but if we return to OP issue, is it available and usable now? If there’s a service provider I’d trust to do this, it’s CF, they have a good, solid product and they have not given a reason to doubt their business ethics yet.
Did I ever say something else? People act as if i had condemned anyone or forced them to choose an alternative. No, for now cloudflare is a solid option but I think that we should consider more open alternatives.
They exist sure, but as others have said, there’s still a lot of links in the chain to smooth out. And for a mission critical application like this you’ll always want to chose the most stable offering.
That’s the first reasonable argument. That’s also why I said that we should stick with cloudflare for now but I’d love to see something more decentralized and open in the future.
They lost me at building a CDN on top of a blockchain. Why?
It’s rather the other way round. Complement a distributed CDN with a blockchain.
Stupid
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
If you don’t know what a content delivery network is, here : https://www.cloudflare.com/learning/cdn/what-is-a-cdn/
A CND is very costly to run in an effective way. And because it is an intermediary server between the user and content server, the market is already pretty full. So competing with the CDN giants is practically impossible in a decentralised manner.
Because of what a CDN does (cache website elements closer to the user, protect the website against ddos…), it cannot be a cheap weak server, or it’s the one which will get overwhelmed by the ddos, or even the users.
Another limiting factor is that in decentralisation, that means different companies, and so many separate plans to pay, which is just impossible for a company.
If it was decentralized, a company would have to go and pay 100 different companies (which is more expensive, du to the server costs and each companies having their own staff to may (even if it’s just 1 person per company)) just to offer a quick access to the users around the world, which is just impossible.
A CDN isn’t a great comparison to DDOS mitigations. CDN spreads the load amongst multiple locations that are distinct entities. Any one can be down and the rest functions fine. They generally exist on separate domains and are not inherently codependent.
DDOS requires an inline solution. A layer acting as a man in the middle to deflect or absorb the traffic destined to Lemmy.world, for example. That’s not something that can be readily be decentralized while there’s only one ingress to Lemmy.world.
I know well what a CDN is and that’s why I don’t understand why you build a DISTRIBUTED content delivery network on a single corporation. I mean, the whole architecture is based on decentralised servers that precache the content and share the service load. Why not create an independent network that provides this bandwidth and where each node is rewarded according to its contribution? I know blockchain is a term that pisses a lot of people off, but it’s basically the best way to incorporate trust and monetisation into a decentralised system.
Removed by mod
It’s an interesting question but the knee jerk reaction towards decentralization isn’t always a silver bullet. Bitcoin always screamed that concept while ignoring the role of clearinghouses. Decentralization can actually compound the issue. Not to dispel the solution but good to keep these things in mind.
It isn’t a silver bullet but in this case it is particularly suitable. I mean, the architecture of CDN is decentralised, but all these servers are controlled by ONE company. So why not leave the whole task to an independent network?
I think the biggest problem with such services is that they require lots of money to run which means that any well-meaning effort will eventually end up becoming a commercial service.
…and that’s where the blockchain comes in. This means that the individual contributions of the node operators can be directly recorded and compensated adequately.
Sure.
Tell me a good argument why not? How would you reward those people that contribute to said netowork?
The downsides of blockchain / cryptocurrency are well documented at this point.
You’re being down voted, but a p2p cdn is something that sort of already exists. IPFS is probably the most mature. As far as I know, it’d only work for static content though. It’s also an entirely different protocol so you’d have to use some sort of local gateway or plugin to make use of it.
I have several vms and dedicated servers that I sort of use as a DIY cdn. No where near as spread out or capable as something like cloudflare, but its also not incredibly expensive to do on a small low performance scale. DDOS mitigation is another story though, generally that is best handled by large networks that can soak up the throughput.
Yeah it’s also more of a potential that I wanted to point out. Over the years that I have been involved with blockchain projects, I have developed a feeling for where blockchains and decentralised networks are suitable and where they are not. In this case, however, it seems very feasible to me. In the end, CDNs are nothing more than a server network that caches the data locally and distributes the bandwidth. This is exactly what an independent network could do with the advantage of the blockchain to remunerate the contributions of the individual node operators. But I see that the notion of blockchain triggers a great aversion in most people.
I don’t have half the knowledge in IT you have, but i totally agree we should find a solution to seperate from mastadons who owns the whole network.
It’s very similar to how we shouldn’t give big corpos like GAFAM willingly our data/privacy or our foodchain shouldn’t be controled by a few corpos who serve poison… (the list goes on).
Most people just don’t care, they have nothing to hide or they won’t die if they eat one cheesburger from McDonald’s a week…
But in the case of lemmy I think (personal opinion) It’s because it’s easier, simpler, faster to setup right now. I’m sure if they had a better solution to not depend on cloudflare they would chose the other solution.
I mean your idea seems great, but how long would it take to put it inplace? How many highly qulified people are needed to make it work? How much will it cost…
I hope that in the long run, lemmy instances are going to find a better solution 😀
I’m only talking about the long run. For now cloudflare is a solid service. I’d love to see some experental approaches tho maybe from other smaller instances.
Build your own then.
Wanna know the beauty of Lemmy? If you don’t like how instances are ran you can create your own🙂
sure, this is just a wish of mine and I’m totally happy with our mods here.
This is a polite way of saying get lost.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
Oh man, you lost me at blockchain.
I block anyone who mentions a blockchain.
Blockchains are bullshit.
Wait whaat?
Chances are that you’re being sarcastic, but in the event you’re not or if others want to learn…
Interesting tech. Almost zero practically useful applications.
Blockchains are effectively reproducible, verifiable ledger systems. But if the ledger grows infinitely, your storage and compute costs also grow infinitely. I’ve heard this has been solved, but I haven’t seen an implementation yet. (If anyone knows of one, please share!)
Another issue is the proofing system. Bitcoin uses proof of work, which means you need to do more computational work to produce new blocks on the chain. If the computational work grows, that means you need more and more powerful computers. This means increased cost which means centralization as participants with less money to pay for compute get pushed out. Alternatively, there’s proof of stake, where having some amount of a token or some similar value/stake allows you to write new blocks. This does reduce the computation cost but still causes those with lots of tokens/stake to get even more tokens/stake, which in turn allows them to spend more for new blocks… which creates a loop towards centralization.
So basically, the technology that preaches decentralization naturally centralizes in practical use over time.
Blockchains are bullshit.
You can’t mitigate a man in the middle attack on a technical level… Because they are a man in the middle… That’s the point of using DDoS mitigation. Nothing’s stopping them from just sending incoming traffic to a phishing site if a bad actor was in control of it.
Dunno if this guy is just so stupid or is trolling at this point. Using random tech buzzwords that have no relevance to the issue.
You’ve never blockchained your decentralized DDoS backend with a bi-duplex CDN enumerator?
Well I did mitigate an attack before using quantum entanglement calibrated against the cosmological constant to mitigated carbon decay. Does that count? Oh and, blockchain and decentralized. Haha
I myself am not sure who here understands anything about blockchain technology. For you it’s just NFT images and shitcoins that you associate with blockchain, isn’t it? That knowledge is enough for you to understand the whole technology. Read my other comments and ask yourself first if you have a balanced information base.
For sure it’s not you. You sound like an amateur developer no-one would hire.
You are smoking crack. You clearly do not know what you are talking about.
You had me until you mentioned blockchain technology. How would a blockchain system help in that regard, anyway?
A blockchain can complement a decentralised network by introducing trust into such a network, where the individual members cannot be trusted. This makes it possible to accurately document actions and reward or punish them accordingly. If you take such a distributed CDN network as an example, a blockchain could help to directly reward the individual members according to their contributions instead of building everything on voluntariness and goodwill as in the Tor network.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxy
There are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Sure, but you still have to pay for servers to run the proxy instances on. Any DDoS of appreciable size will knock over the number of instances that lemmy.world could stand up. Interesting thought, though. Maybe CloudFlare or others use HAProxy internally? I’m actually not sure what tech they use
Is “decentralised” the new “blockchain”?
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
This explains nothing. It’s literally saying “one thing is bad, the other thing isn’t”. Try to explain why instead, if you do happen to have an explanation.
One of the things that makes Lemmy unique is the underlying decentralized infrastructure. I think it’s just a request to keep that mantra.
Why are the Lemmy devs asking for snake oil on their Donate page then?
Sitting comfy in a country where the financial system works for you elites is the real snake oil.
Just because you’re smart at writing code doesn’t mean you’re smart at other things :) Or more likely, maybe they’re ideology-driven rather than by practicality.
Lemmy is an unusual but fortunate example of where ideology and practicality line up.
If you can find an entire nation state that runs on crypto currency with a functional, stable economy, I’ll eat my words.
Why would I want a nation state to run on anything? The end of the nation state is the communist utopia!
I am for whatever erodes the illegitimate violence exerted by nation states to safeguard their parasitic domain. If it’s crypto, it’s crypto. If people not eating apples brought about their end, I would be out there burning orchards.
…what are you even talking about? Your sentence makes zero sense.
100% of the crypto hate I see is from citizens of neocolonial states. You lord your control of the financial system over us and when something threatens it, it’s always delegitimised for any number of reasons.
Take your pick: scam, destroying the environment, eroding state power etc.
A decentralised system/society will need a value layer to transact. You think Visa should be in control of that?
Just because you don’t like it, doesn’t make it snake oil. I hope you never find yourself at the mercy of a government that persecutes you and imposes capital control so you can’t even run away with your money. If crypto existed when my people were literally being genocided, my parents would not have to end up in a new country with nothing to their name.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
this is a meme right
It is obviously not. Why would it be?
Looks like a meme to me.
Stupid
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Thank you for the clear explanation. It seems a lot of folks here don’t understand the tech, but this explains things clearly and accurately
The goal is to mitigate attacks, it costs a lot of money to purpose build world spanning networks than can absorb large amounts of traffic. P2P type options are not a good fit.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
@jimmy90 @PropaGandalf And I was able to finally open up and stop being a lurker
I meant the CDN. The Fediverse per see is great!
You sound fanatical with this statement
What a
This isn’t a helpful reply. There’s no reason to just call someone a name without even explaining why you think what he said is moronic.
Agreed