• 0 Posts
  • 7 Comments
Joined 1 year ago
cake
Cake day: October 25th, 2023

help-circle

  • Even if you ran a basic sqlite nexcloud, if properly optimized, you can deal with millions of files like its nothing. And that is the issue, the bugs and lacking optimization…

    4650g + 64GB ram + Mysql and it was file locking on just a 21k 10GB folder constantly.

    I have written apps (in Go) that do similar and process data 100 times faster then nextcloud. Hell, my scrapers are faster then nextcloud in a local netwerk, and that is dealing with external data, over the internet.

    Its BADLY designed software that puts the blame on the consumer to get bigger and better hardware, for what is essentially, early 2000 functionality.


  • Any media you want to long term store, you want to rotate. Aka, move data from A, to B, to C, B to C to A, C to … Any storage media not specific designed with redundancy in mind, will see bit rot over time, depending on humidity, heat, location (if you live 2k high vs sea level), etc.

    I always moved my media around like that, never had a issue with data that goes back 20 years.

    For me any media is no different then having a car parked out for 6 months. Sure, it will work, but your battery may (will) be death, you brakes may get locked up, etc… Or leaving a house or apartment unattended for 6 month.

    SSD for cold storage is just as good. Been using some SSDs going back to when 256GB was expensive. But like with HDDs, Tapes, etc… rotate and rewrite.

    One of the biggest mistakes i have seen in a government job, was making backups, never checking / rewritting those backups and the day they needed those backups. Well, the taps had damage (despite only being 2 years cold stored). So the loads of money they spend on a dedicated guy, who’s job it was to do backups every day, the expensive tap machines, etc, was all for nothing. Simply they did not do checks/rewrites. And nobody was able to blame him, because he was following the exact instructions for making those backups. He even brought up the issue but … “instructions are made by people more qualified then you”. O, they tried to blame him but he stepped to the union and it was quickly resolved in a “nobody was to blame” (because can not blame the managers above him, now can we ;0 ). His instruction got changed to: Verify every tape at minimum X times per year, and rewrite on new taps, with backups for every tape (also increased budget to buy more tapes instead of recycling). And that was like i said, government ;)

    Any storage media will fail, and its up to the end user to ensure you have regular rewrites, checks, recoverable parity of the files itself, … Say this as somebody that lost months of programming work, when his primary HDD failed and multiple backups ended up useless ( in the old floppy days ). Multiple backups helps but even that can be fraud. I have seen written DVDs go bad all at the same time (one reason i never backup to DVD or Bluray)…

    So far i found SSDs more reliable then most people realize. As long as i rewrite them, every so often.




  • The price premium for high capacity HDDs is the reduced number of required disks. Hence, less power consumption and simpler hardware to host the HDDs (e.g., 4 slots instead of 6 or 8).

    But also increased risk… Its way easier to secure 8* 5TB drives, then 2* 20TB drives.

    A 2* 20TB means you need to go Raid 1, where as 8* 5TB drives means you can go 2 Parity and 6 data drives. Aka 30TB usable and 4 times shorter rebuild times. And this also scales to the same effect…

    Combine that with something like Unraid, where you do not strip your data over a raid, your chance of a catastrophic loss of all data is WAY less with more smaller drives + parity (and the benefit of more independent filesystems, then one big bulk one. Speaking from experience).

    For companies, sure, those guys just make massive arrays with multiple redundancies but for self stores More and smaller is better (in my opinion), what is in major conflict because ironically, the cheapest drives in Germany are in the 14 to 20TB range. And you pay price premiums on the 12 to … 4TB drives.