(Hopefully this is an acceptable question. It’s my first time posting to this subreddit.)
Hoarding data is something that comes naturally to me, but unlike the average poster here my technical knowledge is limited. That can all be learned, but something I was curious about is roughly how much income one needs to maintain a sizable data hoard. I see all those crazy set ups with terrabytes and terrabytes of data, back-ups upon back-ups, people recommend services or software.
I myself have a modest living situation, several roommates, a single room of my own, the usual for someone starting out in the world. I’m not here to complain, I just need to know what my goals are and what I’m getting myself into by answering this call from my brain.
So is it possible to meaningfully hoard data on a budget by scraping together whatever storage I can, or is it better to focus on one’s career until they’re in a good enough place to afford a “proper setup” ?
Well my story goes:
- Laptop without a screen, windows, manual ‘acquisition’ of content, single usb 2tb ssd (4 months) - plex only (200$ i think)
- A tiny factor optiplex (with hw acceleration finally), still windows (3 months) - plex + started to use arrs (+200$)
- Switch to ubuntu & docker containers, added two external hdds 12tb + 18 tb (8 months) - plex, jellyfin, self hosted password manager etc + extensive monitoring with prometheus & exporters + everything behind a reverse proxy on own domain with SSO (+800$)
- Bought a huge case, i7 13700k, 64 gigs of ram, 5x12tb exos drives, put all in proxmox on vms, drives in raidz1, designated previous pc and drives as offsite backup (present) - 51+ services, self hosted google photos (immich), drive, usenet (+2500$ or so), plex, jellyfin, multiple arrs for different qualities etc.
So yeah, just start with whatever you have, it’ll rip your finances downhill from there :D
Get an SBC (Orange Pi or Raspi), Radxa SATA HAT, spare router plus extra LAN cables, and a handful of SATA HDD. Hardware cost…let’s just put it at 300-ish USD. Electric cost, let’s just put it at double digit (at worst, low triple digit) kWh per month. I started out with an external HDD which downs me about 120 USD a couple years ago
Can I have a question what do you mainly use your lab do?
As little as possible. Almost everything in my NAS is hardware from other systems that I’ve upgraded.
UnRAID is great in that you can start off small and grow at your own speed.
You can start hoarding data at any budget and in any living context.
The most basic, primitive, and easily usable/accessible hoard can start in the form of an external hard drive plugged directly into your PC. I have met people who store hundreds of terabytes of data across many external hard disks in this manner.
While this approach “works” for many people, more technically-savvy hoarders understand that keeping their hoards on external hard disks is a flawed approach that doesn’t scale well, provides limited availability, and doesn’t intrinsically account for backup/redundancy. Hence, you get to see the “crazy setups” with “back-ups upon back-ups” in this sub.
Fortunately, you don’t need a “crazy setup” to start hoarding a modest amount of data. In your living context, you can start by investing in a simple NAS.
A NAS, or Network-Attached Storage device, is basically just a PC with a bunch of disks installed. There are off-the-shelf options that you could go with (Synology) however most people in this sub would recommend that you use software like TrueNAS, Unraid, etc. which you can install on a typical desktop PC with internal disks installed. What sets a NAS apart from simply using external drives is that a NAS provides an always-on location on your network where you data can live, and also sports features and functionality, redundancy layers, pooling, etc, all of which you cannot readily/easily accomplish with external drives alone.
In a shared living situation where you only have access to a single room, form factor and noise are probably going to be your key constraints. You will want to start by identifying how much space you have that can be allocated for a PC to use as a NAS. This PC could be as small as a raspberry pi with a few attached disks, to as large as a full ATX tower with 8+ 3.5" bays. You also don’t need a particularly powerful PC to use as a NAS; anything from the past 10-15 years will work as a dumb NAS, however newer machines will benefit from lower power consumption and broader functionality beyond working as a simple storage device.
Now, moving onto “backups upon backups”… How much do you value your hoard? How devastated would you be if a brownout fried your NAS and toasted all of your hard disks? Put a dollar value on mitigating that sadness, and you now have your backup budget.
Backups can take many different shapes, forms and sizes. To provide you with the simplest example, let’s say you’ve just finished migrating your hoard from an external hard drive onto a NAS… Maybe the external drive is now your primary backup. Maybe you can encrypt the external drive and keep it in a locker at work or somewhere off-site so that your hoard is protected in-case your house burns down. Maybe you can purchase a cheap cloud subscription where you can encrypt and upload some of the most important data from your hoard. Maybe you “sync” your NAS with these backup locations on a routine basis so that your backups are fresh.
Generally speaking, the more important your data is, the more disparate backups you should have. On the flipside, if you’re hoarding something silly like Linux ISOs, maybe you don’t need any backups at all!
Also, and this can never be repeated enough, if you use any RAID/zPool redundancy in your NAS, this is NOT a backup.
Happy hoarding!
My web scraping cost about 3k-4k a year just in proxy bandwidth.
So you back up like websites? Why use proxy for that?
Time, care and stress is by far the most expensive resource you spend, the price of disks and hosting is minuscule in comparison.