What data hoarding and archiving is
Data hoarding is keeping large collections for the long term: family photos and videos, music, home recordings, old projects, saved websites. Archiving is the discipline behind it, making sure that what you saved years ago still opens, unchanged, years from now.
A big collection faces slow threats that an ordinary computer ignores. Drives wear out. Files can be damaged silently, with a few bits on a disk changing and no error message, so a photo opens years later with a grey band across it. People call this bit rot. Websites you relied on disappear. The answer is storage that checks itself, copies in more than one place, and the habit of testing both.
Matched drives or a mix: decide this first
How you’ll buy and add drives decides which tool fits.
- Matched drives of the same size, planned up front: ZFS. It pools the drives together, checks every block and repairs damage as it goes. Drives in the same group should be the same size, or the extra space on the bigger ones goes to waste.
- A mix of sizes, added one at a time, holding files that rarely change: SnapRAID, from the options below. Each drive keeps its own ordinary file system, and a parity drive can rebuild a drive that fails. Add a second parity drive to survive two failures at once. Unraid, a paid server system, uses a similar parity layout for mixed drives.
If you’re buying drives for the purpose, buy matched ones and use ZFS. If you’re putting drives you already own to use, the mixed approach wastes less.
How it works
Each threat has its own defence, and you need all of them:
| Threat | What protects you | In ZFS |
|---|---|---|
| A drive dies | A second copy, or parity, on other drives in the same box | Mirrors and RAIDZ |
| Silent damage | Checksums: a fingerprint for every block, checked on every read | Scrubs repair bad blocks from the good copy |
| Deleting or overwriting by mistake | Read-only copies of a folder as it was | Snapshots |
| Fire, theft, a failed box, ransomware | A copy somewhere else | Replication to another machine, or a backup |
| Drives wearing out | Watching each drive’s health report | Not ZFS’s job. Use a monitor such as Scrutiny |
A mirror is two or more drives holding identical copies. RAIDZ spreads data over three or more drives with parity, extra information that can rebuild a missing drive. RAIDZ1 survives one failed drive, and RAIDZ2 survives two.
What it needs
- Matched drives: two for a mirror, or three or more for RAIDZ. Choose CMR hard drives, not SMR, as the list at the bottom of the page explains.
- A computer to hold them, with enough drive bays or ports. TrueNAS asks for at least 8 GB of memory, plus a small SSD of its own to boot from.
- ECC memory, if your hardware supports it. It catches memory errors before bad data is written to disk. TrueNAS recommends it, and many systems run without it.
- A battery backup (UPS), so a power cut doesn’t stop the box halfway through writing. See power, noise and UPS.
- A second place for a copy. Even a perfect pool is still one box.
How ZFS fits in
- Checksums on everything. Every block of data has a fingerprint, checked each time it’s read.
- Self-repair. When a block fails its check, ZFS reads the good version from the mirror or parity and rewrites it.
- Scrubs. A scheduled read of the whole pool, usually monthly, that finds and fixes damage in files nobody has opened lately.
- Snapshots. Instant, read-only copies of a folder as it was. They take almost no space at first, and a deleted folder comes back in seconds.
- Compression, which TrueNAS turns on by default, saving space on documents with no noticeable slowdown.
- Replication. Snapshots can be copied to a second ZFS machine, even one at a friend’s house, and after the first copy only the changes travel.
The easy way to set it up
Most people use ZFS through TrueNAS Community Edition, a free server operating system built around it, with a full web interface. You install it like a new operating system, and after that everything happens in a browser:
- Make an installer stick. Download TrueNAS from its website and write it to a USB stick with a free app such as balenaEtcher or Rufus.
- Install it onto the small boot SSD, following the on-screen menu, and set the admin password when it asks. Your data drives stay separate.
- Open the web page at the address TrueNAS shows on screen, and sign in.
- Create a pool from your data drives: a mirror for two drives, or RAIDZ2 for four or more large ones.
- Check the scrub schedule. TrueNAS creates one for each new pool by itself.
- Add a snapshot schedule for the folders you care about, for example hourly snapshots kept for a week.
- Add drive health monitoring. Scrutiny is in the TrueNAS app catalogue.
- Set up the copy elsewhere, by replicating to a second TrueNAS box or sending a backup to cloud storage. See backups.
On a Synology, its Btrfs volumes do much of the same: turn on scheduled data scrubbing in Storage Manager. The terminal route, ZFS on any Linux machine, is there for people who want it.
Questions people ask
How often should I scrub?
Once a month is the common choice for home pools. A scrub reads every drive in full, so on a big pool it can take many hours. The server stays usable while it runs.
RAIDZ1 or RAIDZ2?
With large hard drives, rebuilding after a failure takes a long time and works every remaining drive hard. A second failure during the rebuild loses a RAIDZ1 pool, so many people choose RAIDZ2 for four or more large drives.
Can I add drives later?
Yes, with more planning than you might expect. You can add another group of drives to a pool, and recent versions of ZFS can widen an existing RAIDZ one drive at a time. Mixing sizes still wastes space, so plan the layout before you fill it.
How do I keep copies of websites?
The tools for that are listed lower on the page. ArchiveBox, the one that saves many pages at once, installs in one click from the ZimaOS and CasaOS App Store.
What’s the difference between an archive and a backup?
The archive is the collection itself, kept healthy by checksums, scrubs and snapshots. A backup is a second copy of it somewhere else, for the day the whole box is lost. A large collection needs both, and a single ZFS pool on its own only provides the first.
