I’m happy to hear that, always good news
I checked my backup drives, too!
I still don’t have them.
Better than some cloud leech service, there are tools you can use to back up your hard drive onto a friend’s spare space (and vice versa if you want) so that you’re both protected without added clowd fees.
Uh oh!
Good news, you can use someone else’s. Backblaze is pretty cheap. It adds up to about $5USD per Terrabyte, and it’s pro-rated. So 500GB is $2.50,etc.
Hetzner recently introduced some S3 compatible object storage, too.
Object Storage and “S3” are just ways to refer to a certain type of system. To interact with it, you just need tools that have S3 support built in.
Restic is fantastic. If you’d like some help coming up with the command to use, let me know.
Pretty much every popular backup solution supports S3 type storage.
Anyway, backups are good.
In this case, no news is great news!
I finally switched to a brtfs filesystem and set up regular snapshots. I’m far more pleased about it than I have any right to be.
Beware of bit rot though. Snapshots help with user error, not with cosmic rays or worse.
A separate full backup is still required as well.
Thanks! Yeah, I have a separate SBC connected to a spinny HDD for backups of user data, plus I mirror it in the cloud. The snapshots are for when I mess up something on my system.
BTRFS has native checksumming, so it will detect any bitrot that occurs. Additionally it supports various RAID levels. So if you have some level of replication or parity, then combined with the checksums, it will automatically correct bitrot as well.
A proper backup strategy is of course still necessary.
I’m running a 60TB btrfs RAID with all the bells and whistles myself and just recently had an instance of some file being fucked up (probably just the wrong metadata bit being affected or something), which I noticed because
btrfs sendwould repeatedly crash at that inum. All the redundancy may be there, but sometimes it’s not able to recover automagically.Not hating on btrfs at all - it helped me recover from a few fubar situations that could easily have been total data loss - but magical thinking (about all the fancy features) is dangerous.
Huh, that sound very weird… If for example you’re running RAID1, then all bits of the metadata should be duplicated. So unless the same bit of metadata was also corrupted on the other disk, it should be recoverable…
What checksum algorithm are you running?
blake2b checksum, zstd compression, raid1c4 metadata and raid6 data. Kernel 6.12, btrfs-progs 6.17, ECC RAM.
The files in the affected inode haven’t been touched for a few years. Dmesg was something about zstd decompression failed and prevented btrfs send of an incremental snapshot as well as accessing one single file.
Due to the size of the array, I don’t always get around to do a full scrub after a (albeit rare) system crash, so I wrote it off as probably that and didn’t analyze much further at the time.
Ah, it’s probably a result of running RAID6 then. All the parity RAID modes in BTRFS still has some issues, such as suffering from the “write hole” issue. This can result in data loss when the filesystem isn’t unmounted cleanly, such as a crash or power loss.
RAID5 and RAID6 are still not recommended for production use.
I know, but I’m poor and can’t afford RAID1 of the same capacity. Thanks for the advice anyhow.
