BorgBackup

Stuff related to BorgBackup, a deduplicating archiver with compression and encryption.

Patches

• Enable placeholder usage in all extra archive arguments
• Support xxh64 checksum in addition to the hashlib hashes in borg list
• Optimize compressor instantiation
• Correctly calculate compression ratio, taking header size into account, too
• Only store compressed data if the result actually is smaller
• Rename local preload() function to not overwrite keyword argument of same name , fixing a significant memory leak
• Add progress indicator for archive check

USeful recipes

Running borg non-root using capabilities.

List repository showing archive sizes.

List an archive along with the chunk IDs associated to files.

Performance benchmarks

Compression

Results of a non-scientific test of some of the compression options offered by BorgBackup.

Tests performed on a Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM, 2 TB Samsung 970 EVO SSD (LUKS encrypted).

The entire system was backed up, consisting of the Debian buster system files, sereveral Debinan buster and stretch VM images, a few Debian installer and live ISO images, some photos and my Email, yielding a somewhat representative mix of data.

A new local repository was created each time, so different chunker hash seeds are part of the equiation.

BorgBackup version 1.1.9 was used for these tests.

Compression of RAW image fises

Results of a non-scientific test of some types of RAW image formats using LZMA compression as well as ZSTD in the case where it did prove efficient, LZ4 did in none of the cases. Deduplication among the unique images did not happen in a noticeable way.

A new repository was created once and each test used a copy of that empty repository, so identical chunker hash seeds and thus identical chunking and deduplication are used.

BorgBackup version 1.1.11 with above patches was used for these tests, ie. including the copmression improvementns.

Check

Benchmarks of borg check operations, comparing different hardware, to get a feeling for the demands and run-time of these in practice.

Tests performed using identical copies of the same repository on a Samsung 970 EVO Plus 2 TB M.2 NVME and Western Digital WD Elements 10 TB USB 3 hard disk. Each installed into, respectively connected to a Supermicro X11SCA-F, Xeon E-2146G, 64 GB ECC RAM, with the NVME either in the on-board M.2 slot connected via DMI (yielding about 1.7 GB/s sequential read) or using a Supermicro AOC-SLG3-2M2-O PCIe to M.2 adapter connected installed in a PCIe slot directly connected to the CPU (yielding about 3.2 GB/s sequential read). Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM

BorgBackup version 1.1.9 was used for these benchmarks.

Sequential disk read speeds

Approximate sequential read speed estimated using dd with 4 GB block size.

Repository

This LZMA compressed repository was used in the benchmarks:

borg check

The segment verification step is I/O bound and takes the majority of the time of the entire operation. The other verification steps taking place after this are mostly CPU bound.

borg check --verify-data

The data verification stepe is mostly client CPU bound, but I/O latency seems to be important, as well.

Links

BorgBackup homepage
BorgBackup Documentation