Stuff related to BorgBackup, a deduplicating archiver with compression and encryption.
Running borg non-root using capabilities.
List repository showing archive sizes.
List an archive along with the chunk IDs associated to files.
Results of a non-scientific test of some of the compression options offered by BorgBackup.
Tests performed on a Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM, 2 TB Samsung 970 EVO SSD (LUKS encrypted).
The entire system was backed up, consisting of the Debian buster system files, sereveral Debinan buster and stretch VM images, a few Debian installer and live ISO images, some photos and my Email, yielding a somewhat representative mix of data.
A new local repository was created each time, so different chunker hash seeds are part of the equiation.
BorgBackup version 1.1.9 was used for these tests.
Method | Time | Original Size | Compressed | Deduplicated |
---|---|---|---|---|
none |
34:08 | 332.36 GB | 332.38 GB | 145.74 GB |
lz4 |
31:36 | 332.36 GB | 133.98 GB | 90.67 GB |
auto,zstd |
35:20 | 332.36 GB | 115.12 GB | 78.87 GB |
zstd |
34:44 | 332.36 GB | 114.50 GB | 78.57 GB |
zstd,6 |
44:58 | 332.40 GB | 113.18 GB | 77.73 GB |
auto,zstd,10 |
59:55 | 332.37 GB | 112.35 GB | 77.51 GB |
zstd,7 |
48:17 | 332.41 GB | 112.35 GB | 77.18 GB |
zstd,8 |
55:55 | 332.50 GB | 112.03 GB | 77.11 GB |
zstd,9 |
57:06 | 332.50 GB | 111.85 GB | 77.11 GB |
zstd,10 |
1:05:11 | 332.49 GB | 111.61 GB | 76.92 GB |
zstd,16 |
3:34:29 | 332.40 GB | 110.35 GB | 76.10 GB |
auto,zstd,22 |
6:27:56 | 332.48 GB | 109.26 GB | 75.59 GB |
zstd,22 |
7:34:02 | 332.41 GB | 108.35 GB | 74.72 GB |
zstd,19 |
6:22:30 | 332.45 GB | 108.33 GB | 74.61 GB |
auto,lzma |
5:34:14 | 332.45 GB | 106.42 GB | 73.50 GB |
lzma |
7:58:28 | 332.36 GB | 105.38 GB | 73.09 GB |
Results of a non-scientific test of some types of RAW image formats using LZMA compression as well as ZSTD in the case where it did prove efficient, LZ4 did in none of the cases. Deduplication among the unique images did not happen in a noticeable way.
A new repository was created once and each test used a copy of that empty repository, so identical chunker hash seeds and thus identical chunking and deduplication are used.
BorgBackup version 1.1.11 with above patches was used for these tests, ie. including the copmression improvementns.
RAW Image | Original Size | Compressed | RAW Image Size (avg.) | ||||
---|---|---|---|---|---|---|---|
Camera | Type | Method | Size | Ratio | Original | Compressed | |
Canon EOS 6D | CR2 | 8.01 GB | lzma |
7.75 GB | 1.033 | 22.2 MB | 21.5 MB |
Sony A7R IV | ARW (compressed) | 4.65 GB | lzma |
4.23 GB | 1.099 | 59 MB | 53.7 MB |
Sony A7R IV | ARW (uncompressed) | 8.62 GB | zstd,6 |
6.27 GB | 1.375 | 117.4 MB | 85.3 MB |
Sony A7R IV | ARW (uncompressed) | 8.62 GB | lzma |
5.14 GB | 1.677 | 117.4 MB | 70 MB |
Benchmarks of borg check
operations, comparing different
hardware, to get a feeling for the demands and run-time of these in
practice.
Tests performed using identical copies of the same repository on a Samsung 970 EVO Plus 2 TB M.2 NVME and Western Digital WD Elements 10 TB USB 3 hard disk. Each installed into, respectively connected to a Supermicro X11SCA-F, Xeon E-2146G, 64 GB ECC RAM, with the NVME either in the on-board M.2 slot connected via DMI (yielding about 1.7 GB/s sequential read) or using a Supermicro AOC-SLG3-2M2-O PCIe to M.2 adapter connected installed in a PCIe slot directly connected to the CPU (yielding about 3.2 GB/s sequential read). Lenovo Thinkpad P52, Core i7-8850H, 64 GB RAM
BorgBackup version 1.1.9 was used for these benchmarks.
Approximate sequential read speed estimated using dd
with
4 GB block size.
Disk | System | OS | Squential read speed |
---|---|---|---|
Samsung 970 EVO Plus 2 TB | Thinkpad P52 | Debian buster | 3.6 GB/s |
Samsung 970 EVO Plus 2 TB | X11SCA-F (CPU PCIe) | Debian stretch | 3.2 GB/s |
Samsung 970 EVO Plus 2 TB | X11SCA-F (DMI PCIe) | Debian buster | 2.5 GB/s |
Samsung 970 EVO Plus 2 TB | X11SCA-F (DMI PCIe) | Debian stretch | 1.7 GB/s |
WD Elements 10 TB | 190 MB/s |
This LZMA compressed repository was used in the benchmarks:
Archives | Original Size | Compressed | Deduplicated | Unique chunks |
---|---|---|---|---|
110 | 56.31 TB | 44.69 TB | 1.68 TB | 2024761 |
borg check
Client | Server | Disk | Time |
---|---|---|---|
Core i7-8850H | - | Samsung 970 EVO Plus 2 TB (4 MB readahead) |
21:48 |
Xeon E-2146G (Debian buster) |
- | Samsung 970 EVO Plus 2 TB (4 MB readahead) |
23:05 |
Xeon E-2146G | Core i7-8850H | Samsung 970 EVO Plus 2 TB (4 MB readahead) |
25:10 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (CPU PCIe, 4 MB readahead) |
25:59 |
Core i7-8850H | - | Samsung 970 EVO Plus 2 TB | 25:53 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (4 MB readahead) |
26:39 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (CPU PCIe) |
27:49 |
Xeon E-2146G (Debian buster) |
- | Samsung 970 EVO Plus 2 TB | 29:47 |
Xeon E-2146G | Core i7-8850H | Samsung 970 EVO Plus 2 TB | 29:48 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB | 31:27 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (LUKS encrypted) |
47:17 |
Xeon E-2146G | - | WD Elements 10 TB | 2:37:20 |
Xeon E-2146G | Core i7-8850H | WD Elements 10 TB | 2:37:31 |
Xeon E-2146G | AMD GX-412TC SOC | WD Elements 10 TB (4 MB readahead) |
2:40:01 |
Xeon E-2146G | AMD GX-412TC SOC | WD Elements 10 TB | 3:05:30 |
The segment verification step is I/O bound and takes the majority of the time of the entire operation. The other verification steps taking place after this are mostly CPU bound.
borg check --verify-data
Client | Server | Disk | Time |
---|---|---|---|
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (CPU PCIe, 4 MB readahead) |
11:19:31 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (4 MB readahead) |
11:24:27 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (CPU PCIe) |
11:28:59 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB | 11:41:20 |
Xeon E-2146G | - | Samsung 970 EVO Plus 2 TB (LUKS encrypted) |
12:19:14 |
Xeon E-2146G | Core i7-8850H | Samsung 970 EVO Plus 2 TB | 13:55:40 |
Xeon E-2146G | - | WD Elements 10 TB | 16:36:56 |
Xeon E-2146G | Core i7-8850H | WD Elements 10 TB | 16:43:17 |
Xeon E-2146G | AMD GX-412TC SOC | WD Elements 10 TB (4 MB readahead) |
27:15:09 |
Xeon E-2146G | AMD GX-412TC SOC | WD Elements 10 TB | 28:39:23 |
The data verification stepe is mostly client CPU bound, but I/O latency seems to be important, as well.