Tools for resumable transfers

[ceremcem]

Description

On a mostly filled disk state (say %80 in use) there are some edge cases that breaks the "cache" and forces users to send everything as a whole:

1. If all common snapshots are deleted: https://unix.stackexchange.com/questions/462451/manually-synchronizing-two-btrfs-snapshots-with-rsync
2. If first transfer is interrupted.

Proposal

If a snapshot has to be sent as a whole (when sending diffs is not an option), rsync should be used instead of btrfs send | btrfs receive.

This boils down to the point: A snapshot has to be constructed manually

Resources

1. https://unix.stackexchange.com/questions/462451/manually-synchronizing-two-btrfs-snapshots-with-rsync
2. https://www.spinics.net/lists/linux-btrfs/msg83457.html

[ceremcem]

Consider using BUP: https://github.com/bup/bup

[ceremcem]

https://unix.stackexchange.com/questions/290996/can-you-tell-if-a-file-has-been-renamed

https://serverfault.com/questions/171871/tool-or-script-to-detect-moved-or-renamed-files-on-linux-prior-to-a-backup

Syrep (from: https://serverfault.com/a/201321/261445) See below test.

[ceremcem]

Syrep is tested

test-syrep.sh:

#!/bin/bash
#set -x
#info(){ unused(){ :; } }

info(){ echo $@; }

mk_test_file(){
    dd if=/dev/urandom of=$1 bs=256 count=100000 2> /dev/null
}

compare_folders(){
    info "Compare folders"
    diff ./x ./y -r --exclude=".syrep"
}

info "Preparing test environment"
rm -r ./x ./y patch-for-x patch-for-y 2> /dev/null
mkdir -p x
mkdir -p y
mk_test_file ./x/file1
mk_test_file ./y/file2

compare_folders

sync_folders(){
info "Syncing folders..."
info "------------------"
info "Update repos"
syrep -p --update ./x
syrep -p --update ./y

info "Create patches"
syrep -p -o patch-for-y --makepatch ./x ./y/.syrep/current.syrep
ls -lh patch-for-y
syrep -p -o patch-for-x --makepatch ./y ./x/.syrep/current.syrep
ls -lh patch-for-x

info "Apply patches"
syrep -pT --merge patch-for-y ./y
syrep -pT --merge patch-for-x ./x

compare_folders
}

info "See how Syrep handles empty files and folders (expected no diff)"
mkdir ./x/a
sync_folders || { info "ERROR: Can not handle!"; rmdir ./x/a; }

info "See how Syrep handles renames (expected only a few bytes of patch file)"
mv ./x/file{1,4}
sync_folders

if true; then
    info 'See how Syrep handles small modifications (expected a 10M patch)'
    dd if=/dev/urandom bs=10M count=1 >> ./x/file1
    sync_folders
fi

Output:

$ ./test-syrep.sh 
Preparing test environment
Compare folders
Only in ./x: file1
Only in ./y: file2
See how Syrep handles empty files and folders (expected no diff)
Syncing folders...
------------------
Update repos
Create patches
-rw-r--r-- 1 ceremcem ceremcem 25M Jan 11 19:52 patch-for-y
-rw-r--r-- 1 ceremcem ceremcem 25M Jan 11 19:52 patch-for-x
Apply patches
Compare folders
Only in ./x: a
ERROR: Can not handle!
See how Syrep handles renames (expected only a few bytes of patch file)
Syncing folders...
------------------
Update repos
Create patches
-rw-r--r-- 1 ceremcem ceremcem 65K Jan 11 19:52 patch-for-y
-rw-r--r-- 1 ceremcem ceremcem 65K Jan 11 19:52 patch-for-x
Apply patches
Compare folders
See how Syrep handles small modifications (expected a 10M patch)
1+0 records in
1+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.158956 s, 66.0 MB/s
Syncing folders...
------------------
Update repos
Create patches
-rw-r--r-- 1 ceremcem ceremcem 35M Jan 11 19:52 patch-for-y
-rw-r--r-- 1 ceremcem ceremcem 25M Jan 11 19:52 patch-for-x
Apply patches
Replace existing file? [N|y] n
Replace existing file? [N|y] n
Compare folders
Binary files ./x/file1 and ./y/file1 differ
Binary files ./x/file4 and ./y/file4 differ

Conclusions

Approach of Syrep can be used under the hood of a new tool, however, Syrep is not a correct tool for a general purpose file backup tasks.

[ceremcem]

See: https://github.com/datproject/dat