Deleting files doesn't cause a new snapshot, file deletions are not logged

[Germar]

I don't know if it's intentional, but for me it's a bug.
When full rsync mode is enabled, if I delete a file or a folder from source, it won't be deleted in the next snapshot.
Also, if the deletion is the only change in source, BIT says there's no difference and hence doesn't take any snapshot.

BTW: isn't there a way to get a snapshot in full rsync mode even if there's no change? The checkbox for this gets disabled when you choose "full rsync mode".

---

Imported from Launchpad using lp2gh.

• date created: 2015-03-20T22:02:57Z
• owner: mauromol
• the launchpad url was https://bugs.launchpad.net/bugs/1434722

[Germar]

(by germar)
I can't confirm this here. Deleted files or folders are gone in next snapshot. And this would be a serious bug otherwise

Could you please describe your setup? Local or remote drive, source and destination filesystem, rsync version on both if remote...

'Check for changes' is disabled with 'Full rsync mode' because there is no such check. With normal mode this will deactivate the rsync --dry-run. But with 'Full rsync mode' BIT will create a new snapshot anyways and delete it afterwards if there was no changes. This is still faster than before.

[Germar]

(by mauromol)
It's a local backup. In my test case, both source and destination folders are on the same ext4 filesytem, but I observe this even on an older Debian Wheezy on a NAS box with BIT 1.0.10 installed on it, backing up from an ext4 volume to another ext4 filesystem on a different device.

Here is my test-case profile:

profile4.name=test
profile4.qt4.last_path=/
profile4.qt4.places.SortColumn=1
profile4.qt4.places.SortOrder=0
profile4.qt4.settingsdialog.exclude.SortColumn=1
profile4.qt4.settingsdialog.exclude.SortOrder=0
profile4.qt4.settingsdialog.include.SortColumn=1
profile4.qt4.settingsdialog.include.SortOrder=0
profile4.snapshots.automatic_backup_anacron_period=1
profile4.snapshots.automatic_backup_anacron_unit=20
profile4.snapshots.automatic_backup_day=1
profile4.snapshots.automatic_backup_mode=0
profile4.snapshots.automatic_backup_time=0
profile4.snapshots.automatic_backup_weekday=7
profile4.snapshots.backup_on_restore.enabled=true
profile4.snapshots.bwlimit.enabled=false
profile4.snapshots.bwlimit.value=3000
profile4.snapshots.check_for_changes=true
profile4.snapshots.continue_on_errors=false
profile4.snapshots.copy_links=false
profile4.snapshots.copy_unsafe_links=false
profile4.snapshots.cron.ionice=true
profile4.snapshots.cron.nice=true
profile4.snapshots.custom_backup_time=8,12,18,23
profile4.snapshots.dont_remove_named_snapshots=true
profile4.snapshots.exclude.1.value=.gvfs
profile4.snapshots.exclude.10.value=/dev/*
profile4.snapshots.exclude.11.value=/run/*
profile4.snapshots.exclude.12.value=/etc/mtab
profile4.snapshots.exclude.13.value=/var/cache/apt/archives/.deb
profile4.snapshots.exclude.14.value=lost+found/
profile4.snapshots.exclude.15.value=/tmp/*
profile4.snapshots.exclude.16.value=/var/tmp/*
profile4.snapshots.exclude.17.value=/var/backups/*
profile4.snapshots.exclude.18.value=.Private
profile4.snapshots.exclude.2.value=.cache/*
profile4.snapshots.exclude.3.value=.thumbnails*
profile4.snapshots.exclude.4.value=[Tt]rash*
profile4.snapshots.exclude.5.value=.backup
profile4.snapshots.exclude.6.value=_~
profile4.snapshots.exclude.7.value=.dropbox_
profile4.snapshots.exclude.8.value=/proc/*
profile4.snapshots.exclude.9.value=/sys/*
profile4.snapshots.exclude.bysize.enabled=false
profile4.snapshots.exclude.bysize.value=500
profile4.snapshots.exclude.size=18
profile4.snapshots.full_rsync=true
profile4.snapshots.include.1.type=0
profile4.snapshots.include.1.value=/home/mauro/tmp/test bit/src
profile4.snapshots.include.size=1
profile4.snapshots.local.nocache=false
profile4.snapshots.local.password.save=false
profile4.snapshots.local.password.use_cache=true
profile4.snapshots.local_encfs.path=/home/mauro/tmp/test bit/dst
profile4.snapshots.log_level=3
profile4.snapshots.min_free_inodes.enabled=true
profile4.snapshots.min_free_inodes.value=2
profile4.snapshots.min_free_space.enabled=true
profile4.snapshots.min_free_space.unit=20
profile4.snapshots.min_free_space.value=1
profile4.snapshots.mode=local
profile4.snapshots.no_on_battery=false
profile4.snapshots.notify.enabled=true
profile4.snapshots.path=/home/mauro/tmp/test bit/dst
profile4.snapshots.path.host=hppb
profile4.snapshots.path.profile=4
profile4.snapshots.path.user=mauro
profile4.snapshots.preserve_acl=false
profile4.snapshots.preserve_xattr=false
profile4.snapshots.remove_old_snapshots.enabled=true
profile4.snapshots.remove_old_snapshots.unit=80
profile4.snapshots.remove_old_snapshots.value=10
profile4.snapshots.rsync_options.enabled=false
profile4.snapshots.rsync_options.value=
profile4.snapshots.smart_remove=false
profile4.snapshots.smart_remove.keep_all=2
profile4.snapshots.smart_remove.keep_one_per_day=7
profile4.snapshots.smart_remove.keep_one_per_month=24
profile4.snapshots.smart_remove.keep_one_per_week=4
profile4.snapshots.ssh.cipher=default
profile4.snapshots.ssh.host=
profile4.snapshots.ssh.ionice=false
profile4.snapshots.ssh.nice=false
profile4.snapshots.ssh.nocache=false
profile4.snapshots.ssh.path=
profile4.snapshots.ssh.port=22
profile4.snapshots.ssh.private_key_file=/home/mauro/.ssh/id_dsa
profile4.snapshots.ssh.user=mauro
profile4.snapshots.use_checksum=false
profile4.snapshots.user_backup.ionice=false

Here is my test case:

---

cd /home/mauro/test\ bit
mkdir src
mkdir -p dst/backintime/hppb/mauro/4
cd src
touch one
mkdir test
touch test/two
backintime --profile-id 4 --backup-job

Back In Time
Version: 1.1.3-4

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Inhibit Suspend started. Reason: take snapshot
INFO: on process begins
INFO: Profile_id: 4
Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from /.fonts.conf is deprecated. please move it to /home/mauro/.config/fontconfig/fonts.conf manually
INFO: Call rsync to take the snapshot
INFO: Command "rsync -rtDHh --links -pEgo --info=progress2 --no-i-r --delete --delete-excluded -v --chmod=Du+wx --exclude="/home/mauro/tmp/test bit/dst" --exclude="/home/mauro/.local/share/backintime" --exclude=".local/share/backintime/mnt" --include="/home/mauro/tmp/test bit/src/" --include="/home/mauro/tmp/test bit/" --include="/home/mauro/tmp/" --include="/home/mauro/" --include="/home/" --exclude=".gvfs" --exclude=".cache/" --exclude=".thumbnails" --exclude="[Tt]rash_" --exclude=".backup" --exclude="_" --exclude=".dropbox_" --exclude="/proc/" --exclude="/sys/" --exclude="/dev/" --exclude="/run/" --exclude="/etc/mtab" --exclude="/var/cache/apt/archives/.deb" --exclude="lost+found/" --exclude="/tmp/" --exclude="/var/tmp/" --exclude="/var/backups/" --exclude=".Private" --include="/home/mauro/tmp/test bit/src/*" --exclude="_" / "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/" -i --out-format="BACKINTIME: %i %n%L" 2>&1" returns 0
INFO: Save config file
cp: l'obiettivo "bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/.." non è una directory
WARNING: Command "cp /home/mauro/.config/backintime/config /home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/.." returns 256
INFO: Create info file
INFO: Remove backups older than: 20050301-000000
INFO: Keep min free disk space: 1024 Mb
INFO: Keep min 2% free inodes
INFO: [qt4systrayicon] begin loop
INFO: Unlock
INFO: Release inhibit Suspend
INFO: [qt4systrayicon] end loop

ls -Rx ../dst/backintime/hppb/mauro/4/2*/
../dst/backintime/hppb/mauro/4/20150321-141346-456/:
backup info takesnapshot.log.bz2
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup:
home
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home:
mauro
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro:
tmp
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp:
test bit
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit:
src
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit/src:
one test
../dst/backintime/hppb/mauro/4/20150321-141346-456/backup/home/mauro/tmp/test bit/src/test:
two

rm test/two
backintime --profile-id 4 --backup-job

Back In Time
Version: 1.1.3-4

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Inhibit Suspend started. Reason: take snapshot
INFO: on process begins
INFO: Profile_id: 4
INFO: Call rsync to take the snapshot
INFO: Command "rsync -rtDHh --links -pEgo --info=progress2 --no-i-r --delete --delete-excluded -v --chmod=Du+wx --exclude="/home/mauro/tmp/test bit/dst" --exclude="/home/mauro/.local/share/backintime" --exclude=".local/share/backintime/mnt" --include="/home/mauro/tmp/test bit/src/" --include="/home/mauro/tmp/test bit/" --include="/home/mauro/tmp/" --include="/home/mauro/" --include="/home/" --exclude=".gvfs" --exclude=".cache/" --exclude=".thumbnails" --exclude="[Tt]rash_" --exclude=".backup" --exclude="~" --exclude=".dropbox" --exclude="/proc/" --exclude="/sys/" --exclude="/dev/" --exclude="/run/" --exclude="/etc/mtab" --exclude="/var/cache/apt/archives/.deb" --exclude="lost+found/" --exclude="/tmp/" --exclude="/var/tmp/" --exclude="/var/backups/" --exclude=".Private" --include="/home/mauro/tmp/test bit/src/*" --exclude="_" / "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot/backup/" --link-dest="../../20150321-142752-455/backup" -i --out-format="BACKINTIME: %i %n%L" 2>&1" returns 0
INFO: Command "find "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot" -type d -exec chmod u+wx "{}" +" returns 0
INFO: Command "rm -rf "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/new_snapshot"" returns 0
INFO: Nothing changed, no back needed
INFO: Command "rm -rf "/home/mauro/tmp/test bit/dst/backintime/hppb/mauro/4/20150321-142818-628"" returns 0
WARNING: No new snapshot
INFO: Remove backups older than: 20050301-000000
INFO: Keep min free disk space: 1024 Mb
INFO: Keep min 2% free inodes
Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 14: reading configurations from ~/.fonts.conf is deprecated. please move it to /home/mauro/.config/fontconfig/fonts.conf manually
INFO: [qt4systrayicon] begin loop
INFO: Unlock
INFO: Release inhibit Suspend
INFO: [qt4systrayicon] end loop

ls -Rx ../dst/backintime/hppb/mauro/4/2*/
../dst/backintime/hppb/mauro/4/20150321-142752-455/:
backup info takesnapshot.log.bz2

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup:
home

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home:
mauro

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro:
tmp

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp:
test bit

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit:
src

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit/src:
one test

../dst/backintime/hppb/mauro/4/20150321-142752-455/backup/home/mauro/tmp/test bit/src/test:
two

---

As you can see, I just deleted file "two", but BIT didn't find any change, and hence took no new snapshot.
If I uncheck the "full rync mode" I don't observe this behaviour and the new snapshot is taken correctly.

BTW: regarding the full rsync mode and no new snapshot if there are no changes: what I meant to say is that I would find very useful if I had the option to prevent BIT from deleting the new snapshot if no changes were detected (at least if either the option "keep partial snapshots in case of failure" is checked OR if there were no errors at all). In this way, if I backup every day, I will find a snapshot for every day, even if two days are exactly equal. Right now, if I don't find a snapshot, I can't know if it is missing because the backup failed (I don't want to keep partial snapshots in case of errors!) or rather because there were no changes from the previous day... I must check the logs to see.

[Germar]

(by mauromol)
As an additional comment: this test case doesn't show that if I have a deletion in folder "test" and other changes in another folder (not in "test"), then a new snapshot IS actually taken, but the deletion of files in "test" is not applied to this new snapshot. I observed this in real cases on the NAS box I mentioned.

[Germar]

(by mauromol)
Forgot to write about the rsync version: the test case was conducted on a Linux Mint 17.1 (based on Ubuntu 14.04):
rsync version 3.1.0 protocol version 31
The NAS box is using a backported rsync version:
rsync version 3.1.1 protocol version 31

[Germar]

(by germar)
Okay, I was able to reproduce it with your config. Thanks

First think I recognized is that paths for 'save config file' are not quoted correctly and so break with spaces. But that's not the root of this problem. I'll skip new release again and fix this first ;D

BTW BIT 1.0.10 didn't have 'full rsync mode'. So that must be something different.

[Germar]

(by germar)
Okay, here is what happens:

• BIT will decide if there where changes and a new snapshot should be taken/kept based on the output of rsync
• with 'Full rsync mode' deactivated we create hard-links in a separate step with 'cp -aRl' and rsync will sync the current local status over the existing hard-links. If a file was locally removed rsync will remove the hard-link on remote, too and print a message about what it has done
• with 'Full rsync mode' activated rsync will create hard-links with '--link-dest' and sync over current local status in one step. So if there was a local file deleted rsync will simply just not hard-link that file on remote and won't say anything about that. So BIT doesn't know what happened and if there where no other changes it will suppose that there was no changes at all and remove the new snapshot again.

I'm not sure if there is a way to tell rsync to print out a message when there was more files in --link-dest than local which could be used to determine there actually was a change.

But I'll add an option (as you suggested) to keep snapshots with 'Full rsync mode' regardless if there were changes or not.

[Germar]

(by mauromol)
IMHO if we're unlucky and in full rsync mode there's no way to know if there are changes or not in this particular case, having BIT create a new snapshot every time even if it might be equal to the previous one when full rsync mode is enabled, is a compromise that is far better than having a "wrong" snapshot or suppress the full rsync mode at all.
On the contrary, as I said earlier I would prefer to have a snapshot preserved in any case, independently from this problem.

[Germar]

(by germar)
I added the above mentioned option (http://bazaar.launchpad.net/~bit-team/backintime/trunk/revision/1072). It is default off.

So with next release 1.1.4 you can use the 'take snapshot regardless of changes' option and I'll try to find a way to be noticed of deleted files for over-next release.

[Germar]

(by mauromol)
Hi Germar,
please note that the English phrase for 'take snapshot regardless of changes' has a typo ("reagardless" instead of "regardless"). I couldn't find a way to change it myself through LaunchPad.

[Germar]

(by germar)
I didn't found any suitable way of detecting deleted files with 'Full rsync mode'. Only way would be to either go back and run a --dry-run first or to create hard-links with 'cp -aRl' again. Which both would make it very slow again.

There is already a bug report for rsync about this https://bugzilla.samba.org/show_bug.cgi?id=5263
and a discussion on mailing list https:///msg25381.html
Hopefully they will add an option to get informed about deleted files some day.