How To Vanquish Rsync

It’s a shame! Rsync is an old, stable, field-tested, nice, quick and versatile tool for backups. And if you had bad luck, like I did, and just cut and pasted the first best example found on the Internet, and that example didn’t behave the way I thought it supposed to, you are lost, because all the solution I found at Google at the first page, sound more like voodoo, rather than a real and reasonable explanation. :-( And because I write this Blog in order not run ever into same issues again, I decided to write this little solution for my rsync problem, so I can easily look it up one fine day I need it again. And I really hope, this little article will be higher rated at Google, at least higher than the other solutions. This would definitely save ppls time. Here we go. There are at least two known reasons for the “cannot delete non-empty directory” problem:

The Backup Problem

You’ve got two directories. One is called SOURCE and the other is called BACKUP. Now you want to sync these to directories, e.g. via a cron-job. This cron-job is the bad one, which I used the first time:

# m h dom mon dow command
0 2 * * * /usr/bin/rsync -avb --delete /SOURCE/ /BACKUP >/dev/null 2>&1

The first day, this cron-job was running, all was fine, and I didn’t saw anything suspicious. On the other day I’ve deleted a few files in SOURCE, but to my surprise, the files haven’t been deleted in BACKUP. Instead the deleted files have been renamed simply by adding a suffix to the filename. This suffix was a dash (“~“). So since these directories weren’t empty, they couldn’t be deleted, no matter what switch I’ve tried with rsync. Using the “--dry-run” switch, I could see, which directories couldn’t be deleted:

/usr/bin/rsync -avb --delete /SOURCE/ /BACKUP --dry-run

I’ve got a bunch of

cannot delete non-empty directory

The problem is the “-b” or the “--backup” switch. If you specify the backup option, you tell rsync not to delete the files, but to back them up. This is done by the previously mentioned suffix, the dash. And this whole matter is done in the directory, where you deleted your files. There are two solutions actually. The first one is not to use the –backup switch at all. You can even specify the “--prune-empty-dirs“, or short “-m” switch, to ensure empty folder are being deleted also.

/usr/bin/rsync -avm --delete /SOURCE/ /BACKUP --dry-run

Or you really intend to use the --backup switch, but then please also specify a backup directory for it, where all these deleted files are moved into. This switch is called: “--backup-dir=“. If you don’t specify it, don’t expect rsync to delete empy directories, because rsync cannot delete empty directories, and it really doesn’t matter whether you additionally specify “--force“, or not. In combination with “--delete” it’s even double senseless.

The Exclude Problem

The second problem wasn’t figured out by myself, but this Blog article referring to this article here. It describes a deadlock between the switches listed right below, even without using the notorious backup switch (-b) at all:

  • --delete
  • --exclude <exclude pattern>
  • --ignore-existing

Also here the --force switch is without effect. Here the writer points out, an additional

  • --delete-excluded

lead to the solution.

Hope, this helps. The first problem was tested on a Ubuntu x86_64 – Jaunty Jackalope (9.04), the second solution haven’t been verified by myself yet.

8 thoughts on “How To Vanquish Rsync

  1. Wil Alvarez

    Excellent explanation, now i understand why rsync wasn’t deleting all those files!

    I hope too that this good article be higher rated by Google, so other people can realize about the issue.

    Cheers

  2. rduke15

    Thanks a lot for this clear explanation. However, I still don’t know how to solve the problem. I modified a filter, and now exclude more stuff than before, so I run into the problem described here.

    rsync -a –delete –backup –backup-dir=”_before-`date +%F`” –exclude=”_before-*” –filter=”. filter.txt” /SOURCE/ /DEST/

    In other words, I want to save the deleted files under /DEST/_before-xxx. Which doesn’t exist in the SOURCE of course.

    If I add –delete-excluded, I suppose it will delete my backup-dir where I keep deleted/changed files.

    So I don’t know how to elegantly solve this problem (without manually deleting all the newly excluded files from the destination).

  3. Gold

    acmelab68: You’ll be pleased to hear that this is now the first hit on Google. At least of the search terms I used, which are appropriate for this issue.

    +1′d it also.

    Cheers,
    Gold

  4. GG

    I’d like to share that –backup-dir= should be an absolute path.. I tried a local path for the user and I ended up with infinite subfolders and no deletions/error reporting.

    Interested in versioning keeping a number larger than 1 previous versions. Any cool suggestion to share?

Comments are closed.