rsync - create all missing parent directories?

Question

I'm looking for an rsync-like program which will create any missing parent directories on the remote side.

For example, if I have /top/a/b/c/d on one server and only /top/a exists on the remote server, I want to copy d to the remote server and have the b and c directories created as well.

The command:

rsync /top/a/b/c/d remote:/top/a/b/c

won't work because /tmp/a/b doesn't exist on the remote server. And if it did exist then the file d would get copied to the path /top/a/b/c.

This is possible to do with rsync using --include and --exclude switches, but it is very involved, e.g.:

rsync -v -r a dest:dir  \
  --include 'a/b'       \
  --include 'a/b/c'     \
  --include 'a/b/c/d'   \
  --include 'a/b/c/d/e' \
  --exclude 'a/*'       \
  --exclude 'a/b/*'     \
  --exclude 'a/b/c/*'   \
  --exclude 'a/b/c/d/*'

will only copy a/b/c/d/e to dest:dir/a/b/c/d/e even if the intermediate directories have files. (Note - the includes must precede the excludes.)

Are there any other options?

See also [rsync: how can I configure it to create target directory on server?](http://stackoverflow.com/questions/1636889/rsync-how-can-i-configure-it-to-create-target-directory-on-server) — mivk, Oct 21 '16 at 11:29
Possible duplicate of [How to force rsync to create destination folder](https://stackoverflow.com/questions/9242135/how-to-force-rsync-to-create-destination-folder) — Evandro Coan, Oct 10 '17 at 15:07

score 30 · Answer 1 · edited May 23 '17 at 12:10

30

You may be looking for

rsync -aR

for example:

rsync -a --relative /top/a/b/c/d remote:/

See also this trick in other question.

edited May 23 '17 at 12:10

Community

1
1

answered Apr 01 '14 at 23:13

Balu

537
4
9

2

Why include `-a` too? I'm pretty sure it's not needed. – Kenny Evitt Jan 17 '15 at 21:42
1

@KennyEvitt `-a` uses recursive (`-r`) but not relative (`-R`) so `-a` is still needed sometimes to preserve owner/perms (`-aR`). You can test it running rsync as root (or another user) to see this behavior. – dhaupin May 03 '16 at 19:09
8

`--relative` will not _create_ missing directory components on the remote side like in the question. – juzzlin Oct 10 '16 at 12:42
9

Doesn't work for me, complains `rsync: mkdir "/home/constantine/Projects/schifra-pkg/pkg/schifra-git/usr/include/schifra" failed: No such file or directory` – Hi-Angel Jun 08 '18 at 12:51
5

@juzzlin, according to https://unix.stackexchange.com/a/496181/5783, since `rsync` 2.6.7, `--relative` will create missing directory components on the remote side if you use `.` to anchor the starting parent directory to create at the destination. See my answer at https://stackoverflow.com/a/55231772/107158. – Derek Mahar Mar 19 '19 at 00:11
@DerekMahar Great instructions. I have followed yours and was able to get missing directories created and copy worked. Thanks – Sergei G Dec 20 '19 at 05:21

score 18 · Answer 2 · edited Jun 18 '22 at 18:12

18

As of version 3.2.3 (6 Aug 2020), rynsc has a flag for this purpose.

From the rsync manual page (man rsync):

--mkpath                 create the destination's path component

edited Jun 18 '22 at 18:12

Yaroslav Nikitenko

1,695
2
23
31

answered May 09 '22 at 21:07

RaZ0rr

343
2
8

Eddie · Answer 3 · 2014-12-23T19:06:36.167

17

rsync -aq --rsync-path='mkdir -p /tmp/imaginary/ && rsync' file user@remote:/tmp/imaginary/

From http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/, but don't copy and paste from there, his syntax is butchered.

it lets you execute arbitrary command to setup the path for rsync executables.

edited Dec 23 '14 at 19:06

answered Dec 23 '14 at 18:41

Eddie

9,696
4
45
58

1

Very nice, this is preferable because it doesn't require a separate ssh connection. Forgot the `&& rsync` and was getting cryptic error messages. – John Lunzer Dec 14 '18 at 18:20
this is what i tried `rsync --archive --remove-source-files --rsync-path='mkdir -p /node1/node3/ && rsync' /xx/node1/node3/file.txt /node1/node3/file.txt' ` But throws error. `node1/node3/ ` path doesnt exist in destination. Am i missing anything?. I am copying in the same host. So `remote` is not used. – sjd Oct 07 '20 at 09:21

score 10 · Answer 4 · answered Aug 30 '13 at 10:11

10

i suggest that you enforce the existence manually:

ssh user@remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c

this creates the target folder if it does not exists already.

answered Aug 30 '13 at 10:11

mnagel

6,729
4
31
66

7

Yes - that is one way to do it. But it seems that my use case is such a useful scenario that there should be a command for it or some options to rsync. – ErikR Aug 30 '13 at 22:06
1

i am not aware of any such flag. – mnagel Aug 31 '13 at 08:11

score 7 · Answer 5 · answered Mar 18 '19 at 23:56

According to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, --relative works if you use . to anchor the starting parent directory to create at the destination:

derek@DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top1/a/b/c/d
derek@DESKTOP-2F2F59O:~/projects/rsync$ mkdir --parents top2/a
derek@DESKTOP-2F2F59O:~/projects/rsync$ rsync --recursive --relative --verbose top1/a/./b/c/d top2/a/
sending incremental file list
b/
b/c/
b/c/d/

sent 99 bytes  received 28 bytes  254.00 bytes/sec
total size is 0  speedup is 0.00

https://stackoverflow.com/a/19173623/107158 describes the same solution. — Derek Mahar, Mar 19 '19 at 00:01

score 2 · Answer 6 · answered Mar 16 '16 at 11:46

2

--relative does not work for me since I had different setup. Maybe I just didn't understood how --relative works, but I found that the

ssh remote mkdir -p /top/a/b/c
rsync /top/a/b/c/d remote:/top/a/b/c

is easy to understand and does the job.

answered Mar 16 '16 at 11:46

andrej

4,518
2
39
39

`--relative` didn't work for you, because it has nothing to do with this :) – juzzlin Oct 10 '16 at 12:43
@juzzlin Maybe I just didn't understood how --relative works. ;) – andrej Oct 10 '16 at 13:24
1

@juzzlin, according to https://unix.stackexchange.com/a/496181/5783, since rsync 2.6.7, `--relative` works if you use `.` to anchor the starting parent directory to create at the destination. See my answer at https://stackoverflow.com/a/55231772/107158. – Derek Mahar Mar 18 '19 at 23:58

jesseme · Answer 7 · 2018-07-10T15:50:46.213

2

I was looking for a better solution, but mine seems to be better suited when you have too many sub-directories to create them manually.

Simply use cp as an intermediate step with the --parents option

cp --parents /your/path/sub/dir/ /tmp/localcopy
rsync [options] /tmp/localcopy/* remote:/destination/path/

cp --parents will create the structure for you.

You can call it from any subfolder if you want only one subset of the parent folders to be copied.

edited Jul 10 '18 at 15:50

answered Oct 17 '17 at 18:20

jesseme

21
3

1

`cp: with --parents, the destination must be a directory` – Hi-Angel Jun 08 '18 at 12:53

score 2 · Answer 8 · answered May 13 '20 at 14:33

2

A shorter way in Linux to create rsync destination paths is to use the '$_' Special Variable. (I think, but cannot confirm, that it is also the same in OSX).

'$_' holds the value of the last argument of the previous command executed. So the question could be answered with:

ssh remote mkdir -p /top/a/b/c/ && rsync -avz /top/a/b/c/d remote:$_

answered May 13 '20 at 14:33

Dig

261
2
3

1

I didn't like this solution at first but it looks like it's the best choice for my situation where I'm trying to create directories that don't exist in the source hierarchy. – Sridhar Sarnobat Dec 28 '20 at 17:47

rsync - create all missing parent directories?

8 Answers8

Linked