Error in check_call() subprocess, executing 'mv' unix command: "Syntax error: '(' unexpected"

Question

I'm making a python script for Travis CI.

.travis.yml

...

script:
  - support/travis-build.py

...

The python file travis-build.py is something like this:

#!/usr/bin/env python
from subprocess import check_call

...

check_call(r"mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder", shell=True)

...

When Travis building achieves that line, I'm getting an error:

/bin/sh: 1: Syntax error: "(" unexpected

I just tried a lot of different forms to write it, but I get the same result. Any idea?

Thanks in advance!

Edit

My current directory layout:

- my_project/final_folder/
- cmake-3.0.2-Darwin64-universal/
- fileA
- fileB
- fileC

I'm trying with this command to move all the current files fileA, fileB and fileC, excluding my_project and cmake-3.0.2-Darwin64-universal folders into ./my_project/final_folder. If I execute this command on Linux shell, I get my aim but not through check_call() command.

Note: I can't move the files one by one, because there are many others

I don't know which shell Travis are using by default because I don't specify it, I only know that if I write the command in my .travis.yml:

.travis.yml

...

script:
  # Here is the previous Travis code
  - mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder

...

It works. But If I use the script, it fails.

I found this command from the following issue:

How to use 'mv' command to move files except those in a specific directory?

Answer 1

You're using the bash feature extglob, to try to exclude the files that you're specifying. You'll need to enable it in order to have it exclude the two entries you're specifying.

The python subprocess module explicitly uses /bin/sh when you use shell=True, which doesn't enable the use of bash features like this by default (it's a compliance thing to make it more like original sh).

If you want to get bash to interpret the command; you have to pass it to bash explicitly, for example using:

subprocess.check_call(["bash", "-O", "extglob", "-c", "mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder"])

I would not choose to do the job in this manner, though.

Answer 2

Let me try again: in which shell do you expect your syntax !(...) to work? Is it bash? Is it ksh? I have never used it, and a quick search for a corresponding bash feature led nowhere. I suspect your syntax is just wrong, which is what the error message is telling you. In that case, your problem is entirely independent form python and the subprocess module.

If a special shell you have on your system supports this syntax, you need to make sure that Python is using the same shell when invoking your command. It tells you which shell it has been using: /bin/sh. This is usually just a link to the real shell executable. Does it point to the same shell you have tested your command in?

Edit: the SO solution you referenced contains the solution in the comments:

Tip: Note however that using this pattern relies on extglob. You can enable it using shopt -s extglob (If you want extended globs to be turned on by default you can add shopt -s extglob to .bashrc)

Just to demonstrate that different shells might deal with your syntax in different ways, first using bash:

$ !(uname)
-bash: !: event not found

And then, using /bin/dash:

$ !(uname)
Linux

Answer 3

The argument to a subprocess.something method must be a list of command line arguments. Use e.g. shlex.split() to make the string be split into correct command line arguments:

import shlex, subprocess
subprocess.check_call( shlex.split("mv !(...)") )

EDIT: So, the goal is to move files/directories, with the exemption of some file(s)/directory(ies). By playing around with bash, I could get it to work like this:

mv `ls | grep -v -e '\(exclusion1\|exclusion2\)'` my_project

So in your situation that would be:

mv `ls | grep -v -e '\(myproject\|cmake-3.0.2-Darwin64-universal\)'` my_project

This could go into the subprocess.check_call(..., shell=True) and it should do what you expect it to do.