7

I'm programming on Windows 7 and in one of my Python projects I need to call bedtools, which only works with Cygwin on Windows. I'm new to Cygwin, installed the default version + everything needed for bedtools and then used Cygwin to install bedtools by using make as described in the installation instructions.

$ tar -zxvf BEDTools.tar.gz
$ cd BEDTools-<version>
$ make

When I use the Cygwin terminal to call it manually like below, it works without problem and the output file contains the correct result.

bedtools_exe_path intersect -a gene_bed_file -b snp_bed_file -wa -wb > output_file

But when I use subprocess.call in my program it seems to use Windows cmd instead of Cygwin, which doesn't work.

arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
             snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)

Results in no output file and a return code of 3221225781.

arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
             snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)

Results in an empty output file and a return code of 3221225781.

cygwin_bash_path = 'D:/Cygwin/bin/bash.exe'
arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
             snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)

Results in no output file, a return code of 126 and

D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file

arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
             snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)

Results in an empty output file, a return code of 126 and

D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file

Any ideas how I can get it to work?

Georgy
  • 12,464
  • 7
  • 65
  • 73
Niimiitz
  • 81
  • 1
  • 8
  • 1
    try http://pastebin.com/Q4DZcpA7. You cannot use `'>'` with shell=False. 126 means a permission problem or the file is not executable – Padraic Cunningham Jul 04 '15 at 13:45
  • If you want stderr redirected to the file also use `check_call(arguments,stdout=f,stderr=sunprocess.STDOUT)` – Padraic Cunningham Jul 04 '15 at 13:51
  • Thank you for your quick comment, unfortunately it produces the same result as the last example (empty output file, return code 126, bedtools.exe: cannot execute binary file). Why can't I use `'>'`? Edit: When adding `stderr=subprocess:STDOUT` the output file is not empty but contains only the _bedtools.exe: cannot execute binary file_ line. – Niimiitz Jul 04 '15 at 13:57
  • Are you running this from cygwin? Your output file is empty because the code errors so the output is through stderr not stdout – Padraic Cunningham Jul 04 '15 at 14:00
  • I run the code in PyCharm. – Niimiitz Jul 04 '15 at 14:05
  • Also using shell=True you would pass a single string of args string – Padraic Cunningham Jul 04 '15 at 14:45
  • Using `check_call([cygwin_bash_path,"ls"])` writes _/usr/bin/bash: ls: No such file or directory_ into the output file. – Niimiitz Jul 04 '15 at 15:03
  • That produces an empty output file, return code 127 and _/usr/bin/bash: ls: No such file or directory_ – Niimiitz Jul 04 '15 at 16:18
  • @PadraicCunningham That happens with and without shell=True, with and without passing arguments as a string or as a list. – Niimiitz Jul 04 '15 at 16:54
  • @eryksun Not invoking bash is what I tried first and it doesn't work because then it's Windows cmd which doesn't work with bedtools. When I try your example I get the same result as in my first example, only that I get an empty output file. – Niimiitz Jul 04 '15 at 16:56
  • That results in `/cygdrive/d/BEDTools/bin/bedtools.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory`. – Niimiitz Jul 04 '15 at 18:53
  • When I use the path to mintty.exe of cygwin, instead of the path to the bash.exe, the cygwin console opens with the path to my project folder but does not execute bedtools. – Niimiitz Jul 04 '15 at 18:55
  • I added D:\Cygwin\bin to the system PATH environment when I installed it. – Niimiitz Jul 04 '15 at 20:14

2 Answers2

1

Imagine you want to run a Linux command from Windows. You could install Linux into a VM and run commands via ssh (Putty/plink on Windows):

#!/usr/bin/env python
import subprocess

cmd = [r'C:\path\to\plink.exe', '-ssh', 'user@vm_host', '/path/to/bedtools']
with open('output', 'wb', 0) as file:
    subprocess.check_call(cmd, stdout=file)

Cygwin provides run command that allows to run commands directly:

cmd = [r'C:\cygwin\path\to\run.exe', '-p', '/path/to/', 'bedtools', 
        '-wait', 'arg1', 'arg2']

Note: Python script is run from Windows in both cases. bedtools is Linux or Cygwin (non-Windows) command here and therefore you should provide POSIX paths.

jfs
  • 399,953
  • 195
  • 994
  • 1,670
  • @J.F.Sebastian I used the 2nd one in several variations but it still does not work. `cmd = [cygwin_run_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b', 'snp_bed_file', '-wa', '-wb']` results in return_code 0 and I can see the console flash for a millisecond, but the output file is still empty, everything else results in return code 1. If possible I'd prefer not to install a VM. – Niimiitz Jul 04 '15 at 20:21
  • @eryksun passing it as a list or as a string gives the same results, I try every suggestion both ways. Since I use the same path formats I used in the cygwin terminal and it worked there, the paths shouldn't be the problem. – Niimiitz Jul 04 '15 at 20:25
  • @eryksun: `man run`. You could use [this question as a reference](http://stackoverflow.com/q/673278/4279) – jfs Jul 04 '15 at 20:44
  • @MarkusH: could you run anything else e.g., `echo hello world` instead of `bedtools` and see whether the file has `hello world` in it – jfs Jul 04 '15 at 20:46
  • As I said, I tried several variations, the `-wait` did not work. Using the cygwin terminal for `echo hello world` works, using run to do the same via the windows console produces an empty file and other commands do not work that way either. Neither does it work in my program. – Niimiitz Jul 04 '15 at 21:13
  • @MarkusH: if there is no `echo.exe` then run `sh -c "echo hello world"` – jfs Jul 04 '15 at 21:23
  • Thank you. `args = 'sh -c \"' + bedtools_path + ' intersect -a ' + gene_bed_file + ' -b ' + snp_bed_file + ' -wa -wb\"'` did the trick when using `args` for `subprocess.check_call(args, stdout=output_file)` – Niimiitz Jul 04 '15 at 22:15
  • @eryksun it works with `sh` and `bash`. I use paths in the following format `D:/BEDTools/bin/` that work both in Windows cmd and cygwin, cygwin does change them to `/cygdrive/d/BEDTools/bin/` though. It also works with and without escaping `"`. – Niimiitz Jul 05 '15 at 12:07
  • @eryksun: there is no escaping: `len('\"') == 1` i.e., `'\"' == '"'` in Python – jfs Jul 05 '15 at 15:47
  • For this to work for me I had to run my script with a Run as Administrator windows shortcut. – Noumenon Mar 03 '19 at 16:06
1

The following works without a problem. The " does not need to be escaped.

argument = 'sh -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file + 
           ' -b ' + snp_bed_file + ' -wa -wb\"'

with open(output_file, 'w') as file:
    subprocess.call(argument, stdout=file)

Using the following works as well:

argument = 'bash -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file + 
           ' -b ' + snp_bed_file + ' -wa -wb\"'

with open(output_file, 'w') as file:
    subprocess.call(argument, stdout=file)

With:

bedtools_exe_path = 'D:/BEDTools/bin/bedtools.exe'
gene_bed_file = 'output/gene.csv'
snp_bed_file = 'output/snps.csv'
output_file = 'output/intersect_gene_snp.bed'

Using the path to the cygwin bash.exe (D:/Cygwin/bin/bash.exe) instead of bash or sh does not work.

Thank you, eryksun, Padraic Cunningham and J.F. Sebastian.

Niimiitz
  • 81
  • 1
  • 8
  • you could write: `'sh -c "{} intersect -a {} -b {} -wa -wb"'.format(*map(pipes.quote, [bedtools_exe_path, gene_bed_file, snp_bed_file]))` to support POSIX paths with spaces (or other `sh`'s metacharacters). – jfs Jul 05 '15 at 23:53