How to submit multiple commands to AWS Batch using Boto3?

Question

I'm trying to run multiple shell commands through Docker using AWS Batch and boto3. When I try to submit multiple commands using the & sign as follows, the job fails.

My attempt

import boto3

client = boto3.client("batch")
response = client.submit_job(
    jobName='AndrewJob',
    jobQueue='AndrewJobQueue',
    jobDefinition='AndrewJobDefinition',
    containerOverrides={
        'command': 'ls & python myjob.py'.split(),
    },
    timeout = {'attemptDurationSeconds': 100}
)
print(response)

The error is:

ls: cannot access '&': No such file or directory

According to the Docker Docs here https://docs.docker.com/engine/reference/builder/#cmd and this post here docker run <IMAGE> <MULTIPLE COMMANDS> it seems like this should be possible in shell form.

@badger0053 - yes, I've tried && and get the same error (but with '&&') — maurera, Jan 24 '20 at 22:14

score 2 · Answer 1 · answered Apr 23 '20 at 00:19

2

It appears that Batch is behaving like subprocess.Popen in that it executes the command as one command where the first argument is the command name and the remaining arguments are the command arguments. I got this to work with subprocess.Popen, so I bet it would work with Batch:

subprocess.Popen(["/bin/bash", "-c", "ls && echo \"Hello world\""])

answered Apr 23 '20 at 00:19

Nathan Collins

301
2
5

I affirm that simply "cmd1 && cmd2" will not work, even if this could be run as a **docker run** command (the AWS Batch Guide would indicate otherwise). The `command` argument I needed to pass through the Job Definition creation followed the json format ["/bin/bash", "-c", "command1; command2"] – srhoades10 Jul 25 '22 at 18:53

How to submit multiple commands to AWS Batch using Boto3?

1 Answers1