1
process = subprocess.Popen(F"aws s3 sync {s3_base_dir} ./s3_files/{experiment_name}/{experiment_id}", stdout=subprocess.PIPE, shell=True)
for line in iter(process.stdout.readline, b''):
    sys.stdout.buffer.write(line)

After running the above command, nothing gets outputted... I think perhaps because the stdout is a live progress bar and not lines? For context the stdout when running the above command in terminal looks like this:

bash$ aws s3 sync s3://s3_path ./s3_files/2
Completed 14.5 MiB/513.9 MiB (10.1 MiB/s) with 1 file(s) remaining
user3180
  • 1,369
  • 1
  • 21
  • 38
  • I haven't tested but maybe aws prints to stderr instead of stdout. Have you tried redirecting 2&1 ? – Munkeh Jan 06 '19 at 04:46
  • 1
    It doesn't answer your question, but as you're using Python you shouldn't call the AWS CLI, but instead use `boto3`, the AWS SDK for Python to interact with AWS. – Dunedan Jan 06 '19 at 05:17
  • @Dunedan do you have a boto3 replacement of the above "aws s3 sync" command? – user3180 Jan 06 '19 at 08:03
  • https://stackoverflow.com/questions/31918960/boto3-to-download-all-files-from-a-s3-bucket I looked here but the answers seem complicated or incomplete (not replicating file structure when copied) – user3180 Jan 06 '19 at 08:03

0 Answers0