gcloud beta lifesciences, JSON pipeline-file rather than options

Question

I am trying to run gcloud beta lifesciences because genomics API is deprecated. There have been so many changes, genomics API vs lifesciences API.

I ran one of my analysis step in google clooud using beta lifesciences. Here is what I found. (1) wildcard is not working in command line options (2) It is not easy to set the target directory in command line option, I used env-var for copy.

I am now trying to convert commandline option into JSON format pipeline-file, but it is not easy to understand help page in google cloud. Do you have an idea how to convert following options into JSON file, so I could run it with simpler option?

I used YAML formatted pipeline file in genomics API, but beta lifescienes is totally different.

$ more step03_bwa_mem_genome1.run 
#SMALL=
SMALL=chr21.

LIFESCIENCESPATH=/gcloud-shared
#LIFESCIENCESPATH=/mnt
SCRIPTFILENAME=step03_bwa_mem_genome.sh
COHORTID=2_C_222

gcloud beta lifesciences pipelines run \
    --logging gs://${BUCKETID}/ExomeSeq/hResults/step03_bwa_mem_genome.${COHORTID}.log \
    --regions=asia-northeast1,asia-northeast2,asia-northeast3,asia-east1,asia-east2,asia-south1 \
    --boot-disk-size 20 \
    --preemptible \
    --machine-type n1-standard-1 \
    --disk-size "gcloud-shared:10" \
    --docker-image asia.gcr.io/thermal-shuttle-199104/centos8-essential-software-genomics-custom-python3:0.4 \
    --inputs REFERENCE1=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.amb \
    --inputs REFERENCE2=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.ann \
    --inputs REFERENCE3=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.bwt \
    --inputs REFERENCE4=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.fai \
    --inputs REFERENCE5=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.intervals \
    --inputs REFERENCE6=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.pac \
    --inputs REFERENCE7=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.sa \
    --inputs SCRIPTFILE=gs://${BUCKETID}/ExomeSeq/${SCRIPTFILENAME} \
    --inputs COHORTID=${COHORTID} \
    --inputs SAMPLELIST=gs://${BUCKETID}/ExomeSeq/SAMPLELIST.${COHORTID}.lst \
    --inputs INPUTFILE1=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_01_1.chr21.fastq.gz \
    --inputs INPUTFILE2=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_01_2.chr21.fastq.gz \
    --inputs INPUTFILE3=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_02_1.chr21.fastq.gz \
    --inputs INPUTFILE4=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_02_2.chr21.fastq.gz \
    --inputs INPUTFILE5=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_03_1.chr21.fastq.gz \
    --inputs INPUTFILE6=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_03_2.chr21.fastq.gz \
    --outputs OUTPUTFILE1=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_01.bam \
    --outputs OUTPUTFILE2=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_02.bam \
    --outputs OUTPUTFILE3=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_03.bam \
    --env-vars REFERENCE1=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.amb,REFERENC
E2=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.ann,REFERENCE3=${LIFESCIENCESPATH}/
ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.bwt,REFERENCE4=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRC
h38.primary_assembly.genome.${SMALL}fa.fai,REFERENCE5=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.ge
nome.${SMALL}fa.intervals,REFERENCE6=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.p
ac,REFERENCE7=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.sa,SCRIPTFILE=${LIFESCIE
NCESPATH}/ExomeSeq/${SCRIPTFILENAME},SAMPLELIST=${LIFESCIENCESPATH}/ExomeSeq/SAMPLELIST.${COHORTID}.lst,INPUTFILE1=${LIF
ESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_01_1.chr21.fastq.gz,INPUTFILE2=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID
}_01_2.chr21.fastq.gz,INPUTFILE3=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_02_1.chr21.fastq.gz,INPUTFILE4=${LIFESC
IENCESPATH}/ExomeSeq/hReads/${COHORTID}_02_2.chr21.fastq.gz,INPUTFILE5=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_0
3_1.chr21.fastq.gz,INPUTFILE6=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_03_2.chr21.fastq.gz,OUTPUTFILE1=${LIFESCIE
NCESPATH}/ExomeSeq/hResults/${COHORTID}_01.bam,OUTPUTFILE2=${LIFESCIENCESPATH}/ExomeSeq/hResults/${COHORTID}_02.bam,OUTP
UTFILE3=${LIFESCIENCESPATH}/ExomeSeq/hResults/${COHORTID}_03.bam \
    --command-line="find ${LIFESCIENCESPATH}; /bin/bash ${LIFESCIENCESPATH}/ExomeSeq/${SCRIPTFILENAME} ${COHORTID} 4"

score 1 · Answer 1 · answered Feb 25 '21 at 22:59

Thank you for your answer!

Finally, I made an input YML file for gcloud lifesciences from operation descriptions. I needed to understand basic functionality of gcloud lifesciences because I want to make full version of genome analysis pipeline, from FASTQ to snpEff/third party annotation/text extraction in google cloud (whole-exome in a day, whole-genome in 5 days for a single set). I already made it using genomics API, but am trying to upgrade it to use gcloud lifesciences.

I also tried gcsfuse, but setting up google cloud authentication is a little bit tricky.

Thanks,

$ more step03_bwa_mem_genome2.run 
#SMALL=
SMALL=chr21.

LIFESCIENCESPATH=/gcloud-shared
#LIFESCIENCESPATH=/mnt
SCRIPTFILENAME=step03_bwa_mem_genome.sh
COHORTID=2_C_222

gcloud beta lifesciences pipelines run \
    --logging gs://${BUCKETID}/ExomeSeq/hResults/step03_bwa_mem_genome.${COHORTID}.log \
    --pipeline-file step03_bwa_mem_genome1.yml


$ cat step03_bwa_mem_genome1.yml
actions:
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_03_2.chr21.fastq.gz ${INPUTFILE6}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_03_1.chr21.fastq.gz ${INPUTFILE5}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_02_2.chr21.fastq.gz ${INPUTFILE4}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_02_1.chr21.fastq.gz ${INPUTFILE3}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_01_2.chr21.fastq.gz ${INPUTFILE2}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_01_1.chr21.fastq.gz ${INPUTFILE1}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/SAMPLELIST.2_C_222.lst ${SAMPLELIST}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/step03_bwa_mem_genome.sh ${SCRIPTFILE}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.sa ${REFERENCE7}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.pac ${REFERENCE6}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.intervals ${REFERENCE5}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.fai ${REFERENCE4}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.bwt ${REFERENCE3}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.ann ${REFERENCE2}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.amb ${REFERENCE1}
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - -c
  - find /gcloud-shared; /bin/bash /gcloud-shared/ExomeSeq/step03_bwa_mem_genome.sh 2_C_222 4
  entrypoint: bash
  imageUri: asia.gcr.io/thermal-shuttle-199104/centos8-essential-software-genomics-custom-python3:0.4
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp ${OUTPUTFILE1} gs://genconv1/ExomeSeq/hResults/2_C_222_01.bam
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp ${OUTPUTFILE2} gs://genconv1/ExomeSeq/hResults/2_C_222_02.bam
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp ${OUTPUTFILE3} gs://genconv1/ExomeSeq/hResults/2_C_222_03.bam
  imageUri: google/cloud-sdk:slim
  mounts:
  - disk: gcloud-shared
    path: /gcloud-shared
- alwaysRun: true
  commands:
  - /bin/sh
  - -c
  - gsutil -m -q cp /google/logs/output gs://genconv1/ExomeSeq/hResults/step03_bwa_mem_genome.2_C_222.log
  imageUri: google/cloud-sdk:slim
environment:
  COHORTID: 2_C_222
  INPUTFILE1: /gcloud-shared/ExomeSeq/hReads/2_C_222_01_1.chr21.fastq.gz
  INPUTFILE2: /gcloud-shared/ExomeSeq/hReads/2_C_222_01_2.chr21.fastq.gz
  INPUTFILE3: /gcloud-shared/ExomeSeq/hReads/2_C_222_02_1.chr21.fastq.gz
  INPUTFILE4: /gcloud-shared/ExomeSeq/hReads/2_C_222_02_2.chr21.fastq.gz
  INPUTFILE5: /gcloud-shared/ExomeSeq/hReads/2_C_222_03_1.chr21.fastq.gz
  INPUTFILE6: /gcloud-shared/ExomeSeq/hReads/2_C_222_03_2.chr21.fastq.gz
  OUTPUTFILE1: /gcloud-shared/ExomeSeq/hResults/2_C_222_01.bam
  OUTPUTFILE2: /gcloud-shared/ExomeSeq/hResults/2_C_222_02.bam
  OUTPUTFILE3: /gcloud-shared/ExomeSeq/hResults/2_C_222_03.bam
  REFERENCE1: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.amb
  REFERENCE2: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.ann
  REFERENCE3: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.bwt
  REFERENCE4: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.fai
  REFERENCE5: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.intervals
  REFERENCE6: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.pac
  REFERENCE7: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.sa
  SAMPLELIST: /gcloud-shared/ExomeSeq/SAMPLELIST.2_C_222.lst
  SCRIPTFILE: /gcloud-shared/ExomeSeq/step03_bwa_mem_genome.sh
resources:
  regions:
  - asia-northeast1
  - asia-northeast2
  - asia-northeast3
  - asia-east1
  - asia-east2
  - asia-south1
  virtualMachine:
    bootDiskSizeGb: 20
    disks:
    - name: gcloud-shared
      sizeGb: 10
    machineType: n1-standard-1
    preemptible: true

Thank you for this. I was looking for a similar example especially dealing with a complex pipeline that needs multiple files from and to cloud storage. — ssm, Jul 16 '21 at 16:55

score 0 · Answer 2 · answered Feb 23 '21 at 13:18

I would generally recommend using something like Cromwell, Nextflow, or Snakemake rather than using either API directly. They have more built in functionality for these types of tasks.

However, the output from gcloud beta lifesciences operations describe <operation name> will include the pipeline definition that gcloud created, which could be used as a starting point. One thing you'll notice there is that --inputs and --outputs automatically create environment variables so the LIFESCIENCESPATH variable and the --env-vars parameter are unnecessary which will simplify the command line significantly.

gcloud beta lifesciences, JSON pipeline-file rather than options

2 Answers2