3

so currently I have a codebuild that is triggered by codepipeline when I push to my repository.

The files have a .sh file, a buildspec.yaml file, and a folder which contains 1 to n number of folders, each of which will have their own .sh file that will then deploy cloudformation resources inside the folder. 

deploy.sh
buildspec.yaml
Folder
|
|-- Directory1
|   -- deploy.sh
|   -- template.yaml
|
|-- Directory2
|   -- deploy.sh
|   -- template.yaml
|
|-- Directory3
|   -- deploy.sh
|   -- template.yaml

Currently, the first .sh file is just a for loop that will cd into each directory and run the .sh script sequentially.

I am trying to make this process to be parallel rather than sequential. Since there can be an unknown number of child folders it can potentially be a lengthy process and I don't want it waiting on previous resources.

Does anyone have and advice/experience in this area? I have done some searching but haven't found anything conclusive in my testing.

EDIT: Main build.sh file that runs the for loop

#!/bin/bash
set -e

PROFILE='new-profile'
aws configure --profile $PROFILE set credential_source EcsContainer
REGION=$AWS_REGION

for directory in *; do
    if [ -d ${directory} ]; then
        # Will not run if no directories are available
        chmod +x $directory/deploy.sh
        $directory/deploy.sh -p new-profile &
        # cd $directory
        # echo $directory
        # echo "---------------------------------------------------------------------------------"
        # echo "Start deploying $directory resources..."
        # echo "---------------------------------------------------------------------------------"
        #     chmod +x ./deploy.sh
        #     ./deploy.sh -p new-profile &
        # cd ../
        # echo "---------------------------------------------------------------------------------"
        # echo "End deploying $directory resources..."
        # echo "---------------------------------------------------------------------------------"
    fi
done
wait
echo "finished deployment"
Tyshalle
  • 33
  • 6
  • Does this answer your question? [How do you run multiple programs in parallel from a bash script?](https://stackoverflow.com/questions/3004811/how-do-you-run-multiple-programs-in-parallel-from-a-bash-script) – jordanm Apr 16 '20 at 19:31
  • I don't see anything here to indicate doing this on codebuild would be any different than anywhere else. – jordanm Apr 16 '20 at 19:32
  • I did come across this and its pretty similar to what the guy below suggested. As I said down there, the codebuild finishes "successfully" and it doesnt complete the full deployment. – Tyshalle Apr 16 '20 at 21:29

1 Answers1

0

In your main script just loop all the sub-deploy.sh and run them in background, adding & to the end of string:

$directory/deploy.sh &

At the end of your main script write

wait
echo "all sub-deploy processes are finished"
Saboteur
  • 1,331
  • 5
  • 12
  • This seems to almost do what I need, just the codebuild finishes before the child processes are finished, even with the wait at the end. – Tyshalle Apr 16 '20 at 20:44
  • are you use . or source to call subshells? Please provide your 'main' build.sh to see what exactly are you executing – Saboteur Apr 17 '20 at 00:07
  • I edited the post to show what the main build.sh has – Tyshalle Apr 17 '20 at 16:02
  • I believe the issue is "set -e" -e Exit immediately if a command exits with a non-zero status. Possibly one of your subshells returns error. Try to add more "echo" to catch where your script stops, or use "set -x" to enable debug printing of each command to be executed – Saboteur Apr 17 '20 at 19:30
  • I tried that and adding more echo statements but I still get the same result. I believe the issue is codebuild is finishing with "Succeeded" and therefore stopping the build before everything finishes. – Tyshalle Apr 20 '20 at 16:04