Why can I not override container variables for multi-node parallel jobs in AWS Batch C++ SDK?

Question

I am working with AWS Batch. My goal is to create a multi-node parallel job through the AWS SDK for C++. For this, I have created a job definition as per the instructions here.

I am working with the AWS C++ SDK, and I noticed that when I try to override either environmental variables or commands, nothing is actually transferred to the job.

Interestingly, the same code works perfectly fine for a job which uses a normal job definition (as opposed to a multi-node one):

#include <aws/batch/BatchClient.h>
#include <aws/batch/model/ContainerOverrides.h>
#include <aws/batch/model/KeyValuePair.h>
#include <aws/batch/model/SubmitJobRequest.h>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>

int main(void)
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    Aws::Batch::BatchClient batchClient;
    Aws::Batch::Model::SubmitJobRequest submitJobRequest;
    Aws::Batch::Model::SubmitJobOutcome submitJobOutcome;
    Aws::Batch::Model::ContainerOverrides containerOverrides;
    Aws::Batch::Model::KeyValuePair envVariable;

    envVariable.SetName("foo");
    envVariable.SetValue("bar");

    containerOverrides.AddEnvironment(envVariable); // This does nothing for a multi-node job definition.
    containerOverrides.AddCommand("foobarbaz"); // This does nothing for a multi-node job definition.

    submitJobRequest.SetJobName("myjob");
    submitJobRequest.SetJobDefinition("arn:aws:...."); // This string is an example. I have used the actual job definition ARN.
    submitJobRequest.SetJobQueue("arn:aws:...."); // This string is an exmaple. I have used the actual queue ARN.
    submitJobRequest.SetContainerOverrides(containerOverrides);

    submitJobOutcome = batchClient.SubmitJob(submitJobRequest);

    Aws::ShutdownAPI(options);

    return 0;

}

Should I be using a different API for multi-node parallel jobs?

Were you able to run the multi-node job successfully without using the C++ SDK? — Avin Kavish, Jun 09 '19 at 05:06

score 0 · Answer 1 · answered Jun 09 '19 at 06:17

Instead of setting the top-level container overrides in the request, try setting the container overrides inside of the node overrides of the request. Node overrides has a vector of node property override which have your container overrides including environment variables. In the node property overrides, you need to specify the name of the node you are targeting.

I am not really very good with C++ so I can't get you an exact code example but the JSON for the API that the SDK is calling looks something like this:


POST /v1/submitjob HTTP/1.1
Content-type: application/json

{
   "jobDefinition": "string",
   "jobName": "string",
   "jobQueue": "string",
   "nodeOverrides": { 
      "nodePropertyOverrides": [ 
         { 
            "containerOverrides": { 
               "command": [ "string" ],
               "environment": [ 
                  { 
                     "name": "string",
                     "value": "string"
                  }
               ],
               "instanceType": "string",
               "memory": number,
               "resourceRequirements": [ 
                  { 
                     "type": "string",
                     "value": "string"
                  }
               ],
               "vcpus": number
            },
            "targetNodes": "string"
         }
      ],
      "numNodes": number
   },
   # lots of other stuff
}

Hope that helps.

Why can I not override container variables for multi-node parallel jobs in AWS Batch C++ SDK?

1 Answers1