Supply binary content inside a JSON string via curl

Question

I'm trying to use the Yandex.Cloud function API from bash and cURL.

In order to do that, I'm composing a JSON like this:

#Make header
IAM_TOKEN="myToken"
HEADER="Authorization: Bearer ${IAM_TOKEN}"

#Make data JSON
DATA=""
for ELEMENT in \
  "'"\
  '{"runtime":"dotnetcore31", "entrypoint": "Function.Handler",'\
  '"resources":{"memory":"134217728"}, "content": "PLACEHOLDERFORCONTENT",'\
  '"ServiceAccountId":"accId", '\
  '"function_id": "funcId"}'\
  "'"; do
  DATA+="${ELEMENT}"
done

Then, I need to supply content of the deployment package inside JSON, as API suggests:

In my case, it is a zip archive with function code. So, I read the file as bytes and inject the result into the JSON string. And send the request.

#Read file content
FILE_LOC="/Users/Constantine/Downloads/archiveName.zip"
FILE_BYTES=`(xxd -b ${FILE_LOC}) | base64`

#Inject content into JSON
DATA=${DATA/PLACEHOLDERFORCONTENT/$FILE_BYTES}

#Send the request
eval "$(echo curl -H \"$HEADER\" --data-binary $DATA https://serverless-functions.api.cloud.yandex.net/functions/v1/versions)"

Here, I'm getting the -bash: /usr/bin/curl: Argument list too long error. I suspect this happens because cURL interprets the binary content as the filename to read, but I'm not sure how to resolve this.

How can I supply binary content from a file into a JSON string?

JSON is a text format; strings are Unicode by definition. There is no facility for binary data in JSON itself. A common workaround is to convert your data to base64, and unwrap it in all functions which consume it. Another is to switch from JSON to a format which supports your use case. Perhaps look at protobuffers. But here, the API simply seems to be poorly defined. — tripleee, Nov 05 '22 at 11:41

Poshi · Accepted Answer · 2022-11-06T13:34:32.640

You are sending the request using a cURL call under bash. And the shell have a maximum line length that you are largely surpassing. Your easiest workaround is to save your JSON data to a file and supply cURL with a pointer to this file, so that the line length remains contained:

#Make data JSON
data_file="data.json"
> "${data_file}"
for ELEMENT in \
  "'"\
  '{"runtime":"dotnetcore31", "entrypoint": "Function.Handler",'\
  '"resources":{"memory":"134217728"}, "content": "PLACEHOLDERFORCONTENT",'\
  '"ServiceAccountId":"accId", '\
  '"function_id": "funcId"}'\
  "'"; do
  echo "${ELEMENT}"
done >> "${data_file}"

#Read file content
FILE_LOC="/Users/Constantine/Downloads/archiveName.zip"
FILE_BYTES=$(xxd -b "${FILE_LOC}" | base64)

#Inject content into JSON
sed -i "s/PLACEHOLDERFORCONTENT/${FILE_BYTES}/" "${data_file}"

#Send the request
eval "$(echo curl -H \"$HEADER\" --data-binary "@${data_file}" https://serverless-functions.api.cloud.yandex.net/functions/v1/versions)"

Besides that, your code seems quite complex for the duty. You could simplify it a little. Avoid loops, create the file in one go and avoid eval:

#Make data JSON
data_file="data.json"
FILE_LOC="/Users/Constantine/Downloads/archiveName.zip"

cat <<-EOF > "${data_file}"
  {
    "runtime": "dotnetcore31", 
    "entrypoint": "Function.Handler", 
    "resources": {
      "memory": "134217728"
    }, 
    "content": "$(xxd -b "${FILE_LOC}" | base64)",
    "ServiceAccountId": "accId", 
    "function_id": "funcId"
  }
EOF

#Send the request
curl -H "$HEADER" --data-binary "@${data_file}" https://serverless-functions.api.cloud.yandex.net/functions/v1/versions

Thanks! Please remove the apostrophes before the initial and after the closing curly braces. Otherwise it throws "Root element must be a message" error. — BusinessAlchemist, Nov 05 '22 at 18:24

Supply binary content inside a JSON string via curl

1 Answers1