Convert log files to base64 and upload it using Curl to Github

Question

I am trying to upload a file to Github using its API.

Following code works, but only with smaller size content which is approx less than 1MB.

tar -czvf logs.tar.gz a.log b.log
base64_logs=$(base64 logs.tar.gz | tr -d \\n)

content_response=$(curl \
  -X PUT \
  -u :"$GIT_TOKEN" \
  -H "Accept: application/vnd.github.v3+json" \
  "$content_url" \
  -d '{"message": "Log files", "content": "'"$base64_logs"'"}')

For content that is a bit large, I get the following error:

/usr/bin/curl: Argument list too long

Now, there is already a question on SO about this error message, and it says that to upload a file directly. See here: curl: argument list too long

When I try this, I get a problem parsing JSON error message.

tar -czvf logs.tar.gz a.log b.log
base64_logs=$( base64 logs.tar.gz | tr -d \\ ) > base64_logs.txt

content_response=$(curl \
  -X PUT \
  -u :"$GIT_TOKEN" \
  -H "Accept: application/vnd.github.v3+json" \
  "$content_url" \
  -d '{"message": "Log files", "content": @base64_logs.txt}')

Can anyone point me out where I am making mistake here? Thanks!

`jq -n '{"message": "Log files", "content": ($content | @base64)}' --rawfile content yourfilehere` — Léa Gris, Oct 05 '21 at 10:06
replacing line with `-d $( jq -n '{"message": "Log files", "content": ($content | @base64)}' --rawfile content logs.tar.gz )` returns `argument list too long` error. — disp_name, Oct 05 '21 at 10:43
Curl is complaining argument list too long because the generated JSON length exceeds the arguments buffer length and because without double quotes around the sub-shell expansion it splits at spaces into multiple arguments. Basically: `jq -n '{"message": "Log files", "content": ($content | @base64)}' --rawfile content logs.tar.gz | curl --data-binary @- ...` — Léa Gris, Oct 05 '21 at 10:48
Seems it is uploading but when I try to open the uploaded content it gives the following message `tar: Error opening archive: Unrecognized archive format`. By any chance is it corrupting the content? — disp_name, Oct 05 '21 at 11:00
jq @base64 is only capable of encoding text, and not binary data like a gz archive. So instead of encoding base64 with Jq do it with the base64 command instead. Ill make this an answer. — Léa Gris, Oct 05 '21 at 11:02

Léa Gris · Accepted Answer · 2021-10-05T11:25:14.540

Use the base64 command rather than the @base64 filter from jq, because the later can only encode textual data and not binary data as from a .gz archive.

Pipe the base64 stream to jq to format it into a JSON data stream.

Curl will read the JSON data stream and send it.

# use base64 to encode binary data and -w0 all in one-line stream
base64 --wrap=0 logs.tar.gz |

# JSON format from raw input
jq \
  --raw-input \
  --compact-output \
  '{"message": "Log files", "content": . }' |

# Pipe JSON to curl
curl \
  --request PUT \
  --user ":$GIT_TOKEN" \
  --header "Accept: application/vnd.github.v3+json" \
  --header 'Content-Type: application/json' \
  --data-binary @- \
  --url "$content_url"

Convert log files to base64 and upload it using Curl to Github

1 Answers1

Linked