1

I am running a for loop in bash using variables which are populated from YAML file values. The variable itself contains not only the url to curl but also | jq command to parse the data, sort, and get latest version of a specific binary.

echo $version
https://api.github.com/repos/hashicorp/vault/tags | jq -r '.[].name' | sort -Vr | head -n 1 | cut -c 2-)

When I curl the whole string manually it works, and gives me version number.

curl -s https://api.github.com/repos/hashicorp/vault/tags | jq -r '.[].name' | sort -Vr | head -n 1 | cut -c 2-
1.14.2

However when passing it to curl using a variable:

curl $version
Warning: Invalid character is found in given range. A specified range MUST
Warning: have only digits in 'start'-'stop'. The server's response to this
Warning: request is uncertain.
curl 8.2.1 (x86_64-alpine-linux-musl) libcurl/8.2.1 OpenSSL/3.0.10 zlib/1.2.13 brotli/1.0.9 nghttp2/1.51.0
Release-Date: 2023-07-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets

What I tried so far is using double quotes around the adress using \ as exit characters and another set of double quotes at start and end. That did not help. I also tried using $version | $parse variable to split the curl part and parsing part but that resulted in curl: (23) Failure writing output to destination.

EDIT: Using "$version" did not work and resulted in curl: (3) URL rejected: Malformed input to a URL function. I have tried using double quotes surrounding the URL inside the variable as well.

Welo
  • 11
  • 3
  • See [BashFAQ #50](https://mywiki.wooledge.org/BashFAQ/050). Unquoted expansions only are parsed as a small subset of syntax (spaces and globs; no redirections/pipes/quotes/etc) -- this is a Good Thing; if it weren't true you couldn't handle variables with untrusted or malicious content safely in bash. – Charles Duffy Aug 31 '23 at 14:58
  • And before you follow any advice telling you to use `eval`, read [BashFAQ #48](https://mywiki.wooledge.org/BashFAQ/048) as well. – Charles Duffy Aug 31 '23 at 15:00
  • 1
    `parseVersionFromCmd() { "$@" https://api.github.com/repos/hashicorp/vault/tags | jq -r '.[].name' | sort -Vr | head -n 1 | cut -c 2-; }`, then running `parseVersionFromCmd curl` would be one way to do what you want -- see how we're using a function instead of a string in that case – Charles Duffy Aug 31 '23 at 15:01
  • @Welo, to be clear, your whole command **isn't** "curling the whole string" at all. When you run `curl -s https://api.github.com/repos/hashicorp/vault/tags | jq -r '.[].name' | sort -Vr | head -n 1 | cut -c 2-`, it calls curl with the argument list (as JSON): `["curl", "-s", "https://api.github.com/repos/hashicorp/vault/tags"]`; nothing else is actually part of the curl command; instead, `jq`, `sort`, etc. are completely independent commands and curl has no involvement in starting them. – Charles Duffy Aug 31 '23 at 15:13
  • I mean if there is a better way. Let me hear it. I can split up the values in YAML itself. But since I have to use the YAML file this was the best solution I came up with. Another way would be to 1. download the page itself using curl 2. parse it from that file, because as it seems piping it does not work. Nor does the "$version" as I mentioned in the EDIT: – Welo Aug 31 '23 at 15:13
  • @Welo, I _gave_ you better ways -- two FAQ links, a linked duplicate with an answer that shows correct technique, etc, a _specific_ `parseVersionFromCmd` function, etc. BTW, YAML is a superset of JSON, so all JSON files are also valid YAML files exactly as-is without needing to change anything; so if you're trying to do all the `cut` magic to transform JSON to YAML, you can just... not do that, and use the JSON written by jq directly as input to any valid/legal YAML parser. – Charles Duffy Aug 31 '23 at 15:14
  • @CharlesDuffy it works! I was kinda sceptical to use it since this is being run in alpine container. But it actually did just that! But can I use variables inside the parseVersionFromCmd() ? the whole point is that the $version is populated from YAML file. – Welo Aug 31 '23 at 15:19
  • "Can I use variables inside this function?" and "can I define a function from a variable?" are _completely_ different questions. If you're asking the first one _and you expect the variables to provide only data and not code_, the answer is "yes". If you're asking the second one, then you're explicitly asking for something that calls for `eval` with all the security risks that involves. – Charles Duffy Aug 31 '23 at 15:21
  • The whole point is having a values.yaml file with binaries to run through. Each one has different link and parsing mechanism to get the latest version. In this case vault cli binary so it needs to be reusable with $version variable which is populated by: `version=$(bin_name=$binary yq '.binaries[] | select(.name == env(bin_name)) | .latestVersion' ./values.yaml)` – Welo Aug 31 '23 at 15:22
  • The short, bad answer is `eval "curl $version"` -- but let's be very clear that I do not endorse or recommend any part of that. – Charles Duffy Aug 31 '23 at 15:23
  • (and btw, that short, bad answer is given in several of the duplicates I linked, so you could have been using it 25 minutes ago instead of continuing this discussion... well, hopefully a little less than 25 minutes ago if you follow the advice to read BashFAQ #48 first). – Charles Duffy Aug 31 '23 at 15:25
  • Hence why I steered clear of it :) – Welo Aug 31 '23 at 15:31
  • @CharlesDuffy I opted for using a custom function like you mentioned, that way I can actually utilize the variables without using eval. I have split the curl and parse part. Is there a way to make it 1 function ? It seems that I cannot tell it to wait for the curl to finish and then parse the output. `parseVersionFromCmd() { "$@" $link | $parse; }` the response is: `jq-- curl: (23) Failure writing output to destination` – Welo Sep 01 '23 at 07:20
  • `| $parse` has the same problem as any other `$cmd` use, which BashFAQ #50 goes into. Consider, after `$parse` is set, using `eval 'parseVersionFromCmd() { "$@" "$link" | '"$parse"'; }'` -- that way `eval` is being used only when you _define_ the function but not when you _run_ the function; you're exposed to a hostile `$parse` variable but not to a hostile value of `link` or (anything but the first argument of) `"$@"` – Charles Duffy Sep 01 '23 at 10:40
  • (btw, can your `"$@"` ever start with anything other than curl? `curl "$@"` is safer than `"$@"` as general policy) – Charles Duffy Sep 01 '23 at 10:44

0 Answers0