14

How can a string be urlencoded and embedded into the URL? Please note that I am not trying to GET or POST data, so the -G and --data and --data-urlencode options of curl don't seem to do the job.

For example, if you used 

curl -G http://example.com/foo --data-urlencode "bar=spaced data"

that would be functionally equivalent to 

curl http://example.com/foo?bar=spaced%20data"

which is not desired.

I have a string foo/bar which must be urlencoded foo%2fbar and embedded into the URL.

curl http://example.com/api/projects/foo%2fbar/events

One hypothetical solution (if I could find something like this) would be to preprocess the data in bash, if there exists some kind of urlencode function. 

DATA=foo/bar
ENCODED=`urlencode $DATA`
curl http://example.com/api/projects/${ENCODED}/events

Another hypothetical solution (if I could find something like this) would be some switch in curl, similar to this:

curl http://example.com/api/projects/{0}/events --string-urlencode "0=foo/bar"

The specific reason I'm looking for an answer to this question is the Gitlab API. For example, gitlab get single project NAMESPACE/PROJECT_NAME is URL-encoded, eg. /api/v3/projects/diaspora%2Fdiaspora (where / is represented by %2F). Further to this, you can request individual properties in the project, so you end up with a URL such as http://example.com/projects/diaspora%2Fdiaspora/events

Although this question is gitlab-specific, I imagine it's generally applicable to REST API's in general, and I'm surprised I can't find a pre-existing answer on stackoverflow or internet search.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Edward Ned Harvey
  • 6,525
  • 5
  • 36
  • 45
  • 1
    `urlencode $DATA` is going to behave badly if `DATA='*'` (it would expand the glob, encoding a list of filenames); needs to be `"$DATA"`. Also, see http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html guidelines re: variable names (fourth paragraph): Shell and system tools use all-upper-case names for variables that impact their operation; names with at least one lower-case character are reserved for application use. Since environment variables and shell variables share a namespace, this applies to regular (non-exported) names as well. – Charles Duffy May 18 '16 at 20:37
  • BTW, did you try the answer by Orwellophile in the not-quite-dupe-but-closely-related question at http://stackoverflow.com/a/10660730/14122? – Charles Duffy May 18 '16 at 20:44
  • Seems pretty close to this question: http://stackoverflow.com/questions/29755942/possible-to-urlencode-a-variable-in-a-shell-script has among others also a perl solution that you can craft into a function. –  May 19 '16 at 00:46
  • So far yes, I'm able to find an implementation of `urlencode()` (found [here](https://gist.github.com/cdown/1163649)) that can be pasted into a bash script. It's a solution, but I'm doubtful it's the best solution. – Edward Ned Harvey May 19 '16 at 13:23
  • this answer is a one liner for it: https://stackoverflow.com/a/10797966/1839558 – lauksas Aug 13 '20 at 15:35

1 Answers1

14

The urlencode function you propose is easy enough to implement:

urlencode() {
  python -c 'import urllib, sys; print urllib.quote(sys.argv[1], sys.argv[2])' \
    "$1" "$urlencode_safe"
}

...used as:

data=foo/bar
encoded=$(urlencode "$data")
curl "http://example.com/api/projects/${encoded}/events"

If you want to have some characters which are passed through literally -- in many use cases, this is desired for /s -- instead use:

encoded=$(urlencode_safe='/' urlencode "$data")
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441