0

I am communicating with HDFS using curl. Procedure to interact with HDFS via webhdfs is two steps and I receive a url from a first curl command:

create_request=$(curl -i -X PUT "http:/somewhere:50070/webhdfs/v1/${path}?op=CREATE")

destination=$(echo "$create_request" | grep Location | cut -d " " -f 2)

Using the destination variable, I can upload my file with the following:

curl -i -X PUT -T $local_file_path "$destination"

However, the above commands throws :

curl: (3) URL using bad/illegal format or missing URL

Using the exact same command but changing $destination bby the URL it contains (writting it manually) works file.

Why do I have this problem and how to solve this ?

PS: $destination does contain the data: http://datanode:50075/webhdfs/v1/path/test/file.jar?op=CREATE&namenoderpcaddress=cluster&overwrite=false

Edit: Quoting

path="mypath"
local_file_path="untitled.txt"

create_request="$(curl -i -X PUT "http://ip:50070/webhdfs/v1/${path}?op=CREATE")"

destination="$(echo "$create_request" | grep "Location"| cut -d ' ' -f 2)"

curl -i -X PUT -T "$local_file_path" "$destination"

Gives the same error message.

Itération 122442
  • 2,644
  • 2
  • 27
  • 73
  • 1
    `"$local_file_path"` use quotes around variables. – MichalH Apr 22 '21 at 14:22
  • Does this answer your question? [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – 0stone0 Apr 22 '21 at 14:23
  • @MichalH Quoting does not solve the problem (see edit) – Itération 122442 Apr 22 '21 at 14:30
  • @0stone0 Quoting all the strings does not solve the problem, see edit (why can't we "ping" more than one user at a time in comments... ?) – Itération 122442 Apr 22 '21 at 14:30
  • @FlorianCastelain Then the destination must be wrong, maybe contains a trailing space? – MichalH Apr 22 '21 at 14:38
  • if you `echo ">$destination<"` does it really contain exactly the same as when you type it in (between `>` and `<`)? Not a single character extra? – Ted Lyngmo Apr 22 '21 at 14:38
  • @MichalH A trailing space is probably not it (`cut -d ' ' -f 2` shouldn't leave any spaces at all). – Ted Lyngmo Apr 22 '21 at 14:42
  • 1
    @TedLyngmo true, also Florian you can try `diff <(echo "$destination") <(echo "actual_url")`. – MichalH Apr 22 '21 at 14:46
  • The result of the diff command is: 1c1 ``< http://url:50075/webhdfs/v1/user/florian_castelain/test/?op=CREATE&namenoderpcaddress=cluster&overwrite=false --- > http://url:50075/webhdfs/v1/user/florian_castelain/test/?op=CREATE&namenoderpcaddress=cluster&overwrite=false`` Url is correct, but I am not able to check what has changed... @MichalH – Itération 122442 Apr 22 '21 at 14:50
  • @FlorianCastelain Great - that log explains it. I made an answer. – Ted Lyngmo Apr 22 '21 at 14:54

1 Answers1

2

You get a \r (carriage return) back in $destination. You can remove it with tr -d '\r'

destination=$(echo "$create_request" | grep Location | cut -d " " -f 2 | tr -d '\r')
Ted Lyngmo
  • 93,841
  • 5
  • 60
  • 108
  • Wow. Right. Did you understand that from the diff result ? Is carriage return the only possible reason that the diff says there is a difference while not showing it ? – Itération 122442 Apr 22 '21 at 14:56
  • 1
    @FlorianCastelain I actually misread the output :-) I thought the diff output was the raw output from my `>$destination<` suggestion, but yes, it explains it because `>dest<` would show up as ` – Ted Lyngmo Apr 22 '21 at 14:57