1

I have a list of URL files Test.txt in a text document I am trying to download from a server that requires logging in. I start by creating a .netrc file in curl in my Linux terminal using the following code:

host-0800-a45e60ef38c9:~ myname$ touch .netrc
host-0800-a45e60ef38c9:~ myname$ echo "machine urs.earthdata.nasa.gov login mylogin password mypassword” >> .netrc
>
host-0800-a45e60ef38c9:~ myname$ chmod 0600 .netrc
host-0800-a45e60ef38c9:~ myname$ touch .urs_cookies
host-0800-a45e60ef38c9:~ myname$ cat Test.txt | tr -d '\r' | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

Where ~myname$ is my home directory.

The Test.txt file contains around a thousand unique links such as this one:

https://hydro1.gesdisc.eosdis.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2FNCALDAS%2FNCALDAS_NOAH0125_D.2.0%2F1979%2F01%2FNCALDAS_NOAH0125_D.A19790102.002.nc4&FORMAT=bmM0Lw&BBOX=42.472%2C-71.499%2C48.185%2C-67&LABEL=NCALDAS_NOAH0125_D.A19790102.002.nc4.SUB.nc4&SHORTNAME=NCALDAS_NOAH0125_D&SERVICE=L34RS_LDAS&VERSION=1.02&DATASET_VERSION=2.0&VARIABLES=Evap%2CSoilMoist0_10cm%2CSoilMoist10_40cm%2CSoilMoist100_200cm%2CSoilMoist40_100cm

I get the following errors over and over again:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1207k  100 1207k    0     0  1641k      0 --:--:-- --:--:-- --:--:-- 1641k
Warning: Failed to create the file
curl: (23) Failed writing body (0 != 27)

My guess it has something to do with the permission of credentials .netrc file. But it is set to read and write. I am brand new to Linux. Can someone point out what's going wrong here?

Alyssa.r
  • 137
  • 1
  • 10
  • 2
    Hard to say, not knowing what's in `Test.txt`. – tink Oct 30 '19 at 16:12
  • 1
    agree with above. Please edit your Q to show one URL that isn't working. And does `~ myname` indicate that you are in your `$HOME` dir? (presumably you can write files there). I think there is a debug option for curl, maybe `-v` or `-vvv` where you may be able to confirm exactly where `curl` is trying to write the file (never assume anything when debugging!) ;-) . Good luck. – shellter Oct 30 '19 at 16:17
  • Can you try it like this? `tr -d '\r' < Test.txt | cat | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies` – lainatnavi Oct 30 '19 at 16:23
  • Thanks @tink I've made the edits, and it appears that none of the URLs work (I get the same errors for each one). – Alyssa.r Oct 30 '19 at 16:30
  • @lainatnavi that seems to return the same errors as above :( – Alyssa.r Oct 30 '19 at 16:30
  • It is the `-O` option to make it fail but still I don't understand why. – lainatnavi Oct 30 '19 at 17:22
  • @lainatnavi: `-O` AND `-J` together cause curl to write to the filename given in the response Content-Disposition header. Alyssa: add `-v` and look at Content-Disposition to see if that specifies a valid place for you to write. – dave_thompson_085 Oct 30 '19 at 18:29
  • 1
    Haven't read thru it in detail, but this might provide a couple of hints... [Why CURL return and error (23) Failed writing body?](https://stackoverflow.com/questions/16703647/why-curl-return-and-error-23-failed-writing-body). While it's tagged `macos`, it's pertinent since it's tagged and clearly concerns `bash` and `curl`. – David Yockey Oct 30 '19 at 18:45
  • In my case curl is giving this error when using the `-O` option and passing the URL parameter with xargs or the `-K` option. The Content-Disposition filename is being written when not passing the URL with xargs. So this works `curl -O ` and this doesn't `xargs -n 1 curl -O < urls.txt`. This does not work either `curl -OK urls.txt`. – lainatnavi Oct 31 '19 at 10:48
  • This line of code: `curl -n -c ~/.urs_cookies -b ~/.urs_cookies -LJO --url ` works for each individual URL, the problem is with the list of files in the Test.txt document. There is a valid place to right the file as this works so the problem seems to be with xargs. Any ideas? – Alyssa.r Oct 31 '19 at 15:34
  • @Alyssa.r: cannot try it right now but I was thinking about alternative solutions. For example you could loop on the lines of the file and call curl on each iteration. Something like `for url in $(cat Test.txt); do ... $url; done` . give it a try. – lainatnavi Oct 31 '19 at 18:33

0 Answers0