1

I have been trying to download a set of files from NASA's website https://search.earthdata.nasa.gov/search. I was able to sign in and request the file but I get error. 

Succesfully logged in
.
.
.
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized

I tried requests, wget, urllib2 

import requests
import os
import urllib2

destination = r'C:\PROJECTS\ShallowGW\MODIS\\'
os.chdir(destination)

session_req = requests.session()

home_url = 'https://urs.earthdata.nasa.gov/home'
response0 = session_req.get(home_url)

login_url = 'https://urs.earthdata.nasa.gov/login'
login_data = {'authenticity_token': '????',
          'username': '*******',
          'password': '*******'}

response1 = session_req.post(login_url, data=login_data, 
headers=dict(referer=login_url))

try:
    response1.raise_for_status()
    print("Succesfully logged in")
except Exception as e:
    print('Login failed')

response3 = session_req.get('https://search.earthdata.nasa.gov/search')
filepath = r'C:\PROJECTS\ShallowGW\MODIS\MODIS_LST_Links_2016.txt'

with open(filepath) as f_obj:
    for line in f_obj:
        urllib2.urlopen(line)
        print(line)

I do not understand why it says I logged in then gives 401 HTTP ERROR. Text file has the links (e.g., https://e4ftl01.cr.usgs.gov//MODV6_Dal_E/MOLT/MOD11A1.006/2016.06.30/MOD11A1.A2016182.h12v04.006.2016241041516.hdf) to download the file. I can manually download them with the same line if I am signed in but it does not work with python. It is supposed to go through the loop and download each file. Please help!

Amadeus
  • 157
  • 10

2 Answers2

0

It looks like NASA wants you to include your API key in the request parameters (based on this webpage).

apiKey = # YOUR API KEY HERE
requestParams = {'api_key':apiKey}

response = requests.get('https://search.earthdata.nasa.gov/search', params = requestParams)

There are other request parameters listed here which might be useful. I hope this helps!

Philippa Richter
  • 53
  • 7
  • Ricther Thank you! Do you have any idea where to find my API key? – Amadeus Jul 01 '19 at 14:47
  • You'll have to sign up for an account, the API key is private and user-specific. It's free though, and normally not too much of a hassle! Here's the link: https://api.nasa.gov/index.html#apply-for-an-api-key – Philippa Richter Jul 01 '19 at 16:51
  • @Amadeus Just following up, did including the API key solve the problem? – Philippa Richter Jul 06 '19 at 13:37
  • @Ricther I could not figure out with using python but NASA has good instructions to download data via its own tools. It worked for me. Thank you! I will post the website/tool info when I get chance – Amadeus Jul 10 '19 at 15:32
  • I could not work it out with python but I used NASA's tool Daac2Disk_win and it worked perfectly. Here is the link https://lpdaac.usgs.gov/documents/202/DAAC2DiskUserGuide_QSeQHbQ.pdf Just follow the instructions closely. – Amadeus Jul 10 '19 at 22:08
0

In my case it worked to create a .netrc file (as specified on https://cddis.nasa.gov/Data_and_Derived_Products/CreateNetrcFile.html) This is your earthdata login info. You need to register to get it. If on Unix, change permissions to user-only with chmod 600 .netrc. Place this file in your HOME directory.

Then copy and paste the code provided by NASA (https://cddis.nasa.gov/Data_and_Derived_Products/CDDIS_Archive_Access.html) for the Python download:

import requests
import sys

# Reads the URL from the command line argument
url = sys.argv[1]

# Assigns the local file name to the last part of the URL
filename = url.split('/')[-1]

# Makes request of URL, stores response in variable r
r = requests.get(url)

# Opens a local file of same name as remote file for writing to
with open(filename, 'wb') as fd:
for chunk in r.iter_content(chunk_size=1000):
fd.write(chunk)

# Closes local file
fd.close()

Or, you could use this python package: https://pypi.org/project/slrfield/

Fine
  • 3
  • 2
  • If I'm using Windows, where should I save the file? I'm having the same issue where I have the .netrc file with the required credentials but I'm still unable to get the data. I tried saving the file in C:/Users/xyz/.netrc and also in C:/Users/.netrc but to no avail. – CuriousLearner Mar 29 '22 at 23:33
  • haven't tried on windows. Have you tried following the steps listed in: https://www.labkey.org/Documentation/wiki-page.view?name=netrc ? or try having a look at the answer to this question: https://stackoverflow.com/questions/6031214/git-how-to-use-netrc-file-on-windows-to-save-user-and-password – Fine Mar 31 '22 at 06:37
  • It's working fine for me now actually :) – CuriousLearner Apr 01 '22 at 20:38