I want to download 10 years of modis monthly data, for which I have links as lines in a text file. I used the following code:
import requests
import os
import urllib
file = open("link.txt", "r")
Lines = file.readlines()
for line in Lines:
filename= line.strip()
print(filename)
user, password= 'xx@gmail.com', 'yy@123'
r = requests.get(filename, auth=(user, password), allow_redirects=True)
open(str(filename[48:90]), 'wb').write(r.content)
The files getting downloaded are 11kb each and corrupted. I am unable to figure out what is the problem here. The content of the output file is mentioned below.
link.txt contains files like these:
https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20101201_20101231.L3b.MO.CHL.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20110101_20110131.L3b.MO.CHL.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20110201_20110228.L3b.MO.CHL.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/AQUA_MODIS.20110301_20110331.L3b.MO.CHL.nc
Output nc file looks like this:
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Earthdata Login</title>
<meta name="description" content="Earthdata Login">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Google Tag Manager -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push(
{'gtm.start': new Date().getTime(),event:'gtm.js'}
);var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-WNP7MLF');</script>
<!-- End Google Tag Manager -->
<link href="https://cdn.earthdata.nasa.gov/eui/1.1.3/stylesheets/application.css" rel="stylesheet" />
<link rel="stylesheet" media="all" href="/assets/application-432b3917d4a41042c0fd963eba859548ef2993f5ed7a0dca4bdb446fdf807556.css" />
<!--[if IE 7]>
<link rel="stylesheet" href="/assets/font-awesome-ie7.min.css">
<![endif]-->
<link href="//netdna.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet">
<link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,700' rel='stylesheet' type='text/css'>
<meta name="csrf-param" content="authenticity_token" />
<meta name="csrf-token" content="4cewX3n0xOGOTr5gWHgIlZctPtRm55YQ9lohmdPuvq1SfDO/rNhsQZZB3OUPRDLGsqQ/QG9uVqxV5yaLx9GqAQ==" />
<!-- Grid background: http://subtlepatterns.com/graphy/ -->
</head>
<body class="oauth authorize" data-turbolinks-eval=false>
<!-- Google Tag Manager (noscript) -->
<noscript>
<iframe src="https://www.googletagmanager.com/ns.html?id=GTM-WNP7MLF"
height="0" width="0" style="display:none;visibility:hidden"></iframe>
</noscript>
<!-- End Google Tag Manager (noscript) -->
<header id="earthdata-tophat2" style="height: 32px;"></header>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]-->
<div class="container">
<header role="banner">
<div id="masthead-logo">
<h1><a class="ir" href="/">Earthdata Login</a></h1>
<span class="eui-badge badge daac">Earthdata Login</span>
</div>
<a id="hamburger" href="#"><img title="Mobile Menu" alt="Three horizontal lines stacked" src="/assets/hamburger-68c8505066427f3e3f6ee40b24cfd3c9f7c0fe93ee298b9046564637262115fa.png" /></a>
<nav role="navigation" class="masthead">
<div id="hide">
<ul>
</ul>
</div>
</nav>
</header>
<section id="callout-login">
<div class="client-login">
<br>
<h3 class="client-description">
</h3>
</div>
<form id="login" action="/login" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="✓" autocomplete="off" /><input type="hidden" name="authenticity_token" value="amGKbzpZc+qTleRHBa9q8W4OvZNAUzhOfGu1G9gX0QnZ2gmP73XbSouahsJSk1CiS4e8B0na+PLf1rIJzCjFpQ==" autocomplete="off" />
<p><label for="username">Username</label><i class="fa fa-question-circle fa-question-circle--blue user-name" title="Login using either your Username or Email Address"></i><input type="text" name="username" id="username" autofocus="autofocus" class="default" /></p>
<p><label for="password">Password</label><br /><input type="password" name="password" id="password" /></p>
<p><input type="hidden" name="client_id" id="client_id" value="Z0u-MdLNypXBjiDREZ3roA" autocomplete="off" /></p>
<p><input type="hidden" name="redirect_uri" id="redirect_uri" value="https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict" autocomplete="off" /></p> <p><input type="hidden" name="response_type" id="response_type" value="code" autocomplete="off" /></p>
<p><input type="hidden" name="state" id="state" autocomplete="off" /></p>
<p><input type="checkbox" name="stay_in" id="stay_in" value="1" checked="checked" /> <label for="stay_in">Stay signed in (this is a private workstation)</label></p>
<p class="button-with-notes">
<input type="submit" name="commit" value="Log in" class="eui-btn--round eui-btn--green" data-disable-with="Log in" />
<a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code">Register</a>
</p>
<p class="form-instructions">
<em class="icon-question-sign"></em>
<a class="" href="/retrieve_info">I don’t remember my username</a>
<br /><em class="icon-question-sign"></em>
<a class="" href="/reset_passwords/new">I don’t remember my password</a>
<br />
<em class="icon-question-sign"></em>
<a href="javascript:feedback.showForm();" title = 'Need Help? Click on the Feedback button to request help'>Help</a>
</p>
</form>
<aside class="govt-msg">
<div class="nasa-logo"></div>
<p><strong>Why must I register?</strong></p>
<p>
The Earthdata Login provides a single mechanism for user registration and profile management for all EOSDIS system components (DAACs, Tools, Services).
Your Earthdata login also helps the EOSDIS program better understand the usage of EOSDIS services to improve user experience through customization of tools and improvement of services.
EOSDIS data are openly available to all and free of charge except where governed by international agreements.
</p>
</aside>
</section>
<section id="cta">
<h3>Get single sign-on access to all your favorite EOSDIS sites</h3>
<a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code">Register for a Profile</a>
</section>
<div class="govt-warning eui-info-box">
<div class="warning-desktop">
<p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p>
You are accessing a U.S. Government information system, which includes:
(1) this computer;
(2) this computer network;
(3) all computers connected to this network including end user systems;
(4) all devices and storage media attached to this network or to any computer on this network; and
(5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only.
You have no reasonable expectation of privacy regarding any communication transmitted through or data stored on this
information system. At any time, and for any lawful purpose, the U.S. Government may monitor, intercept, search, and
seize any communication or data transiting, stored on, or traveling to or from this information system. You are
NOT authorized to process classified information on this information system. Unauthorized or improper use of this
system may result in suspension or loss of access privileges, disciplinary action, and civil and/or criminal penalties.
</div>
<div class="warning-mobile">
<p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p>
You are accessing a U.S. Government information system, which includes:
(1) this computer;
(2) this computer network;
(3) all computers connected to this network including end user systems;
(4) all devices and storage media attached to this network or to any computer on this network; and
(5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only.
Unauthorized or improper use of this system may result in suspension or loss of access privileges, disciplinary action,
and civil and/or criminal penalties. By using this information system, you acknowledge and consent to the terms
and conditions established in NASA policy and regulatory guidance for NASA IT Systems.
</div>
<div class="warning-mobile-mini">
<strong>
US Govt Property. Unauthorized use subject to prosecution. Use subject to monitoring per
<a href="https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPD&c=2810&s=1E">NPD2810</a>.
</strong>
</div>
</div>
</div>
<footer role="contentinfo">
<h3>For questions regarding the EOSDIS Earthdata Login, please contact <a href="javascript:feedback.showForm();" title="Earthdata Support form">Earthdata Support</a></h3>
<ul>
<li class="version badge eui-badge--md">V 4.158
</li>
<li><a href="/">Home</a></li>
<li><a href="/users/new">Register</a></li>
<li><a href="/documentation">Documentation</a></li>
<li><a title="NASA Home" href="http://www.nasa.gov">NASA</a></li>
</ul>
<p>NASA Official: Stephen Berrick</p>
</footer>
<script src="/assets/application-15d4faf28e91715dccccf34f3e808b7a348298fb623e29d09281b30cc1f87492.js"></script>
<script type="text/javascript">
$(window).scroll(function(e){
parallax();
});
function parallax(){
var scrolled = $(window).scrollTop();
$('#content').css('background-position', 'right ' + -(scrolled*0.25)+'px ');
}
</script>
<script src="https://cdn.earthdata.nasa.gov/tophat2/tophat2.js" id="earthdata-tophat-script" data-show-fbm="true" data-show-status="true" data-status-api-url="https://status.earthdata.nasa.gov/api/v1/notifications"></script>
<script type="text/javascript" src="https://fbm.earthdata.nasa.gov/for/URS4/feedback.js"></script>
<script type="text/javascript">
feedback.init();
</script>
<script type="text/javascript">
setTimeout(function()
{var a=document.createElement("script"); var b=document.getElementsByTagName("script")[0];
a.src=document.location.protocol+"//dnn506yrbagrg.cloudfront.net/pages/scripts/0013/2090.js?"+Math.floor(new Date().getTime()/3600000);
a.async=true;a.type="text/javascript";b.parentNode.insertBefore(a,b)}
, 1);
</script>
<!-- BEGIN: DAP Google Analytics -->
<script language="javascript" id="_fed_an_ua_tag" src="https://dap.digitalgov.gov/Universal-Federated-Analytics-Min.js?agency=NASA&subagency=GSFC&dclink=true"></script>
<!-- END: DAP Google Analytics -->
</body>
</html>
