Create loop for ncks to extract NetCDF data from Hycom using Python

Question

I already have written a ncks code to extract data from the Hycom Database, however, due to server issues I can only download one timestep at a time. I need a loop that allows me to download singular timestep data for the time period of one month.

ncks -d time,2015-12-01T00:00:00Z -d depth,0.00,1000.00,1 -v water_temp,salinity,water_u,water_v,surf_el -d lon,161.45,171.28 -d lat,-4.75,3.43 'http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015' test.nc

4D Variables I want to download: water_temp, salinity, water_u, water_v (their dimensions are time, latitude, longitude, depth)

3D Variables I want to download: surf_el (dimensions are time, latitude and longitude)

For the 4D variables, I want the depth to bet between 0 and 1000m. That code is for a singular time, I want to cover the period between 2015/12/01 and 2015/12/31. The whole December of 2015.

Here some Python working code:

import netCDF4
from netCDF4 import Dataset
dec = Dataset('http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015')
print(dec.variables.keys())  #odict_keys(['depth', 'lat', 'lon', 'time', 'tau', 'water_u', 'water_u_bottom', 'water_v', 'water_v_bottom', 'water_temp', 'water_temp_bottom', 'salinity', 'salinity_bottom', 'surf_el'])

long = dec.variables['lon']
lati = dec.variables['lat']
time = dec.variables['time']

for d in dec.dimensions.items():
    print(d)

#('depth', <class 'netCDF4._netCDF4.Dimension'>: name = 'depth', size = 40)
#('lat', <class 'netCDF4._netCDF4.Dimension'>: name = 'lat', size = 3251)
#('lon', <class 'netCDF4._netCDF4.Dimension'>: name = 'lon', size = 4500)
#('time', <class 'netCDF4._netCDF4.Dimension'>: name = 'time', size = 2861)

It looks like your question is "How do you run a system command from Pyton?" — Robert Wilson, May 26 '21 at 11:10

score 1 · Answer 1 · answered May 26 '21 at 21:42

1

The NCO manual has examples of looping over the time dimension here:

for idx in {1..12}; do # Bash Shell (version 3.0+) 
  idx=`printf "%02d" ${idx}` # Zero-pad to preserve order
  ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
done

Something similar should work for you if you replace the 2-digit month in the date hyperslab with the ${idx} variable as above.

answered May 26 '21 at 21:42

Charlie Zender

5,929
14
19

nice - just to add that if this form of loop doesn't work for the shell version you are using, one can also use the external "seq" as `for idx in $(seq 1 12); do` – ClimateUnboxed Jun 04 '21 at 10:06

Alessandro Gianfelici · Answer 2 · 2022-09-10T17:32:48.770

I had the same issue using python's netCDF4 for these data, it's really slow for time range under 1y (I figured out that it assemble the full 1 year dataset located in http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015 and only after that it applies the filters). I solved the issue downloading the individual hourly files from the server and putting them togheter using xarray. In this way only the needed time range is actually downloaded, so it's much faster. I wrote a python library that download Hycom's GLBv0.08/expt_53.X data using this method, I leave here the link to my repository if it can be useful for you!

https://github.com/AlessandroGianfelici/hycom_downloader

Create loop for ncks to extract NetCDF data from Hycom using Python

2 Answers2