1

I'm reading a large wrfout data(about 100x100 in space, 30 in vertical, 400 in times) by ncl.

fid=addfile("wrfout_d03.nc","r")
u=fid->U

The variable U is about 500M, so it takes much time, and I also need to read other variables.Is there any way for ncl to read large netcdf data quickly? Or can I use other languages?

jstzysq
  • 13
  • 4
  • One good starting point is compressing the file (e.g. with CDO or NCO) beforehand. The slowest part of the reading is usually the disk access, so reducing that to a minimum speeds things up, even if the CPU has to do the decompressing. Another option is reducing the variables to smaller types, such as converting precipitation (mm/day) to unsigned integer, if the loss in precision is acceptable. If you can/want to read only parts of the file at a time (e.g. timeseries or vertical profiles), you can learn about chunking, this has huge impacts on NeCDF reading times, even more if compressed. – AF7 Jun 11 '17 at 07:56
  • *"One good starting point is compressing...**; see my comment on Adrian's answer. I tried it with Python and it seems to be (at least with Python) very slow. I would personally also start by selecting the data that you actually need to analyze. If that is the full 3D field + all time steps then I guess you have to live with the long loading time, unless you want to do fancy stuff like parallel reading and post-processing. – Bart Jun 12 '17 at 17:29

1 Answers1

1

It may be more helpful to extract the variables and timeslices you need before reading them into NCL.

To select by variable:

cdo selvar,var in.nc out.nc

To select by level: 

cdo sellevel

or levels selected by their index:

cdo sellevidx

you can also extract subsets in terms of dates or times...

More info here: https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo#Documentation

ClimateUnboxed
  • 7,106
  • 3
  • 41
  • 86
  • I was curious about this compression, so I quickly tried it with a 576 MB large netCDF file from our model. Compression (using your command) reduced it to 472MB, but when I read them both to memory in Python, it takes 8 (!) times longer to read the compressed one, I guess because of the time needed to decompress it in Python. – Bart Jun 12 '17 at 17:14
  • Good point, it uses up less disc space but access times will be longer, a good compromise is zip5 or 6. But as your question is about access times only in fact I decided to delete that part of the answer. – ClimateUnboxed Jun 13 '17 at 05:40
  • It is not my question, I was just curious :) – Bart Jun 13 '17 at 06:41