7

Given this file:

http://mtarchive.geol.iastate.edu/2019/02/18/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_20190218-150000.grib2.gz

… I get different output depending on the software I use.

wgrib2 2019021815.grib2 -csv wgrib2.csv

and

cdo outputtab,date,time,lat,lon,value 2019021815.grib2 > cdo.txt

both output:

-3 (undefined) :  8,869,250 records
0              : 14,848,865 records
Other values   :    781,885 records
___________________________________
Total          : 24,500,000 records

However:

gdal_translate.exe -of xyz 2019021815.grib2 gdal.csv

outputs -3 for all 24,500,000 records.

I get the same result if I parse the file using Python + GDAL.

Is this a bug in GDAL? I'd like to be able to parse these files using Python.

I'm using Windows for wgrib2, which can be downloaded from ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/Windows_7/

I'm also using Windows for GDAL, which can be downloaded from https://gdal.org/download.html

I'm using Linux for cdo, which can be downloaded from https://code.mpimet.mpg.de/projects/cdo/

Rick Hitchcock
  • 35,202
  • 5
  • 48
  • 79
  • Did you ever get this to work? I get the same -3. – David Sep 24 '19 at 09:35
  • @David, thanks for the confirmation that this doesn't work in GDAL. I may make this a bounty question to get more interest. – Rick Hitchcock Sep 24 '19 at 09:39
  • I work with MRMS quite a bit on my own personal project. It's worth knowing (just in case) that the Weather-Climate Toolkit from NOAA can export MRMS and other gridded/radial data to CSV, json, tiff, etc. If you are ever in a pinch, check it out. – David Sep 25 '19 at 10:31
  • @David, I've used the Toolkit but hadn't tried exporting to CSV. It's twice as fast as wgrib2 and provides a better output format. And it seems to automatically exclude the undefined values. This doesn't solve the GDAL issue, but it looks to be a better solution altogether. If you make a post and I don't get a better answer, I'll be happy to give you the bounty! – Rick Hitchcock Sep 25 '19 at 21:06
  • thanks. I will do that. Hopefully we will see some other posts on this question. GDAL is a beast. – David Sep 26 '19 at 08:54
  • I'm assuming you're on *Win*. Could you share the tools you're using? Are there some that require a license? Did you download them or you build them? Can you share their versions? – CristiFati Sep 26 '19 at 21:55
  • Good questions, @CristiFati. No license needed, and there are executables and instructions for all at the links I've now added to my post. Latest versions. – Rick Hitchcock Sep 26 '19 at 22:31
  • Ok, it's what I was afraid of (regarding *gdal* - it comes as an archive from *conda* repos). And on *Win*, everything is under *Cygwin*. Will test, although my experience in this area is somewhere close to 0. – CristiFati Sep 26 '19 at 22:41
  • Just to add another data point: I get _correct_ results running `gdal_translate` on macOS. – Nikolas Stevenson-Molnar Sep 26 '19 at 22:53
  • Don't go to a lot of trouble. David's solution may be the best. – Rick Hitchcock Sep 26 '19 at 22:53
  • Hmm, that's why I asked about the tool versions. The download *URL*s remain the same but their content is subject to change (update). I wasn't specific enough. Maybe the same *URL* will point to different versions for different platforms. @NikolasStevenson-Molnar kind of supports this. I don't know how to get *gdal* on *OSX*, but I imagine it's not from the *conda* repos. Bottom line: if it works on *OSX*, it should also work on *Win* (the same code version). – CristiFati Sep 26 '19 at 23:36
  • @NikolasStevenson-Molnar, interesting! I'll see if there's an update to GDAL that fixes things, but traveling at the moment. – Rick Hitchcock Sep 27 '19 at 00:29
  • Sorry for being pushy, but the *gdal* download (from *conda* repos) didn't include (as expected) executables (like *gdal\_translate*). The *whl*s are available at https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal (I don't have to build them myself :) ). – CristiFati Sep 27 '19 at 02:19
  • @RickHitchcock I'm on version 2.4.2. – Nikolas Stevenson-Molnar Sep 27 '19 at 18:49
  • Thanks, @CristiFati. Been a while since I did the installation, and I've been traveling so I couldn't confirm. I actually got the Windows binaries from http://www.gisinternals.com/release.php. – Rick Hitchcock Sep 30 '19 at 21:59
  • @NikolasStevenson-Molnar, that version fixes the bug! I'll probably still use David's solution, but your answer answers the question and could qualify for the bounty if you'd like to post. – Rick Hitchcock Sep 30 '19 at 22:21
  • Thanks, @RickHitchcock, I'll turn it into an answer. – Nikolas Stevenson-Molnar Sep 30 '19 at 22:24
  • 1
    Just as a note, I got that error using *gdal\_translate* (versions *2.4.1* and *3.0.1*). which was included in the *whl*s that I mentioned earlier. Anyway, good to hear the problem is fixed! – CristiFati Oct 01 '19 at 04:48

2 Answers2

5

My answer to your issue would be to switch to using the Weather-Climate Toolkit from NOAA.

Download Here for Windows/Mac/Linux

It can export gridded and radial NEXRAD/MRMS data to CSV. A sample batch command for anyone needing this assistance would be :

wct-export DataSetName.grib2.gz csv Output.csv wctbatchconfig.xml

If it throws errors, try a different/earlier/beta version of WCT from : https://www1.ncdc.noaa.gov/pub/data/wct/dist/ and be sure to check the sample commands on the website.

David
  • 605
  • 2
  • 14
  • 44
  • 1
    Even though you've provided me with a fantastic alternative, I had to give the bounty to Nikolas since he provided a direct answer to the question. But thanks again - I'm working on an API to serve weather data (including MRMS), which you may find useful. – Rick Hitchcock Oct 01 '19 at 11:13
4

As you suggest, this seems to be a bug in GDAL. I tested the same command using GDAL 2.4.2, and it works. The change log for 2.4.0 points to several fixes to the GRIB driver.

GRIB driver:

Nikolas Stevenson-Molnar
  • 4,235
  • 1
  • 22
  • 31