3

The documentation for pygrib shows a function called fromstring which creates a gribmessage instance from a python bytes object representing a binary grib message. I might be misunderstanding the purpose of this function, but it leads me to believe I can use it in place of downloading a GRIB file and using the open function on it. Unfortunately, my attempts to open a multi-message GRIB file from NLDAS2 have failed. Does anyone else know how to use pygrib on GRIB data without first saving the file? My code below shows how I would like it to work. Instead, it gives the error TypeError: expected bytes, int found on the line for grib in gribs:

from urllib import request
import pygrib

url = "<remote address of desired file>"
username = "<username>"
password = "<password>"

redirectHandler = request.HTTPRedirectHandler()
cookieProcessor = request.HTTPCookieProcessor()
passwordManager = request.HTTPPasswordMgrWithDefaultRealm()
passwordManager.add_password(None, "https://urls.earthdata.nasa.gov", username, password)
authHandler = request.HTTPBasicAuthHandler(passwordManager)
opener = request.build_opener(redirectHandler, cookieProcessor, authHandler)
request.install_opener(opener)

with request.urlopen(url) as response:
    data = response.read()
    gribs = pygrib.fromstring(data)
    for grib in gribs:
        print(grib)

Edit to add the entire error output:

Traceback (most recent call last):
  File ".\example.py", line 19, in <module>
    for grb in grbs:
  File "pygrib.pyx", line 1194, in pygrib.gribmessage.__getitem__
TypeError: expected bytes, int found

Edit: This interface does not support multi-message GRIB files, but the authors are open to a pull request if anyone wants to write up the code. Unfortunately, my research focus has shifted and I don't have time to contribute myself.

FlippingBinary
  • 1,357
  • 10
  • 21
  • Please provide the entire error output. In the code you shared you’re still downloading the file yourself, no? – AMC May 11 '20 at 19:21
  • @AMC Either way I'm downloading the file, but I want to avoid saving it to disk. If I want it to work, I have to save it to disk then use `pygrib.open(filename)` – FlippingBinary May 11 '20 at 19:27
  • 1
    `pygrib.fromstring()` apparently works with a single GRIB message - it's not something you can iterate over. You'd have to somehow split the retrieved data into individual messages (I have no idea how you'd do that), and call `.fromstring()` on them individually. – jasonharper May 11 '20 at 19:45
  • @jasonharper That's what I suspected but didn't want to believe. Thank you. – FlippingBinary May 11 '20 at 20:04
  • Take a look to https://github.com/ecmwf/cfgrib. It is the grib engine for xarray! From my point of view this is the state of the art of handling geospatial datasets. – dl.meteo Jun 05 '20 at 06:29

2 Answers2

1

As stated by jasonharper you can use pygrib.fromstring(). I just tried it myself and this works.

Here is the link to the documentation.

Jorn.Beyers
  • 1,784
  • 2
  • 21
  • 26
0

Starting with pygrib v2.1.4, the changelog says that pygrib.open() now accepts io.BufferedReader object as an input argument. see pygrib changelog here

That would theoretically allow you to read a GRIB2 file from RAM memory without writing it to disk.

I think the usage is supposed to be the following :

binary_io = io.BytesIO(bytes_data)
buffer_io = io.BufferedReader(binary_io)
grib_file = pygrib.open(buffer_io)

But I was not able to make it work on my side !

Stephane
  • 175
  • 7
  • Cool find! It looks like those changes have not been released yet, but the pull request bringing in those changes does mention it must be an actual file on disk. https://github.com/jswhit/pygrib/pull/206 Hopefully they expand that to include `io.BytesIO` and not just `io.BufferedReader`. – FlippingBinary Nov 09 '22 at 13:21