Convert grib2 files to csv

Question

I am working on a classification problem with grib2 files. I have been trying Xarray, pynio, pygrib to read these data but couldn't solve it yet. Can anybody explain what is the structure of grib2 files? How they are created? Is there any quick way to convert them into csv?

Answer 1

1. I would recommend to use cfgrib. It is the state of the art python tool using eccodes. It is fully integrated into xarray.

2. When you open a grib file there are two ways you should know: The first one requires (if several variables in the file) a filter argument:

import xarray

grib_data = xarray.open_dataset('/path/to/your/grib_file.grb', engine='cfgrib', backend_kwargs={'filter_by_keys':{'typeOfLevel': 'heightAboveGround','level': 2}})

To get all variables from the file you can use cfgrib's open_datasets:

import cfgrib
grib_data = cfgrib.open_datasets('/path/to/your/grib_file.grb')

grib_data will be a list of xarray.Dataset.

3. If you are receiving default parameter names like paramId0 it is likely that you need the right grib tables. These are provided by the weather service who published the grib file and should be placed at ECCODES_DEFINTION_PATH

Answer 2

GRIB2 is similarly to GRIB1 divided to messages and each message into sections. There can be more messages in one GRIB file. There are just concatenated after each other. If you would have 2 GRIB2 files and want to merge them a simple use of cat command would suffice.

GRIB2 is described for example here: https://www.yumpu.com/en/document/view/11723135/guide-to-wmo-table-driven-code-forms. A good online sources:

• https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/
• https://apps.ecmwf.int/codes/grib/format/grib2/

The online sources describe more in detail the parameters of the different sections.

While GRIB2 basic concepts and start of a message is similar to GRIB1, in later sections (1-7) it is rather different. GRIB2 also allows repetition of some sections in one message:

Section 0: Indicator Section
Section 1: Identification Section
Section 2: Local Use Section (optional)                                  |
Section 3: Grid Definition Section                       |               |
Section 4: Product Definition Section    |               |               |
Section 5: Data Representation Section   | (repeated)    | (repeated)    | (repeated)
Section 6: Bit-Map Section               |               |               |
Section 7: Data Section                  |               |               |
Section 8: End Section

Section 0: is always 16 bytes in GRIB2 (8 bytes in GRIB1) and contains the total length of the message (all sections) the grib edition (GRIB1/GRIB2) and in case of GRIB2 the discipline of the message.

In GRIB2 each section (except section 1 and 8) starts with length of the section (4 bytes) and section number (1 byte). So when you read the binary GRIB2 file you can relatively simply separate a GRIB2 message into its sections.

Section 8: always contains 4 bytes, a string: 7777

Classification

Each GRIB file can contain multiple messages. A message is a grid (section 3) product definition (section 4, one of: wind, temperature, relative humidity, ...) and data (section 7). That means that in each file you may have different data (wind speed, air temperature, current direction, ...). If you want to classify those files, you should probably look into the Product Definition Section (Section 4).

Note that one message usually refers to one reference time (time of measurement or creation of the dataset), defined in Identification Section (section 1) and one forecast time (time when the data, e.g. temparature, is valid for), defined in Product Definition Section (Section 4)

Conversion to CSV

You can read the Data Section (Section 7) with the help of Data Representation Section (Section 5) to get the actual data. Using wgrib2 (https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/) you can easily dump the data into a file. Just note that to know what the data represent and what units they use you need to also take Product Definition Section (Section 4) into account.

Note that the data in Data Section (section 7) are encoded. How they are encoded is described in Data Representation Section (section 5). In some cases there is also a non-empty Bit-Map Section (section 6) which sais on what positions there are valid values. An example may be cloud cover in percent. The Bit-Map section would define points on the grid where there is any cloud cover or not (true/false bit array). The data section than only contains points where there is some cloud cover (bit-map section bit is true) and stores value between 0-100%. This is one of the ways how space can be reduced.