HDF5 error when opening .nc files in python with xarray

Question

I'm attempting to open MERRA-2 files using xarray, as my title suggests. The specific error I am encountering occurs when I attempt to view the values in a certain variable using a print statement. The error is as follows:

HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 5:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #005: H5Oattribute.c line 476 in H5O__attr_open_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #006: H5Adense.c line 394 in H5A__dense_open(): can't locate attribute in name index
    major: Attribute
    minor: Object not found

I believe that this error (warning?) is thrown for every file I attempt to open. If I wait long enough, the command I ran does actually go through, albeit much longer than it should take.

So, my code is below.

import xarray as xr

data = xr.open_mfdataset('/path/to/my/data/*.nc')
print(data.OMEGA.values)

This should print a matrix containing several submatrices (my dimensions are time, lat, lon, level), printing the value at every coordinate. Again, it does eventually do this, but not before giving me the above warning for every single file in my directory.

I've looked at plenty of stackoverflow, HDFforum, and github posts, and none of their solutions have worked/been applicable to my problem.

this seems related: https://stackoverflow.com/questions/72821108/hdf5-warnings-when-accessing-xarray-dataset — Michael Delgado, Oct 27 '22 at 18:04
this looks like it's a bad binary incompatibility issue or something along those lines. how are you installing your environment? — Michael Delgado, Oct 27 '22 at 18:06
if you're using anaconda, I'd recommend deleting anaconda, installing [`miniforge`](https://github.com/conda-forge/miniforge) or [`mambaforge`](https://github.com/conda-forge/miniforge#mambaforge), and then creating a new environment with all the packages you need installed at once with `conda create -n newenv -c conda-forge python=3.10 [packages]` — Michael Delgado, Oct 27 '22 at 18:06
i'm not sure how i'm installing my environment -- I'm just using pycharm — Logan, Oct 28 '22 at 19:39
ok - that's what I mean. you used conda to manage your environments. if you type `where conda`, does it say `anaconda` in the path? or `miniconda`? you can probably solve this issue by deleting the entire `anaconda` directory, then installing `miniforge` and using that instead. try to get used to using environments - see the conda [user guide section on managing environments](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) for more info. if you install everything into one env, especially if you use different channels, it can lead to trouble like this. — Michael Delgado, Oct 28 '22 at 19:55
Just wanted to say this worked! (Reinstalling conda w/ miniconda) — Logan, Oct 29 '22 at 18:54
awesome! I'll bundle this into an answer for anyone else who comes across this question — Michael Delgado, Oct 29 '22 at 19:58

Michael Delgado · Accepted Answer · 2022-11-03T15:21:04.750

This error occurs due to conflicting non-python (e.g. fortran/C/C++) dependencies.

This commonly happens when you install packages using conda with conflicting channels. This happens a lot when using Anaconda. Anaconda is a nice place to start, because it gives you a pre-built bundle (or "distribution") of data science packages. However, these all come from the defaults channel, and if you later install a package from a different channel into the base environment, you can end up with nasty conflicts like this.

I'd recommend uninstalling anaconda (by deleting your anaconda directory), and installing one of the following:

miniconda - similar to anaconda, but with no packages installed by default.
miniforge - a variant of miniconda which prioritizes the conda-forge channel
mambaforge - mamaba is a compiled variant of conda that runs in parallel. It's a little less stable than conda and the error messages tend to be less helpful, so if you run into trouble, you can always just run the same command using conda to see what's happening - they're completely interchangeable. But mamba is much, much faster.

My top recommendation would be mambaforge - it's really fast and by default helps you avoid combining different channels by setting conda-forge as the priority channel.

When using any of these, don't install packages into your base environment, unless they are cross-environment utilities which themselves can activate your various kernel environments, like jupyterlab or an IDE like VSCode or Spyder. So, after deleting your environment, I'd recommend installing one of these, esp. mambaforge, and then re-installing into a new environment, e.g. with mamba create -n myenv python=3.10 xarray dask netCDF4 bottleneck matplotlib scipy pandas [...].

This is a much better answer than I was able to provide in [my question](https://stackoverflow.com/questions/72821108/hdf5-warnings-when-accessing-xarray-dataset). Very nice, I'll link here back in my question/answer. — jpolly, Nov 03 '22 at 14:57

HDF5 error when opening .nc files in python with xarray

1 Answers1

Linked