0

I've been trying to multiply several netCDF files with each other for in-place calculations such as raster multiplication, but I keep getting different errors, and I was hoping I could get some help.

My bands have the following information (ignoring Attributes for clarity, I can add it if needed):

>>> precds

<xarray.Dataset>
Dimensions:    (bnds: 2, lat: 360, lon: 720, time: 1827)
Coordinates:
  * lon        (lon) float32 -179.75 -179.25 -178.75 ... 178.75 179.25 179.75
  * lat        (lat) float32 -89.75 -89.25 -88.75 -88.25 ... 88.75 89.25 89.75
  * time       (time) datetime64[ns] 2016-01-01T12:00:00 ... 2020-12-31T12:00:00
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(10, 2), meta=np.ndarray>
    prAdjust   (time, lat, lon) float32 dask.array<chunksize=(10, 360, 720), meta=np.ndarray>

>>> wsxr

<xarray.DataArray (band: 1, y: 2160, x: 4320)>
[9331200 values with dtype=float32]
Coordinates:
  * band     (band) int64 1
  * y        (y) float64 89.96 89.87 89.79 89.71 ... -89.71 -89.79 -89.88 -89.96
  * x        (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.7 179.8 179.9 180.0

>>> tempds

<xarray.Dataset>
Dimensions:    (bnds: 2, lat: 360, lon: 720, time: 1827)
Coordinates:
  * lon        (lon) float32 -179.75 -179.25 -178.75 ... 178.75 179.25 179.75
  * lat        (lat) float32 -89.75 -89.25 -88.75 -88.25 ... 88.75 89.25 89.75
  * time       (time) datetime64[ns] 2016-01-01T12:00:00 ... 2020-12-31T12:00:00
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(10, 2), meta=np.ndarray>
    tasAdjust  (time, lat, lon) float32 dask.array<chunksize=(10, 360, 720), meta=np.ndarray>
    height     float64 ...

Here, precds and tempds can multiply successfully, but I can't seem to multiply tempds and wsxr or precds and wsxr without getting some form of error.

Here's what I've tried:

>>> multiplied = precds_var1 * cotton_dask500_ds
>>> multiplied.to_netcdf('multiplied.nc')

Error:
MemoryError: Unable to allocate array with shape (1827, 360, 720, 1, 2160, 500) and data type float32

This is weird, cuz we're using a system that has 256GB of RAM installed, and even after trying this or using just one band, the error message didn't change.

Then, thinking that it might be because of the different names, we changed the names of the bands, but that too didn't work. Going to try to make sure that they have the same types as well (tho, float32 and float64 should still work right?)

We also tried to apply the squeeze function to reduce the total number of dimensions (might need to verify this, cuz my co-worker worked on this not me), but this is the error message that popped up for me:

>>> precipitation_slice_squeeze = precipitation_slice.squeeze()
>>> precipitation_slice_squeeze

<xarray.DataArray 'prAdjust' (lat: 360, lon: 720)>
dask.array<getitem, shape=(360, 720), dtype=float32, chunksize=(360, 720), chunktype=numpy.ndarray>
Coordinates:
  * lon      (lon) float32 -179.75 -179.25 -178.75 ... 178.75 179.25 179.75
  * lat      (lat) float32 -89.75 -89.25 -88.75 -88.25 ... 88.75 89.25 89.75
    time     datetime64[ns] 2019-01-01T12:00:00

>>> precipitation_slice_squeeze * wsxr_renamed_squeeze

ValueError: zero-size array to reduction operation minimum which has no identity

Clearly, we're doing something really really wrong. We're fairly new to XArray and don't have a lot of experience with them. Our main goal is to perform in-place raster calculation, so any and all help would be appreciated!

Ali Abbas
  • 136
  • 1
  • 8
  • shape (1827, 360, 720, 1, 2160, 500) counts up to ~465TB, which you'll have a hard time to find in many supercomputers. The reason why this works for the multiplication of `precds` and `tempds` is they share common dimensions. This is not true for the `wsxr`, which holds obviously some kind of latlon-subgrid values in the x,y data. You would need to add at least some information on what you want to achieve with these datasets and multpiplications. – kmuehlbauer Jan 17 '20 at 07:39
  • OK, how do you suggest I change the dimensions of wsrx to resolve this? – Ali Abbas Jan 17 '20 at 13:23
  • This depends on what you want to achieve. Until now, I have no idea what you are trying to do. So I'm just guessing: You have kind of precipitation/temperature data in 30-arc-minute (0.5 deg resolution), your `precds`/`tempds`-datasets. Then you have some `wsxr` dataset which represents latlon-coordinates in 5-arc-minute (0.083333 deg) resolution. The question is: what do you expect as an output? – kmuehlbauer Jan 17 '20 at 13:38
  • Well, essentially raster algebra with qgis, but doing it for a large number of bands. Multiplying the band wsxr with tempds for each of its time bands. What I intend is to get the answer weighted on the individual bands for each of the ppt raster – Ali Abbas Jan 17 '20 at 13:40
  • Not sure, if this is what you want. But if you want to have your output in the 5-arc-minute resolution you need to interpolate your source datasets. Something along these lines: `ds_interp = tempds.interp(lat=wsxr.y, lon=wsxr.x)` – kmuehlbauer Jan 17 '20 at 13:48
  • Got this error when I tried to multiply them: ```TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(, '__call__', dask.array, array([[[[[0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], ..., [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.]]]]], dtype=float32)): 'Array', 'ndarray' ``` – Ali Abbas Jan 17 '20 at 17:27
  • This is what I was trying to multiply: ``` ds_interp = tempds.interp(lat=wsxr.y, lon=wsxr.x) ds_interp * wsxr ``` – Ali Abbas Jan 17 '20 at 17:29
  • This is what I got btw for the ds_interp: ```Dimensions: (bnds: 2, time: 1827, x: 4320, y: 2160) Coordinates: * time (time) datetime64[ns] 2016-01-01T12:00:00 ... 2020-12-31T12:00:00 lat (y) float64 89.96 89.87 89.79 89.71 ... -89.79 -89.88 -89.96 lon (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0 * y (y) float64 89.96 89.87 89.79 89.71 ... -89.79 -89.88 -89.96 * x (x) float64 -180.0 -179.9 -179.8 -179.7 ... 179.8 179.9 180.0``` – Ali Abbas Jan 17 '20 at 17:31

0 Answers0