RuntimeError: NetCDF: HDF error · Issue #6877 · pydata/xarray

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

What happened?

I am reading the era5 nc data and extracting the values by latitude and longitude.
My code worked fine for 8 hours but then suddenly it broke. Here is the error message.
I'm guessing its an error related to the threads?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.

Complete example — the example is self-contained, including all data and the text of any traceback.

Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook , returning the result.

New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

RemoteTraceback: 
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "D:\project\assimilation\fun_tools.py", line 54, in cell_assimilation
    result = run_ensemble(observations, longitude, latitude)
  File "D:\project\assimilation\enkf.py", line 142, in run_ensemble
    enkf.setup_wofost()
  File "D:\project\assimilation\enkf.py", line 48, in setup_wofost
    self.weather_db = NCWeatherDataProvider(self.longitude, self.latitude)     
  File "D:\project\assimilation\ncweatherdataprovider.py", line 56, in __init__
    self._read_observations()
  File "D:\project\assimilation\ncweatherdataprovider.py", line 105, in _read_observations
    value = pnt[varname.replace("-","_")].values[0]
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\dataarray.py", line 642, in values
    return self.variable.values
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\variable.py", line 512, in values
    return _as_array_or_item(self._data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\variable.py", line 252, in _as_array_or_item
    data = np.asarray(data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 552, in __array__
    self._ensure_cached()
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 549, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 522, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 423, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\coding\variables.py", line 70, in __array__
    return self.func(self.array)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\coding\variables.py", line 137, in _apply_mask
    data = np.asarray(data, dtype=dtype)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 423, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\backends\netCDF4_.py", line 93, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 712, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\backends\netCDF4_.py", line 106, in _getitem
    array = getitem(original_array, key)
  File "src\netCDF4\_netCDF4.pyx", line 4406, in netCDF4._netCDF4.Variable.__getitem__
  File "src\netCDF4\_netCDF4.pyx", line 5350, in netCDF4._netCDF4.Variable._get
  File "src\netCDF4\_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error
Anything else we need to know?
My code is not easily simplified so it may not be reproducible, the last time I ran the code it was an error after 5 hours of normal work.

first: 53%|█████▎    | 952/1803 [5:32:09<4:56:55, 20.93s/it]

second:  78%|███████▊  | 1408/1803 [8:02:31<2:15:22, 20.56s/it]

The error is the same
Environment
commit: None

python: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)]

python-bits: 64

OS: Windows

OS-release: 10

machine: AMD64

processor: Intel64 Family 6 Model 140 Stepping 1, GenuineIntel

byteorder: little

LC_ALL: None

LANG: zh_CN.UTF8

LOCALE: ('Chinese (Simplified)_China', '936')

libhdf5: 1.12.1

libnetcdf: 4.8.1
xarray: 2022.3.0

pandas: 1.3.5

numpy: 1.22.3

scipy: 1.7.3

netCDF4: 1.5.8

pydap: None

h5netcdf: 1.0.1

h5py: 3.7.0

Nio: None

zarr: None

cftime: 1.5.1.1

nc_time_axis: None

PseudoNetCDF: None

rasterio: 1.2.10

cfgrib: None

iris: None

bottleneck: 1.3.4

dask: 2022.6.1

distributed: 2022.6.1

matplotlib: 3.5.1

cartopy: 0.20.1

seaborn: 0.11.2

numbagg: None

fsspec: 2022.3.0

cupy: None

pint: None

sparse: None

setuptools: 61.2.0

pip: 21.2.2

conda: 4.13.0

pytest: 7.1.2

IPython: 7.31.1

sphinx: 4.4.0