Skip to content

open_edataset

open_edataset(remote_conn, file_path, flag='r', dataset_type='grid', compression=utils.default_compression, compression_level=None, *, group_bytes=_NOT_GIVEN, lock_timeout=300, force_lock=False, num_groups=_NOT_GIVEN, **kwargs)

Open a cfdb that is linked with a remote S3 database.

Parameters:

Name Type Description Default
remote_conn S3Connection, str, or dict

The object to connect to a remote. It can be an S3Connection object, an http url string, or a dict with the parameters for initializing an S3Connection object.

required
file_path str or Path

It must be a path to a local file location. If you want to use a tempfile, then use the name from the NamedTemporaryFile initialized class.

required
flag str

Flag associated with how the file is opened according to the dbm style.

  • 'r' -- Open existing database for reading only (default).
  • 'w' -- Open existing database for reading and writing.
  • 'c' -- Open database for reading and writing, creating it if it doesn't exist.
  • 'n' -- Always create a new, empty database, open for reading and writing.

A dataset "exists" if it exists locally OR remotely: with 'w'/'c', a fresh local file path attaches to an existing remote dataset (its structure is pulled on demand) rather than creating a new one. A new dataset is only created when neither exists (or with 'n', which always creates new).

'r'
dataset_type str

The dataset type when CREATING a new dataset. Default is 'grid'. Existing datasets (local or remote) always open with their stored type (and the matching class); this parameter is then ignored.

  • 'grid' -- The standard CF conventions dimensions/coordinates. Each coordinate must be unique and increasing in ascending order. Each coordinate represents a single axis (i.e. x, y, z, t). The z axis is currently optional.
  • 'ts_ortho' -- A special time series coordinate structure representing the orthogonal multidimensional array representation of time series. Designed for time series data with sparse geometries (e.g. station time series data). The Geometry dtype must represent the xy axis. The z axis is currently optional.
  • 'ts_forecast' -- Forecasts at stations or points: the ts_ortho layout with the time axis replaced by forecast_reference_time (when the run was issued) and forecast_period (the lead), dims (point, forecast_reference_time, forecast_period).
  • 'grid_forecast' -- Gridded forecasts: the grid layout with the time axis replaced by forecast_reference_time and forecast_period, dims (x, y, forecast_reference_time, forecast_period).
'grid'
compression str

The compression for all chunks, used only when a NEW dataset is created: attaching to an existing local or remote dataset always uses the compression it recorded, whatever is passed here. One of 'zstd_shuffle' (default), 'zstd', 'lz4_shuffle' or 'lz4'; see open_dataset.

default_compression
compression_level int or None

The compression level. None uses the defaults, which is 1 for every compression option.

None
group_bytes int, None, or omitted

The remote storage mode (ebooklet >= 0.11). An int: grouped storage - at each push the chunks new to the remote are packed, in the order they were written, into group objects of up to group_bytes bytes, so appending to a dataset uploads only the new chunks (plus at most one partly filled group). None: per-key storage, one remote object per chunk. Omitted: an existing remote keeps its mode and the group_bytes it records (whatever its last push packed with); a NEW dataset is grouped at ebooklet.DEFAULT_GROUP_BYTES (32 MiB). Passing another int packs new chunks to it from then on and is recorded in place of the old value. Writing chunks in the order they will usually be read together (e.g. time band by time band) keeps them in the same groups. See ebooklet's "Grouped Storage". The mode itself (grouped or per-key) is fixed per remote. Against an existing remote, a group_bytes that asks for the other mode is ignored with a UserWarning (the remote wins), and with flag='n' it raises ValueError; to change the mode, delete_remote() first. A hash-grouped remote written by ebooklet < 0.11 cannot be opened at all, not even with flag='r': ebooklet raises UnsupportedFormatError, whose message gives the migration steps (only flag='n', a full replacement, is allowed).

_NOT_GIVEN
lock_timeout int

Maximum time in seconds to wait for the write lock when opening for write. Default is 300 (5 minutes). Only applies when flag is not 'r'. Raises TimeoutError if the lock cannot be acquired within the timeout.

300
force_lock bool

If True, break stale write locks before acquiring. Use this to recover from stale locks left by crashed processes. Age-gated by ebooklet: only lock tickets older than 2 hours are broken, so a live writer is protected. Default is False.

False
num_groups None

Removed (hash grouping, ebooklet < 0.11). Forwarded to ebooklet for one release: an explicit num_groups=None keeps its old meaning, per-key storage; an int raises ValueError - use group_bytes. Passing both num_groups and group_bytes raises ValueError.

_NOT_GIVEN
**kwargs

Any kwargs that can be passed to ebooklet.open_ebooklet.

{}

Returns:

Type Description
EDataset