open_edataset¶
open_edataset(remote_conn, file_path, flag='r', dataset_type='grid', compression=utils.default_compression, compression_level=None, *, group_bytes=_NOT_GIVEN, lock_timeout=300, force_lock=False, num_groups=_NOT_GIVEN, **kwargs)
¶
Open a cfdb that is linked with a remote S3 database.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
remote_conn
|
S3Connection, str, or dict
|
The object to connect to a remote. It can be an S3Connection object, an http url string, or a dict with the parameters for initializing an S3Connection object. |
required |
file_path
|
str or Path
|
It must be a path to a local file location. If you want to use a tempfile, then use the name from the NamedTemporaryFile initialized class. |
required |
flag
|
str
|
Flag associated with how the file is opened according to the dbm style.
A dataset "exists" if it exists locally OR remotely: with |
'r'
|
dataset_type
|
str
|
The dataset type when CREATING a new dataset. Default is
|
'grid'
|
compression
|
str
|
The compression for all chunks, used only when a NEW dataset is created: attaching to an existing local or remote dataset always uses the compression it recorded, whatever is passed here. One of |
default_compression
|
compression_level
|
int or None
|
The compression level. None uses the defaults, which is 1 for every compression option. |
None
|
group_bytes
|
int, None, or omitted
|
The remote storage mode (ebooklet >= 0.11). An int: grouped storage - at each push the chunks new to the remote are packed, in the order they were written, into group objects of up to group_bytes bytes, so appending to a dataset uploads only the new chunks (plus at most one partly filled group). None: per-key storage, one remote object per chunk.
Omitted: an existing remote keeps its mode and the group_bytes it records (whatever its last push packed with); a NEW dataset is grouped at ebooklet.DEFAULT_GROUP_BYTES (32 MiB). Passing another int packs new chunks to it from then on and is recorded in place of the old value. Writing chunks in the order they will usually be read together (e.g. time band by time band) keeps them in the same groups. See ebooklet's "Grouped Storage".
The mode itself (grouped or per-key) is fixed per remote. Against an existing remote, a group_bytes that asks for the other mode is ignored with a UserWarning (the remote wins), and with |
_NOT_GIVEN
|
lock_timeout
|
int
|
Maximum time in seconds to wait for the write lock when opening for write. Default is 300 (5 minutes). Only applies when flag is not |
300
|
force_lock
|
bool
|
If True, break stale write locks before acquiring. Use this to recover from stale locks left by crashed processes. Age-gated by ebooklet: only lock tickets older than 2 hours are broken, so a live writer is protected. Default is False. |
False
|
num_groups
|
None
|
Removed (hash grouping, ebooklet < 0.11). Forwarded to ebooklet for one release: an explicit num_groups=None keeps its old meaning, per-key storage; an int raises ValueError - use group_bytes. Passing both num_groups and group_bytes raises ValueError. |
_NOT_GIVEN
|
**kwargs
|
Any kwargs that can be passed to |
{}
|
Returns:
| Type | Description |
|---|---|
EDataset
|
|