S3 Remote (EDataset)¶
cfdb supports syncing datasets with S3-compatible object storage via EBooklet. The EDataset class extends Dataset with remote sync capabilities.
Installation¶
Opening an EDataset¶
import cfdb
from ebooklet import S3Connection
remote_conn = S3Connection(
endpoint_url='https://s3.example.com',
access_key_id='YOUR_KEY',
access_key='YOUR_SECRET',
bucket='my-bucket',
db_key='datasets/example.cfdb',
)
with cfdb.open_edataset(remote_conn, 'local_cache.cfdb', flag='r') as ds:
print(ds)
S3Connection¶
The remote_conn parameter accepts:
| Type | Description |
|---|---|
S3Connection |
Fully configured connection object |
str |
HTTP URL for the remote |
dict |
Parameters for S3Connection() |
Parameters¶
open_edataset() accepts the same parameters as open_dataset() plus the ones below. Everything from
group_bytes on is keyword-only, and any other keyword argument (for example offline=) is passed to
ebooklet.open_ebooklet.
| Parameter | Type | Description |
|---|---|---|
remote_conn |
S3Connection, str, or dict | Remote connection |
group_bytes |
int, None, or omitted | How chunks are stored as S3 objects (ebooklet ≥ 0.11). An int: grouped - chunks new to the remote are packed, in the order they were written, into group objects of up to that many bytes. None: ungrouped, one object per chunk. Omitted: an existing remote keeps its mode and the group_bytes it records (whatever its last push packed with); a new dataset is grouped at 32 MiB. Another int packs new chunks to it from then on and replaces the recorded value. The mode itself is fixed per remote: against an existing remote, a request for the other mode is ignored with a warning, and with flag='n' it raises; call delete_remote() first to change it. See Chunk sizes for remote datasets. |
lock_timeout |
int | Seconds to wait for the write lock when opening for write (default 300); raises TimeoutError after that. |
force_lock |
bool | Break stale write locks before acquiring, to recover from a crashed writer (default False). Age-gated: only lock tickets older than 2 hours are broken, so a live writer is protected. |
Reading Remote Data¶
When reading from an EDataset, chunks are loaded from S3 on demand. Use load() on a variable to pre-fetch chunks:
with cfdb.open_edataset(remote_conn, 'local.cfdb') as ds:
temp = ds['temperature']
temp.load() # fetch all chunks from S3
for slices, data in temp.iter_chunks():
print(data.shape)
Selections also trigger loading only the required chunks:
with cfdb.open_edataset(remote_conn, 'local.cfdb') as ds:
temp = ds['temperature']
subset = temp[0:10, :]
subset.load() # loads only the chunks needed
Writing and Pushing¶
Writes only modify the local file. Nothing is ever uploaded automatically —
publishing to the remote is always an explicit push():
with cfdb.open_edataset(remote_conn, 'local.cfdb', flag='w') as ds:
ds['temperature'][0:10, :] = new_data
ds.push()
push() can be called at any point in the session — including in the same
session that creates the dataset — and publishes the current state of the
dataset (variables, data, and attributes). Closing without pushing simply
leaves the changes local; a later session can push them.
Tracking Changes¶
Check what has changed during the current session:
with cfdb.open_edataset(remote_conn, 'local.cfdb', flag='w') as ds:
ds['temperature'][0, 0] = 42.0
changes = ds.changes()
print(changes)
Attaching to an Existing Remote¶
The local file is just a cache/working copy: opening with 'w' or 'c' and a
fresh local file path attaches to the existing remote dataset (its structure
is pulled on demand). A new dataset is only created when one exists neither
locally nor remotely (or with flag='n', which always creates new).
Dataset Types¶
All four dataset types work as remotes: pass dataset_type (for example 'ts_ortho') when
creating the remote (ts_ortho as of 0.9.1 — earlier versions raised).
Existing remotes always open with their stored type and the matching class;
check it via the dataset_type property:
with cfdb.open_edataset(remote_conn, 'stations.cfdb') as ds:
print(ds.dataset_type) # 'grid', 'ts_ortho', 'ts_forecast' or 'grid_forecast'
Chunk sizes for remote datasets¶
cfdb's default chunk size (about 2¹⁸ elements; see Chunking & Storage) is tuned for local reads. A remote adds a per-request cost of roughly 0.15–0.4 s. That usually outweighs the local differences between chunk sizes, so pick the chunk shape for a remote with requests in mind.
Grouped or ungrouped¶
Grouped (group_bytes=<int>, the default) |
Ungrouped (group_bytes=None) |
|
|---|---|---|
| Storage | chunks packed in write order into objects of up to group_bytes |
one object per chunk |
| Requests per read | one per group touched | one per chunk touched |
| An append uploads | the new chunks, plus at most the partly filled last group | only the new chunks |
| An update of existing chunks re-uploads | each group holding a changed chunk | only the changed chunks |
| Suits | most datasets, including ones that grow by appending, and (with small groups) continuously updated station data | small or rarely updated datasets where one object per chunk is simplest |
Grouping follows write order: chunks written together share groups. A dataset built and extended
time band by time band keeps each band in its own groups, so an append touches only new groups, and
a read of one band's chunks needs few requests. (Before ebooklet 0.11 groups were assigned by hashing
the chunk key, which scattered every append over nearly all groups; such remotes must be moved to the
current format - see ebooklet's changelog. Until then ebooklet >= 0.11 refuses to open them, even
read-only, with UnsupportedFormatError.)
Continuously updated station data (an hourly feed rewriting every station's latest chunk) is best
grouped too, with small groups: one station × ~2520 steps per chunk, written block by block across
stations, and group_bytes at most half of one block (all stations × one time chunk). An hourly
update then uploads a few groups instead of one object per station. envlib-ingest-base's ts_ortho
builder does this by default since 0.6.0.
Ungrouped remotes: make chunks at least ~1.5–2 MB compressed¶
Every chunk is a separate request, and requests are expensive. In measurements over a home connection in New Zealand, a remote served 15–22 requests per second on 10 threads regardless of object size. Below about 1.5–2 MB per object, reads are limited by the number of requests rather than by bandwidth. So for an ungrouped remote:
- Size chunks at or above ~1.5–2 MB compressed. That is usually larger than cfdb's default, so
pass
chunk_shapeexplicitly when creating the variables. - Shape them for the most common query, which for a growing dataset is usually "the latest data". The ECMWF forecast layout (one run × all lead times × one level × the full grid per chunk) answers "latest run" in a few requests.
- Don't leave empty chunks behind. A chunk that holds only missing values still costs a request. Write only where there is data, rather than whole rows padded with missing values.
For station data (ts_ortho), the choice is between one station per chunk and many stations per
chunk, and it depends on which query matters most. Measured on an hourly streamflow dataset
(141 stations, 21 years), with times projected from the real compressed chunk sizes:
| Chunk shape | Latest 7 days, all stations | One station, full history | Whole dataset | Objects uploaded per hourly update |
|---|---|---|---|---|
| (1 station, 25 000 hours) | 141 requests, 6.6 s | 7 requests, 0.3 s | 801 requests, 39 s | 141 |
| (141 stations, 2 190 hours) | 1 request, 0.1 s | 84 requests, 6.5 s | 84 requests, 6.5 s | 1 |
Putting every station into quarter-year chunks makes the latest-data read and each update a single request, but turns one station's history into a scan (0.3 s becomes 6.5 s). Keep one station per chunk when single-station histories are the main read; use multi-station chunks when "latest data, all stations" is. The (141, 2 190) shape is ~310 K elements, inside cfdb's best range for local reads.
# Growing station dataset on an ungrouped remote, read mostly as "latest data, all stations":
# all stations x one quarter (2 190 hours) per chunk. (This layout predates write-order groups; for a
# continuously updated feed, small groups are now the better choice, as above. flag='n' with
# group_bytes=None raises if the remote already exists grouped: delete_remote() it first.)
with cfdb.open_edataset(remote_conn, 'flow.cfdb', flag='n', dataset_type='ts_ortho', group_bytes=None) as ds:
... # create the point and time coordinates
ds.create.data_var.generic('streamflow', ('point', 'time'), dtype=flow_dtype,
chunk_shape=(n_stations, 2190))
Grouped remotes: chunk shape decides the bytes¶
Grouping bundles chunks into fewer requests, so smaller chunks cost little extra. The cost that
remains is bytes downloaded: a read fetches whole chunks, and one request per group spans from the
first to the last chunk needed in that group. On a (hash-grouped, pre-0.11) SST archive with chunks
120 days deep and 134 MB groups, reading one day downloaded all 120 days, and one point's 45-year
series downloaded 24× the data it needed. Shape the chunks for the queries you expect. Write-order
groups of the default 32 MiB bound that span to one group; a smaller group_bytes lowers it further
at the cost of more objects (the default has not yet been benchmarked against alternatives).
The measurements behind this section are in the repository's benchmarks/RESULTS.md, sections
"Remote (EDataset) reads vs chunk size" and "Station (ts_ortho) data".
Remote Management¶
Copy Remote¶
Copy the entire remote dataset to another S3 location:
new_remote = S3Connection(
endpoint_url='https://s3.example.com',
access_key_id='YOUR_KEY',
access_key='YOUR_SECRET',
bucket='backup-bucket',
db_key='datasets/copy.cfdb',
)
with cfdb.open_edataset(remote_conn, 'local.cfdb', flag='w') as ds: # copy_remote needs a writable handle
ds.copy_remote(new_remote)
Delete Remote¶
Remove the remote dataset while keeping the local file:
Thread and Multiprocess Safety¶
EDataset inherits the same safety properties as Dataset — thread locks for concurrent reads/writes and file locks for multiprocessing. The S3 remote uses object locking for consistency.