Rechunker

The Rechunker class provides on-the-fly rechunking without modifying the stored data. Access it via variable.rechunker().

cfdb.support_classes.Rechunker

guess_chunk_shape(target_chunk_size)

Guess an appropriate chunk layout for a dataset, given its shape and the size of each element in bytes. The returned chunk's pre-compressed size targets target_chunk_size and may exceed it by up to 1.5x. Chunk dims are snapped to composite numbers: the least common multiple of two composite numbers is very likely far smaller than their product, which benefits later rechunking between two guessed layouts.

Parameters:

Name	Type	Description	Default
target_chunk_size	int	The maximum size per chunk in bytes.	required

Returns:

Type	Description
tuple of ints	shape of the chunk

rechunk(target_chunk_shape, max_mem=2 ** 29)

This method takes a target chunk_shape and max memory size and returns a generator that converts to the new target chunk shape. It optimises the rechunking by using an in-memory numpy ndarray with a size defined by the max_mem.

Parameters:

Name	Type	Description	Default
target_chunk_shape		The chunk_shape of the target.	required
max_mem	int	The memory budget for the rechunking operation in bytes (default 2**29 = 512 MB). This bounds the read buffer plus reorder/batch buffers, except for the documented irreducible floors and the wide-array pending residual — see docs/concepts/rechunking-internals.md.	2 ** 29

Returns:

Type	Description
Generator	tuple of the target slices to the np.ndarray of data

Notes

Yielded arrays may be views into an internal buffer that is reused as iteration advances. Consume (or .copy()) each yielded array BEFORE advancing the generator, and treat yielded arrays as read-only.

calc_n_chunks()

Calculate the total number of chunks in the existing variable.

calc_n_reads_rechunker(target_chunk_shape, max_mem=2 ** 29)

Calculate the total number of reads and writes using the rechunker.

Parameters:

Name	Type	Description	Default
target_chunk_shape	Tuple[int, ...]	The chunk_shape of the target.	required
max_mem	int	The memory budget for the rechunking operation in bytes (default 2**29 = 512 MB). This bounds the read buffer plus reorder/batch buffers, except for the documented irreducible floors and the wide-array pending residual — see docs/concepts/rechunking-internals.md.	2 ** 29

Returns:

Type	Description
tuple	of n_reads, n_writes

calc_ideal_read_chunk_shape(target_chunk_shape)

Calculates the minimum ideal read chunk shape between a source and target, clipped to the variable's extent (per dim, the LCM never exceeds the smallest source-aligned cover of the dim).

calc_ideal_read_chunk_mem(target_chunk_shape)

Calculates the minimum ideal read chunk memory between a source and target. The ideal shape is clipped to the variable's extent, so this reflects what the rechunker would actually allocate on the ideal path.

calc_source_read_chunk_shape(target_chunk_shape, max_mem)

Calculates the optimum read chunk shape given a maximum amount of available memory.

Parameters:

Name	Type	Description	Default
target_chunk_shape	Tuple[int, ...]	The target chunk shape	required
max_mem	int	The max allocated memory to perform the chunking operation in bytes.	required

Returns:

Type	Description
optimal chunk shape: tuple of ints