HDF5 Options

class deker_local_adapters.storage_adapters.hdf5.hdf5_options.HDF5CompressionOpts(compression, compression_opts)

Bases: Serializer

HDF5 compression options.

Parameters

compression (Optional[Union[str, int]]) – compression filter name
compression_opts (Optional[Union[list, tuple, int]]) –
compression level

Depending on compressing filter compression_opts may be None, integer or tuple. E.g.:
- hdf5plugin.Blosc object accepts several parameters, including clevel,
- hdf5plugin.Zstd object accepts an integer in range 1-22,
- szip filter accepts a tuple of a string and an integer,
- hdf5plugin.FciDecomp or lzf filters do not accept any parameters at all.
Read correspondent filter documentation for more information.

class deker_local_adapters.storage_adapters.hdf5.hdf5_options.HDF5Options(chunks=None, compression_opts=None)

Bases: BaseCollectionOptions

Class for collection on-disk configuration.

Provided that local storage is based on hdf5-files, it uses chunks and compression for disk usage optimization.

Parameters

chunks (Optional[Union[bool, Tuple[int, ...], List[int]]]) –
Data, stored in hdf5, may be chunked.

HDF5-format and h5py library provide 3 use cases:
1. None: no chunking is used
2. True: data is being chunked automatically with the algorithms and in pieces calculated by HDF5
3. tuple of integers: user defined chunk-size. E.g. data shape is (100, 100, 100). Upon his/her mind user may chunk it by (1, 10, 100) or by any other applicable chunk size. See chunked storage reference.
compression_opts (Optional[HDF5CompressionOpts]) –
HDF5 compression options.

HDF5 chunked data may be transformed by the HDF5 filter pipeline. To use built-in and custom compression filters correctly you shall unpack such filter into a HDF5CompressionOpts. There are 3 built-in lossless compression filters:
1. ”gzip”,
2. ”lzf”,
3. ”szip”