HDF5 Options

class deker_local_adapters.storage_adapters.hdf5.hdf5_options.HDF5CompressionOpts(compression, compression_opts)

Bases: Serializer

HDF5 compression options.

  • compression (Optional[Union[str, int]]) – compression filter name

  • compression_opts (Optional[Union[list, tuple, int]]) –

    compression level

    Depending on compressing filter compression_opts may be None, integer or tuple. E.g.:

    • hdf5plugin.Blosc object accepts several parameters, including clevel,

    • hdf5plugin.Zstd object accepts an integer in range 1-22,

    • szip filter accepts a tuple of a string and an integer,

    • hdf5plugin.FciDecomp or lzf filters do not accept any parameters at all.

    Read correspondent filter documentation for more information.

property as_dict: dict

Serialize self into a dictionary.

compression: Optional[Union[str, int]]
compression_opts: Optional[Union[list, tuple, int]]
class deker_local_adapters.storage_adapters.hdf5.hdf5_options.HDF5Options(chunks=None, compression_opts=None)

Bases: BaseCollectionOptions

Class for collection on-disk configuration.

Provided that local storage is based on hdf5-files, it uses chunks and compression for disk usage optimization.

  • chunks (Optional[Union[bool, Tuple[int, ...], List[int]]]) –

    Data, stored in hdf5, may be chunked.

    HDF5-format and h5py library provide 3 use cases:

    1. None: no chunking is used

    2. True: data is being chunked automatically with the algorithms and in pieces calculated by HDF5

    3. tuple of integers: user defined chunk-size. E.g. data shape is (100, 100, 100). Upon his/her mind user may chunk it by (1, 10, 100) or by any other applicable chunk size. See chunked storage reference.

  • compression_opts (Optional[HDF5CompressionOpts]) –

    HDF5 compression options.

    HDF5 chunked data may be transformed by the HDF5 filter pipeline. To use built-in and custom compression filters correctly you shall unpack such filter into a HDF5CompressionOpts. There are 3 built-in lossless compression filters:

    1. ”gzip”,

    2. ”lzf”,

    3. ”szip”

More filters can be found in hdf5plugin. ref.: https://docs.h5py.org/en/stable/high/dataset.html#chunked-storage ref.: https://docs.h5py.org/en/stable/high/dataset.html#filter-pipeline

property as_dict: dict

Serialize object as dict.

chunks: Optional[Union[bool, Tuple[int, ...], List[int]]] = None
compression_opts: Optional[HDF5CompressionOpts] = None