What is DEKER™?
DEKER™ is pure Python implementation of petabyte-scale highly parallel data storage engine for multidimensional arrays.
DEKER™ name comes from term dekeract, the 10-cube.
DEKER™ was made with the following major goals in mind:
provide intuitive interface for storing and accessing huge data arrays
support arbitrary number of data dimensions
be thread and process safe and as lean on RAM use as possible
DEKER™ empowers users to store and access a wide range of data types, virtually anything that can be represented as arrays, like geospacial data, satellite images, machine learning models, sensors data, graphs, key-value pairs, tabular data, and more.
DEKER™ does not limit your data complexity and size: it supports virtually unlimited number of data dimensions and provides under the hood mechanisms to partition huge amounts of data for scalability.
Features
Open source under GPL 3.0
Scalable storage of huge virtual arrays via tiling
Parallel processing of virtual array tiles
Own locking mechanism enabling arrays parallel read and write
Array level metadata attributes
Fancy data slicing using timestamps and named labels
Storage level data compression and chunking (via HDF5)
Code and Documentation
Open source implementation of DEKER™ storage engine is published at
API documentation and tutorials for the current release could be found at