What is DEKER™?

_images/logo.png

DEKER™ is pure Python implementation of petabyte-scale highly parallel data storage engine for multidimensional arrays.

DEKER™ name comes from term dekeract, the 10-cube.

DEKER™ was made with the following major goals in mind:

  • provide intuitive interface for storing and accessing huge data arrays

  • support arbitrary number of data dimensions

  • be thread and process safe and as lean on RAM use as possible

DEKER™ empowers users to store and access a wide range of data types, virtually anything that can be represented as arrays, like geospacial data, satellite images, machine learning models, sensors data, graphs, key-value pairs, tabular data, and more.

DEKER™ does not limit your data complexity and size: it supports virtually unlimited number of data dimensions and provides under the hood mechanisms to partition huge amounts of data for scalability.

Features

  • Open source under GPL 3.0

  • Scalable storage of huge virtual arrays via tiling

  • Parallel processing of virtual array tiles

  • Own locking mechanism enabling arrays parallel read and write

  • Array level metadata attributes

  • Fancy data slicing using timestamps and named labels

  • Support for industry standard NumPy and Xarray

  • Storage level data compression and chunking (via HDF5)

Code and Documentation

Open source implementation of DEKER™ storage engine is published at

API documentation and tutorials for the current release could be found at