TileDB Python API Reference

Warning

The Python interface to TileDB is still under development and the API is subject to change.

Modules

Typical usage of the Python interface to TileDB will use the top-level module tiledb, e.g.

import tiledb

Arrays may be opened with the tiledb.open function:

tiledb.open(uri, mode='r', key=None, attr=None, config=None, ctx=None)

Open a TileDB array at the given URI

Parameters:
  • uri – any TileDB supported URI
  • key – encryption key, str or None
  • mode (str) – (default ‘r’) Open the array object in read ‘r’ or write ‘w’ mode
  • attr – attribute name to select from a multi-attribute array, str or None
  • config – TileDB config dictionary, dict or None
Returns:

open TileDB {Sparse,Dense}Array object

There is also a submodule libtiledb which contains the necessary bindings to the underlying TileDB native library. Most of the time you will not need to interact with tiledb.libtiledb unless you need native-library specific information, e.g. the version number:

import tiledb
tiledb.libtiledb.version()  # Native TileDB library version number

Data import helpers

tiledb.from_numpy(uri, array, ctx=None, **kw)

Convenience method to create a TileDB array from a Numpy array. See documentation of tiledb.DenseArray.from_numpy().

tiledb.from_csv(uri, csv_file, **kwargs)

Create TileDB array at given URI from a CSV file or list of files

Parameters:
  • uri – URI for new TileDB array
  • csv_file – input CSV file or list of CSV files. Note: multi-file ingestion requires a chunksize argument. Files will be read in batches of at least chunksize rows before writing to the TileDB array.
Keyword Arguments:
 
  • Any pandas.read_csv supported keyword argument.
  • TileDB-specific arguments:
    • allows_duplicates: Generated schema should allow duplicates
    • cell_order: Schema cell order
    • tile_order: Schema tile order
    • mode: (default ingest), Ingestion mode: ingest, schema_only, append
    • full_domain: Dimensions should be created with full range of the dtype
    • attrs_filters: FilterList to apply to all Attributes
    • coords_filters: FilterList to apply to all coordinates (Dimensions)
    • sparse: (default True) Create sparse schema
    • tile: Schema tiling (capacity)
    • date_spec: Dictionary of {column_name: format_spec} to apply to date/time columns which are not correctly inferred by pandas ‘parse_dates’. Format must be specified using the Python format codes: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
Returns:

None

Example:

>>> import tiledb
>>> tiledb.from_csv("iris.tldb", "iris.csv")
>>> tiledb.object_type("iris.tldb")
'array'
tiledb.from_pandas(uri, dataframe, **kwargs)

Create TileDB array at given URI from pandas dataframe

Parameters:
  • uri – URI for new TileDB array
  • dataframe – pandas DataFrame
  • mode – Creation mode, one of ‘ingest’ (default), ‘schema_only’, ‘append’
Keyword Arguments:
 

optional keyword arguments for TileDB, see tiledb.from_csv.
Raises:

tiledb.TileDBError
Returns:

None

Exceptions

exception tiledb.TileDBError

TileDB Error Exception

Captures and raises error return code (TILEDB_ERR) messages when calling libtiledb functions. The error message that is raised is the last error set for the tiledb.Ctx

A Python MemoryError is raised on TILEDB_OOM

message

The TileDB error message string

Return type:str
Returns:error message

Context

class tiledb.Ctx(config=None)

Class representing a TileDB context.

A TileDB context wraps a TileDB storage manager.

Parameters:config (tiledb.Config or dict) – Initialize Ctx with given config parameters
config(self)

Returns the Config instance associated with the Ctx.

set_tag(self, key, value)

Sets a (string, string) “tag” on the Ctx (internal).

Config

class tiledb.Config(params=None, path=None)

TileDB Config class

See: https://docs.tiledb.io/en/stable/tutorials/config.html#summary-of-parameters

Unknown parameters will be ignored!

Parameters:
  • params (dict) – Set parameter values from dict like object
  • path (str) – Set parameter values from persisted Config parameter file
clear(self)

Unsets all Config parameters (returns them to their default values)

dict(self, prefix=u'')

Returns a dict representation of a Config object

Parameters:prefix (str) – return only parameters with a given prefix
Return type:dict
Returns:Config parameter / values as a a Python dict
from_file(self, path)

Update a Config object with from a persisted config file

Parameters:path – A local Config file path
get(self, key, *args)

Gets the value of a config parameter, or a default value.

Parameters:
  • key (str) – Config parameter
  • args – return arg if Config does not contain parameter key
Returns:

Parameter value, arg or None.
items(self, prefix=u'')

Returns an iterator object over Config parameters, values

Parameters:prefix (str) – return only parameters with a given prefix
Return type:ConfigItems
Returns:iterator over Config parameter, value tuples
keys(self, prefix=u'')

Returns an iterator object over Config parameters (keys)

Parameters:prefix (str) – return only parameters with a given prefix
Return type:ConfigKeys
Returns:iterator over Config parameter string keys
static load(uri)

Constructs a Config class instance from config parameters loaded from a local Config file

Parameters:uri (str) – a local URI config file path
Return type:tiledb.Config
Returns:A TileDB Config instance with persisted parameter values
Raises:TypeError – uri cannot be converted to a unicode string
Raises:tiledb.TileDBError
save(self, uri)

Persist Config parameter values to a config file

Parameters:uri (str) – a local URI config file path
Raises:TypeError – uri cannot be converted to a unicode string
Raises:tiledb.TileDBError
update(self, odict)

Update a config object with parameter, values from a dict like object

Parameters:odict – dict-like object containing parameter, values to update Config.
values(self, prefix=u'')

Returns an iterator object over Config values

Parameters:prefix (str) – return only parameters with a given prefix
Return type:ConfigValues
Returns:iterator over Config string values

Array Schema

class tiledb.ArraySchema(domain=None, attrs=(), cell_order='row-major', tile_order='row-major', capacity=0, coords_filters=None, offsets_filters=None, allows_duplicates=False, sparse=False, Ctx ctx=None)

Schema class for TileDB dense / sparse array representations

Parameters:
  • domain – Domain of schema
  • cell_order ('row-major' (default) or 'C', 'col-major' or 'F') – TileDB label for cell layout
  • tile_order ('row-major' (default) or 'C', 'col-major' or 'F') – TileDB label for tile layout
  • capacity (int) – tile cell capacity
  • coords_filters (tiledb.FilterList) – (default None) coordinate filter list
  • offsets_filters (tiledb.FilterList) – (default None) offsets filter list
  • allows_duplicates (bool) – True if duplicates are allowed
  • sparse (bool) – True if schema is sparse, else False (set by SparseArray and DenseArray derived classes)
  • ctx (tiledb.Ctx) – A TileDB Context
Raises:

tiledb.TileDBError
allows_duplicates

Returns True if the (sparse) array allows duplicates.

attr(self, key)

Returns an Attr instance given an int index or string label

Parameters:key (int or str) – attribute index (positional or associative)
Return type:tiledb.Attr
Returns:The ArraySchema attribute at index or with the given name (label)
Raises:TypeError – invalid key type
attr_or_dim_dtype(self, unicode name)
capacity

The array capacity

Return type:int
Raises:tiledb.TileDBError
cell_order

The cell order layout of the array.

coords_compressor

The compressor label and level for the array’s coordinates.

Return type:tuple(str, int)
Raises:tiledb.TileDBError
coords_filters

The FilterList for the array’s coordinates

Return type:tiledb.FilterList
Raises:tiledb.TileDBError
domain

The Domain associated with the array.

Return type:tiledb.Domain
Raises:tiledb.TileDBError
dump(self)

Dumps a string representation of the array object to standard output (stdout)

has_attr(self, name)

Returns true if the given name is an Attribute of the ArraySchema

Parameters:name – attribute name
Return type:boolean
static load(uri, Ctx ctx=None, key=None)
nattr

The number of array attributes.

Return type:int
Raises:tiledb.TileDBError
ndim

The number of array domain dimensions.

Return type:int
offsets_compressor

The compressor label and level for the array’s variable-length attribute offsets.

Return type:tuple(str, int)
Raises:tiledb.TileDBError
offsets_filters

The FilterList for the array’s variable-length attribute offsets

Return type:tiledb.FilterList
Raises:tiledb.TileDBError
shape

The array’s shape

Return type:tuple(numpy scalar, numpy scalar)
Raises:TypeError – floating point (inexact) domain
sparse

True if the array is a sparse array representation

Return type:bool
Raises:tiledb.TileDBError
tile_order

The tile order layout of the array.

Return type:str
Raises:tiledb.TileDBError
tiledb.empty_like(uri, arr, config=None, key=None, tile=None, ctx=None)

Create and return an empty, writeable DenseArray with schema based on a NumPy-array like object.

Parameters:
  • uri – array URI
  • arr – NumPy ndarray, or shape tuple
  • config – (optional, deprecated) configuration to apply to new Ctx
  • key – (optional) encryption key, if applicable
  • tile – (optional) tiling of generated array
  • ctx – (optional) TileDB Ctx
Returns:

Attribute

class tiledb.Attr(name=u'', dtype=np.float64, var=False, filters=None, Ctx ctx=None)

Class representing a TileDB array attribute.

Parameters:
  • ctx (tiledb.Ctx) – A TileDB Context
  • name (str) – Attribute name, empty if anonymous
  • dtype (bool) – Attribute value datatypes
  • var – Attribute is variable-length (automatic for byte/string types)
  • filters (FilterList) – List of filters to apply
Raises:

TypeError – invalid dtype
Raises:

tiledb.TileDBError
compressor

String label of the attributes compressor and compressor level

Return type:tuple(str, int)
Raises:tiledb.TileDBError
dtype

Return numpy dtype object representing the Attr type

Return type:numpy.dtype
dump(self)

Dumps a string representation of the Attr object to standard output (stdout)

filters

FilterList of the TileDB attribute

Return type:tiledb.FilterList
Raises:tiledb.TileDBError
isanon

True if attribute is an anonymous attribute

Return type:bool
isvar

True if the attribute is variable length

Return type:bool
Raises:tiledb.TileDBError
name

Attribute string name, empty string if the attribute is anonymous

Return type:str
Raises:tiledb.TileDBError
ncells

The number of cells (scalar values) for a given attribute value

Return type:int
Raises:tiledb.TileDBError

Filters

class tiledb.FilterList(filters=None, chunksize=None, Ctx ctx=None)

An ordered list of Filter objects for filtering TileDB data.

FilterLists contain zero or more Filters, used for filtering attribute data, the array coordinate data, etc.

Parameters:
  • ctx (tiledb.Ctx) – A TileDB context
  • filters – An iterable of Filter objects to add.
  • chunksize (int) – (default None) chunk size used by the filter list in bytes

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     # Create several filters
...     gzip_filter = tiledb.GzipFilter()
...     bw_filter = tiledb.BitWidthReductionFilter()
...     # Create a filter list that will first perform bit width reduction, then gzip compression.
...     filters = tiledb.FilterList([bw_filter, gzip_filter])
...     a1 = tiledb.Attr(name="a1", dtype=np.int64, filters=filters)
...     # Create a second attribute filtered only by gzip compression.
...     a2 = tiledb.Attr(name="a2", dtype=np.int64,
...                      filters=tiledb.FilterList([gzip_filter]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1, a2))
...     tiledb.DenseArray.create(tmp + "/array", schema)
append(self, Filter filter)

Appends filter to the end of filter list

Parameters:filter (Filter) – filter object to add
Returns:None
chunksize

The chunk size used by the filter list.

nfilters
Returns:Number of filters in the filter list
Return type:int
class tiledb.libtiledb.CompressionFilter(tiledb_filter_type_t filter_type, level, Ctx ctx=None)

Base class for filters performing compression.

All compression filters support a compression level option, although some (such as RLE) ignore it.

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.GzipFilter(level=10)]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
level

The compression level setting for the filter.

Every compressor interprets this value differently (some ignore it, such as RLE).

Returns:compression level
Return type:int
class tiledb.GzipFilter(level=None, Ctx ctx=None)

Filter that compresses using gzip.

Parameters:
  • ctx (tiledb.Ctx) – TileDB Ctx
  • level (int) – (default None) If not None set the compressor level

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.GzipFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.ZstdFilter(level=None, Ctx ctx=None)

Filter that compresses using zstd.

Parameters:
  • ctx (tiledb.Ctx) – TileDB Ctx
  • level (int) – (default None) If not None set the compressor level

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.ZstdFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.LZ4Filter(level=None, Ctx ctx=None)

Filter that compresses using lz4.

Parameters:
  • ctx (tiledb.Ctx) – TileDB Ctx
  • level (int) – (default None) If not None set the compressor level

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.LZ4Filter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.Bzip2Filter(level=None, Ctx ctx=None)

Filter that compresses using bzip2.

Parameters:level (int) – (default None) If not None set the compressor level

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.Bzip2Filter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.RleFilter(Ctx ctx=None)

Filter that compresses using run-length encoding (RLE).

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.RleFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.DoubleDeltaFilter(Ctx ctx=None)

Filter that performs double-delta encoding.

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.DoubleDeltaFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.BitShuffleFilter(Ctx ctx=None)

Filter that performs a bit shuffle transformation.

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.BitShuffleFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.ByteShuffleFilter(Ctx ctx=None)

Filter that performs a byte shuffle transformation.

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.ByteShuffleFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
class tiledb.BitWidthReductionFilter(window=None, Ctx ctx=None)

Filter that performs bit-width reduction.

param ctx:A TileDB Context
type ctx:tiledb.Ctx
param window:(default None) max window size for the filter
type window:int

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.BitWidthReductionFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
window
Returns:The maximum window size used for the filter
Return type:int
class tiledb.PositiveDeltaFilter(window=None, Ctx ctx=None)

Filter that performs positive-delta encoding.

Parameters:
  • ctx (tiledb.Ctx) – A TileDB Context
  • window (int) – (default None) the max window for the filter

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     a1 = tiledb.Attr(name="a1", dtype=np.int64,
...                      filters=tiledb.FilterList([tiledb.PositiveDeltaFilter()]))
...     schema = tiledb.ArraySchema(domain=dom, attrs=(a1,))
...     tiledb.DenseArray.create(tmp + "/array", schema)
window
Returns:The maximum window size used for the filter
Return type:int

Dimension

class tiledb.Dim(name=u'__dim_0', domain=None, tile=None, dtype=np.uint64, Ctx ctx=None)

Class representing a dimension of a TileDB Array.

Parameters:
Dtype:

the Dim numpy dtype object, type object, or string that can be corerced into a numpy dtype object
Raises:
  • ValueError – invalid domain or tile extent
  • TypeError – invalid domain, tile extent, or dtype type
Raises:

TileDBError
domain

The dimension (inclusive) domain.

The dimension’s domain is defined by a (lower bound, upper bound) tuple.

Return type:tuple(numpy scalar, numpy scalar)
dtype

Numpy dtype representation of the dimension type.

Return type:numpy.dtype
isanon

True if the dimension is anonymous

Return type:bool
isvar

True if the dimension is variable length

Return type:bool
Raises:tiledb.TileDBError
name

The dimension label string.

Anonymous dimensions return a default string representation based on the dimension index.

Return type:str
shape

The shape of the dimension given the dimension’s domain.

Note: The shape is only valid for integer and datetime dimension domains.

Return type:tuple(numpy scalar, numpy scalar)
Raises:TypeError – floating point (inexact) domain
size

The size of the dimension domain (number of cells along dimension).

Return type:int
Raises:TypeError – floating point (inexact) domain
tile

The tile extent of the dimension.

Return type:numpy scalar or np.timedelta64

Domain

class tiledb.Domain(Ctx ctx=None, *dims)

Class representing the domain of a TileDB Array.

Parameters:
  • dims – one or more tiledb.Dim objects up to the Domain’s ndim
  • ctx (tiledb.Ctx) – A TileDB Context
Raises:

TypeError – All dimensions must have the same dtype
Raises:

TileDBError
dim(self, dim_id)

Returns a Dim object from the domain given the dimension’s index or name.

Parameters:dim_d – dimension index (int) or name (str)
Raises:tiledb.TileDBError
dtype

The numpy dtype of the domain’s dimension type.

Return type:numpy.dtype
dump(self)

Dumps a string representation of the domain object to standard output (STDOUT)

has_dim(self, name)

Returns true if the Domain has a Dimension with the given name

Parameters:name – name of Dimension
Return type:bool
Returns:
homogeneous

Returns True if the domain’s dimension types are homogeneous.

ndim

The number of dimensions of the domain.

Return type:int
shape

The domain’s shape, valid only for integer domains.

Return type:tuple
Raises:TypeError – floating point (inexact) domain
size

The domain’s size (number of cells), valid only for integer domains.

Return type:int
Raises:TypeError – floating point (inexact) domain

Array

class tiledb.libtiledb.Array(uri, mode='r', key=None, timestamp=None, attr=None, Ctx ctx=None)

Base class for TileDB array objects.

Defines common properties/functionality for the different array types. When an Array instance is initialized, the array is opened with the specified mode.

Parameters:
  • uri (str) – URI of array to open
  • mode (str) – (default ‘r’) Open the array object in read ‘r’ or write ‘w’ mode
  • key (str) – (default None) If not None, encryption key to decrypt the array
  • timestamp (int) – (default None) If not None, open the array at a given TileDB timestamp
  • attr (str) – (default None) open one attribute of the array; indexing a dense array will return a Numpy ndarray directly rather than a dictionary.
  • ctx (Ctx) – TileDB context
attr(self, key)

Returns an Attr instance given an int index or string label

Parameters:key (int or str) – attribute index (positional or associative)
Return type:Attr
Returns:The array attribute at index or with the given name (label)
Raises:TypeError – invalid key type
close(self)

Closes this array, flushing all buffered data.

consolidate(self, Config config=None, key=None)

Consolidates fragments of an array object for increased read performance.

Overview: https://docs.tiledb.com/main/basic-concepts/consolidation

Parameters:
  • config (tiledb.Config) – The TileDB Config with consolidation parameters set
  • key (str or bytes) – (default None) encryption key to decrypt an encrypted array
Raises:

tiledb.TileDBError
coords_dtype

Returns the numpy record array dtype of the array coordinates

Return type:numpy.dtype
Returns:coord array record dtype
create(type cls, uri, ArraySchema schema, key=None, Ctx ctx=None)

Creates a TileDB Array at the given URI

Parameters:
  • uri (str) – URI at which to create the new empty array.
  • schema (ArraySchema) – Schema for the array
  • key (str) – (default None) Encryption key to use for array
  • Ctx (ctx) – (default None) Optional TileDB Ctx used when creating the array, by default uses the ArraySchema’s associated context (not necessarily tiledb.default_ctx).
dim(self, dim_id)

Returns a Dim instance given a dim index or name

Parameters:key (int or str) – attribute index (positional or associative)
Return type:Attr
Returns:The array attribute at index or with the given name (label)
Raises:TypeError – invalid key type
domain

The Domain of this array.

dtype

The NumPy dtype of the specified attribute

dump(self)
isopen

True if this array is currently open.

iswritable

This array is currently opened as writable.

static load_typed(uri, mode='r', key=None, timestamp=None, attr=None, Ctx ctx=None)

Return a {Dense,Sparse}Array instance from a pre-opened Array (internal)

meta

Return array metadata instance

Return type:tiledb.Metadata
mode

The mode this array was opened with.

multi_index

Retrieve data cells with multi-range, domain-inclusive indexing. Returns the cross-product of the ranges.

Parameters:selection (list) – Per dimension, a scalar, slice, or list of scalars or slice objects. Scalars and slice components should match the type of the underlying Dimension.
Returns:dict of {‘attribute’: result}. Coords are included by default for Sparse arrays only (use Array.query(coords=<>) to select).
Raises:IndexError – invalid or unsupported index selection
Raises:tiledb.TileDBError

multi_index[] accepts, for each dimension, a scalar, slice, or list of scalars or slice objects. Each item is interpreted as a point (scalar) or range (slice) used to query the array on the corresponding dimension.

Unlike NumPy array indexing, multi_index respects TileDB’s range semantics: slice ranges are inclusive of the start- and end-point, and negative ranges do not wrap around (because a TileDB dimensions may have a negative domain).

See also: https://docs.tiledb.com/main/api-usage/reading-arrays/multi-range-subarrays

** Example **

>>> import tiledb, tempfile, numpy as np
>>>
>>> with tempfile.TemporaryDirectory() as tmp:
...    A = tiledb.DenseArray.from_numpy(tmp, np.eye(4) * [1,2,3,4])
...    A.multi_index[1]
...    A.multi_index[1,1]
...    # return row 0 and 2
...    A.multi_index[[0,2]]
...    # return rows 0 and 2 intersecting column 2
...    A.multi_index[[0,2], 2]
...    # return rows 0:2 intersecting columns 0:2
...    A.multi_index[slice(0,2), slice(0,2)]
OrderedDict([('', array([[0., 2., 0., 0.]]))])
OrderedDict([('', array([[2.]]))])
OrderedDict([('', array([[1., 0., 0., 0.], [0., 0., 3., 0.]]))])
OrderedDict([('', array([[0.], [3.]]))])
OrderedDict([('', array([[1., 0., 0.], [0., 2., 0.], [0., 0., 3.]]))])
nattr

The number of attributes of this array.

ndim

The number of dimensions of this array.

nonempty_domain(self)

Return the minimum bounding domain which encompasses nonempty values.

Return type:tuple(tuple(numpy scalar, numpy scalar), ..)
Returns:A list of (inclusive) domain extent tuples, that contain all nonempty cells
reopen(self, timestamp=None)

Reopens this array.

This is useful when the array is updated after it was opened. To sync-up with the updates, the user must either close the array and open again, or just use reopen() without closing. reopen will be generally faster than a close-then-open.

schema

The ArraySchema for this array.

shape

The shape of this array.

subarray(self, selection, attrs=None, coords=False, order=None)
timestamp

Returns the timestamp the array is opened at

Return type:int
Returns:tiledb timestamp at which point the array was opened
uri

Returns the URI of the array

tiledb.consolidate(uri, key=None, Config config=None, Ctx ctx=None)

Consolidates TileDB array fragments for improved read performance

Parameters:
  • uri (str) – URI to the TileDB Array
  • key (str) – (default None) Key to decrypt array if the array is encrypted
  • config (tiledb.Config) – The TileDB Config with consolidation parameters set
  • ctx (tiledb.Ctx) – (default None) The TileDB Context
Return type:

str or bytes
Returns:

path (URI) to the consolidated TileDB Array
Raises:

TypeError – cannot convert path to unicode string
Raises:

tiledb.TileDBError
tiledb.vacuum(uri, Config config=None, Ctx ctx=None)

Consolidate underlying array fragments after multiple writes.

Parameters:
  • uri (str) – URI of array to be vacuumed
  • config – Override the context configuration for vacuuming. Defaults to None, inheriting the context parameters.
  • (ctx – tiledb.Ctx, optional): Context. Defaults to tiledb.default_ctx().
Raises:

TypeError – cannot convert uri to unicode string
Raises:

tiledb.TileDBError

After consolidation, you may optionally remove the consolidated fragments with the “vacuum” step. This operation of this function is controlled by the sm.vacuum.mode parameter, which accepts the values fragments, fragment_meta, and array_meta.

Example:

>>> import tiledb, numpy as np
>>> import tempfile
>>> path = tempfile.mkdtemp()
>>> with tiledb.from_numpy(path, np.random.rand(4)) as A:
...     pass # make sure to close
>>> with tiledb.open(path, 'w') as A:
...     for i in range(4):
...         A[:] = np.ones(4, dtype=np.int64) * i
>>> paths = tiledb.VFS().ls(path)
>>> len(paths) # should be 13 (3 base files + 2*5 fragment+ok files)
13
>>> () ; tiledb.consolidate(path) ; () # doctest:+ELLIPSIS
(...)
>>> tiledb.vacuum(path)
>>> paths = tiledb.VFS().ls(path)
>>> len(paths) # should now be 5 (3 base files + 2 fragment+ok files)
5

Dense Array

class tiledb.DenseArray
__getitem__(selection)

Retrieve data cells for an item or region of the array.

Parameters:selection (tuple) – An int index, slice or tuple of integer/slice objects, specifying the selected subarray region for each dimension of the DenseArray.
Return type:numpy.ndarray or collections.OrderedDict
Returns:If the dense array has a single attribute than a Numpy array of corresponding shape/dtype is returned for that attribute. If the array has multiple attributes, a collections.OrderedDict is returned with dense Numpy subarrays for each attribute.
Raises:IndexError – invalid or unsupported index selection
Raises:tiledb.TileDBError

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     # Creates array 'array' on disk.
...     A = tiledb.DenseArray.from_numpy(tmp + "/array",  np.ones((100, 100)))
...     # Many aspects of Numpy's fancy indexing are supported:
...     A[1:10, ...].shape
...     A[1:10, 20:99].shape
...     A[1, 2].shape
(9, 100)
(9, 79)
()
>>> # Subselect on attributes when reading:
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(tiledb.Dim(domain=(0, 9), tile=2, dtype=np.uint64))
...     schema = tiledb.ArraySchema(domain=dom,
...         attrs=(tiledb.Attr(name="a1", dtype=np.int64),
...                tiledb.Attr(name="a2", dtype=np.int64)))
...     tiledb.DenseArray.create(tmp + "/array", schema)
...     with tiledb.DenseArray(tmp + "/array", mode='w') as A:
...         A[0:10] = {"a1": np.zeros((10)), "a2": np.ones((10))}
...     with tiledb.DenseArray(tmp + "/array", mode='r') as A:
...         # Access specific attributes individually.
...         A[0:5]["a1"]
...         A[0:5]["a2"]
array([0, 0, 0, 0, 0])
array([1, 1, 1, 1, 1])
__setitem__(selection, value)

Set / update dense data cells

Parameters:
  • selection (tuple) – An int index, slice or tuple of integer/slice objects, specifiying the selected subarray region for each dimension of the DenseArray.
  • value (dict or numpy.ndarray) – a dictionary of array attribute values, values must able to be converted to n-d numpy arrays. if the number of attributes is one, than a n-d numpy array is accepted.
Raises:
  • IndexError – invalid or unsupported index selection
  • ValueError – value / coordinate length mismatch
Raises:

tiledb.TileDBError

Example:

>>> import tiledb, numpy as np, tempfile
>>> # Write to single-attribute 2D array
>>> with tempfile.TemporaryDirectory() as tmp:
...     # Create an array initially with all zero values
...     with tiledb.DenseArray.from_numpy(tmp + "/array",  np.zeros((2, 2))) as A:
...         pass
...     with tiledb.DenseArray(tmp + "/array", mode='w') as A:
...         # Write to the single (anonymous) attribute
...         A[:] = np.array(([1,2], [3,4]))
>>>
>>> # Write to multi-attribute 2D array
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(
...         tiledb.Dim(domain=(0, 1), tile=2, dtype=np.uint64),
...         tiledb.Dim(domain=(0, 1), tile=2, dtype=np.uint64))
...     schema = tiledb.ArraySchema(domain=dom,
...         attrs=(tiledb.Attr(name="a1", dtype=np.int64),
...                tiledb.Attr(name="a2", dtype=np.int64)))
...     tiledb.DenseArray.create(tmp + "/array", schema)
...     with tiledb.DenseArray(tmp + "/array", mode='w') as A:
...         # Write to each attribute
...         A[0:2, 0:2] = {"a1": np.array(([-3, -4], [-5, -6])),
...                        "a2": np.array(([1, 2], [3, 4]))}
static from_numpy(uri, array, ctx=None, **kwargs)

DenseArrayImpl.from_numpy(uri, ndarray array, Ctx ctx=None, **kw)

Persists a given numpy array as a TileDB DenseArray, returns a readonly DenseArray class instance.

Parameters:
  • uri (str) – URI for the TileDB array resource
  • array (numpy.ndarray) – dense numpy array to persist
  • ctx (tiledb.Ctx) – A TileDB Context
  • **kwargs – additional arguments to pass to the DenseArray constructor
Return type:

tiledb.DenseArray
Returns:

An open DenseArray (read mode) with a single anonymous attribute
Raises:

TypeError – cannot convert uri to unicode string
Raises:

tiledb.TileDBError

Example:

>>> import tiledb, numpy as np, tempfile
>>> with tempfile.TemporaryDirectory() as tmp:
...     # Creates array 'array' on disk.
...     with tiledb.DenseArray.from_numpy(tmp + "/array",  np.array([1.0, 2.0, 3.0])) as A:
...         pass

Sparse Array

class tiledb.SparseArray
__getitem__(selection)

Retrieve nonempty cell data for an item or region of the array

Parameters:selection (tuple) – An int index, slice or tuple of integer/slice objects, specifying the selected subarray region for each dimension of the SparseArray.
Return type:collections.OrderedDict
Returns:An OrderedDict is returned with “coords” coordinate values being the first key. “coords” is a Numpy record array representation of the coordinate values of non-empty attribute cells. Nonempty attribute values are returned as Numpy 1-d arrays.
Raises:IndexError – invalid or unsupported index selection
Raises:tiledb.TileDBError

Example:

>>> import tiledb, numpy as np, tempfile
>>> # Write to multi-attribute 2D array
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(
...         tiledb.Dim(name="y", domain=(0, 9), tile=2, dtype=np.uint64),
...         tiledb.Dim(name="x", domain=(0, 9), tile=2, dtype=np.uint64))
...     schema = tiledb.ArraySchema(domain=dom, sparse=True,
...         attrs=(tiledb.Attr(name="a1", dtype=np.int64),
...                tiledb.Attr(name="a2", dtype=np.int64)))
...     tiledb.SparseArray.create(tmp + "/array", schema)
...     with tiledb.SparseArray(tmp + "/array", mode='w') as A:
...         # Write in the twp cells (0,0) and (2,3) only.
...         I, J = [0, 2], [0, 3]
...         # Write to each attribute
...         A[I, J] = {"a1": np.array([1, 2]),
...                    "a2": np.array([3, 4])}
...     with tiledb.SparseArray(tmp + "/array", mode='r') as A:
...         # Return an OrderedDict with values and coordinates
...         A[0:3, 0:10]
...         # Return just the "x" coordinates values
...         A[0:3, 0:10]["x"]
OrderedDict([('a1', array([1, 2])), ('a2', array([3, 4])), ('y', array([0, 2], dtype=uint64)), ('x', array([0, 3], dtype=uint64))])
array([0, 3], dtype=uint64)

With a floating-point array domain, index bounds are inclusive, e.g.:

>>> # Return nonempty cells within a floating point array domain (fp index bounds are inclusive):
>>> # A[5.0:579.9]
__setitem__(selection, value)

Set / update sparse data cells

Parameters:
  • selection (tuple) – N coordinate value arrays (dim0, dim1, …) where N in the ndim of the SparseArray, The format follows numpy sparse (point) indexing semantics.
  • value (dict or numpy.ndarray) – a dictionary of nonempty array attribute values, values must able to be converted to 1-d numpy arrays. if the number of attributes is one, than a 1-d numpy array is accepted.
Raises:
  • IndexError – invalid or unsupported index selection
  • ValueError – value / coordinate length mismatch
Raises:

tiledb.TileDBError

Example:

>>> import tiledb, numpy as np, tempfile
>>> # Write to multi-attribute 2D array
>>> with tempfile.TemporaryDirectory() as tmp:
...     dom = tiledb.Domain(
...         tiledb.Dim(domain=(0, 1), tile=2, dtype=np.uint64),
...         tiledb.Dim(domain=(0, 1), tile=2, dtype=np.uint64))
...     schema = tiledb.ArraySchema(domain=dom, sparse=True,
...         attrs=(tiledb.Attr(name="a1", dtype=np.int64),
...                tiledb.Attr(name="a2", dtype=np.int64)))
...     tiledb.SparseArray.create(tmp + "/array", schema)
...     with tiledb.SparseArray(tmp + "/array", mode='w') as A:
...         # Write in the corner cells (0,0) and (1,1) only.
...         I, J = [0, 1], [0, 1]
...         # Write to each attribute
...         A[I, J] = {"a1": np.array([1, 2]),
...                    "a2": np.array([3, 4])}

Object Management

tiledb.array_exists(uri, isdense=False, issparse=False)

Check if arrays exists and is open-able at the given URI

Optionally restrict to isdense or issparse array types.

tiledb.group_create(uri, Ctx ctx=None)

Create a TileDB Group object at the specified path (URI)

Parameters:
  • uri (str) – URI of the TileDB Group to be created
  • ctx (tiledb.Ctx) – The TileDB Context
Return type:

str
Returns:

The URI of the created TileDB Group
Raises:

TypeError – cannot convert path to unicode string
Raises:

tiledb.TileDBError
tiledb.object_type(uri, Ctx ctx=None)

Returns the TileDB object type at the specified path (URI)

Parameters:
  • path (str) – path (URI) of the TileDB resource
  • ctx (tiledb.Ctx) – The TileDB Context
Return type:

str
Returns:

object type string
Raises:

TypeError – cannot convert path to unicode string
tiledb.remove(uri, Ctx ctx=None)

Removes (deletes) the TileDB object at the specified path (URI)

Parameters:
  • uri (str) – URI of the TileDB resource
  • ctx (tiledb.Ctx) – The TileDB Context
Raises:

TypeError – uri cannot be converted to a unicode string
Raises:

tiledb.TileDBError
tiledb.move(old_uri, new_uri, Ctx ctx=None)

Moves a TileDB resource (group, array, key-value).

Parameters:
  • ctx (tiledb.Ctx) – The TileDB Context
  • old_uri (str) – path (URI) of the TileDB resource to move
  • new_uri (str) – path (URI) of the destination
Raises:

TypeError – uri cannot be converted to a unicode string
Raises:

TileDBError
tiledb.ls(path, func, Ctx ctx=None)

Lists TileDB resources and applies a callback that have a prefix of path (one level deep).

Parameters:
  • path (str) – URI of TileDB group object
  • func (function) – callback to execute on every listed TileDB resource, URI resource path and object type label are passed as arguments to the callback
  • ctx (tiledb.Ctx) – TileDB context
Raises:

TypeError – cannot convert path to unicode string
Raises:

tiledb.TileDBError
tiledb.walk(path, func, order='preorder', Ctx ctx=None)

Recursively visits TileDB resources and applies a callback to resources that have a prefix of path

Parameters:
  • path (str) – URI of TileDB group object
  • func (function) – callback to execute on every listed TileDB resource, URI resource path and object type label are passed as arguments to the callback
  • ctx (tiledb.Ctx) – The TileDB context
  • order (str) – ‘preorder’ (default) or ‘postorder’ tree traversal
Raises:
Raises:

tiledb.TileDBError

VFS

class tiledb.VFS(Config config=None, Ctx ctx=None)

TileDB VFS class

Encapsulates the TileDB VFS module instance with a specific configuration (config).

Parameters:
  • ctx (tiledb.Ctx) – The TileDB Context
  • config (tiledb.Config or dict) – Override ctx VFS configurations with updated values in config.
close(self, FileHandle fh)

Closes a VFS FileHandle object

Parameters:fh (FileHandle) – An opened VFS FileHandle
Return type:FileHandle
Returns:closed VFS FileHandle
Raises:tiledb.TileDBError
config(self)

Returns the Config instance associated with the VFS.

create_bucket(self, uri)

Create an object store bucket at the given URI

Parameters:uri (str) – full URI of bucket resource to be created.
Return type:str
Returns:created bucket URI
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
create_dir(self, uri)

Create a VFS directory at the given URI

Parameters:uri (str) – URI of directory to be created
Return type:str
Returns:URI of created VFS directory
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
dir_size(self, uri)

Returns the size (in bytes) of a VFS directory at the given URI

Parameters:uri (str) – URI of a VFS directory resource
Return type:int
Returns:dir size in number of bytes
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
empty_bucket(self, uri)

Empty an object store bucket of all objects at the given URI

This function blocks until all objects are verified to be removed from the given bucket.

Parameters:uri (str) – URI of bucket resource to be emptied
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
file_size(self, uri)

Returns the size (in bytes) of a VFS file at the given URI

Parameters:uri (str) – URI of a VFS file resource
Return type:int
Returns:file size in number of bytes
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
is_bucket(self, uri)

Returns True if the URI resource is a valid object store bucket

Parameters:uri (str) – URI of bucket resource
Return type:bool
Returns:True if given URI is a valid object store bucket, False otherwise
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
is_dir(self, uri)

Returns True if the given URI is a VFS directory object

Parameters:uri (str) – URI of the directory resource
Return type:bool
Returns:True if uri is a VFS directory, False otherwise
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
is_empty_bucket(self, uri)

Returns true if the object store bucket is empty (contains no objects).

If the bucket is versioned, this returns the status of the latest bucket version state.

Parameters:uri (str) – URI of bucket resource
Return type:bool
Returns:True if bucket at given URI is empty, False otherwise
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
is_file(self, uri)

Returns True if the given URI is a VFS file object

Parameters:uri (str) – URI of the file resource
Return type:bool
Returns:True if uri is a VFS file, False otherwise
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
ls(self, uri)

Lists contents of directory at the given URI. Raises TileDBError for non-existent directory.

Parameters:uri (str) – URI of a VFS directory resource
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
move_dir(self, old_uri, new_uri)

Moves a VFS dir from old URI to new URI

Parameters:
  • old_uri (str) – Existing VFS file or directory resource URI
  • new_uri (str) – URI to move existing VFS resource to
  • force (bool) – if VFS resource at new_uri exists, delete the resource and overwrite
Return type:

str
Returns:

new URI of VFS resource
Raises:

TypeError – cannot convert old_uri/new_uri to unicode string
Raises:

tiledb.TileDBError
move_file(self, old_uri, new_uri)

Moves a VFS file from old URI to new URI

Parameters:
  • old_uri (str) – Existing VFS file or directory resource URI
  • new_uri (str) – URI to move existing VFS resource to
  • force (bool) – if VFS resource at new_uri exists, delete the resource and overwrite
Return type:

str
Returns:

new URI of VFS resource
Raises:

TypeError – cannot convert old_uri/new_uri to unicode string
Raises:

tiledb.TileDBError
open(self, uri, mode='rb')

Opens a VFS file resource for reading / writing / appends at URI

If the file did not exist upon opening, a new file is created.

Parameters:
  • uri (str) – URI of VFS file resource
  • str (mode) – ‘rb’ for opening the file to read, ‘wb’ to write, ‘ab’ to append
Return type:

FileHandle
Returns:

VFS FileHandle
Raises:
Raises:

tiledb.TileDBError
read(self, FileHandle fh, offset, nbytes)

Read nbytes from an opened VFS FileHandle at a given offset

Parameters:
  • fh (FileHandle) – An opened VFS FileHandle in ‘r’ mode
  • offset (int) – offset position in bytes to read from
  • nbytes (int) – number of bytes to read
Return type:

bytes()
Returns:

read bytes
Raises:

tiledb.TileDBError
readinto(self, FileHandle fh, const unsigned char[:] buffer, offset, nbytes)

Read nbytes from an opened VFS FileHandle at a given offset into a preallocated bytes buffer

Parameters:
  • fh (FileHandle) – An opened VFS FileHandle in ‘r’ mode
  • buffer (bytes) – A preallocated bytes buffer object
  • offset (int) – offset position in bytes to read from
  • nbytes (int) – number of bytes to read
Returns:

bytes buffer
Raises:

ValueError – invalid offset or nbytes values
Raises:

tiledb.TileDBError
remove_bucket(self, uri)

Remove an object store bucket at the given URI

Parameters:uri (str) – URI of bucket resource to be removed.
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
..note:
Consistency is not enforced for bucket removal so although this function will return immediately on success, the actual removal of the bucket make take some (indeterminate) amount of time.
remove_dir(self, uri)

Removes a VFS directory at the given URI

Parameters:uri (str) – URI of the directory resource to remove
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
remove_file(self, uri)

Removes a VFS file at the given URI

Parameters:uri (str) – URI of a VFS file resource
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
supports(self, scheme)

Returns true if the given URI scheme (storage backend) is supported

Parameters:scheme (str) – scheme component of a VFS resource URI (ex. ‘file’ / ‘hdfs’ / ‘s3’)
Return type:bool
Returns:True if the linked libtiledb version supports the storage backend, False otherwise
Raises:ValueError – VFS storage backend is not supported
sync(self, FileHandle fh)

Sync / flush an opened VFS FileHandle to storage backend

Parameters:fh (FileHandle) – An opened VFS FileHandle in ‘w’ or ‘a’ mode
Raises:tiledb.TileDBError
touch(self, uri)

Creates an empty VFS file at the given URI

Parameters:uri (str) – URI of a VFS file resource
Return type:str
Returns:URI of touched VFS file
Raises:TypeError – cannot convert uri to unicode string
Raises:tiledb.TileDBError
write(self, FileHandle fh, buff)

Writes buffer to opened VFS FileHandle

Parameters:
  • fh (FileHandle) – An opened VFS FileHandle in ‘w’ mode
  • buff – a Python object that supports the byte buffer protocol
Raises:

TypeError – cannot convert buff to bytes
Raises:

tiledb.TileDBError

Version

tiledb.libtiledb.version()

Return the version of the linked libtiledb shared library

Return type:tuple
Returns:Semver version (major, minor, rev)

Statistics

tiledb.stats_enable()

Enable TileDB internal statistics.

tiledb.stats_disable()

Disable TileDB internal statistics.

tiledb.stats_reset()

Reset all TileDB internal statistics to 0.

tiledb.stats_dump()

Print all TileDB internal statistics values to standard output.