dantro.groups.labelled module

Implements the LabelledDataGroup, which allows to handle groups and containers that can be associated with further coordinates.

This imitates the xarray selection interface and provides a uniform interface to select data from these groups. Most importantly, it allows to combine all the data of one group, allowing to conveniently work with heterogeneously stored data.

class dantro.groups.labelled.LabelledDataGroup(*args, dims: Tuple[str] = None, allow_deep_selection: bool = None, **kwargs)

Bases: dantro.groups.ordered.OrderedDataGroup

A group that assumes that the members it contains can be labelled with dimension names and coordinates.

Such a group has the great benefit to provide a selection interface that works fully on the dimension labels and coordinates and can cooperate with the xarray selection interface, i.e. the sel and isel methods.

_NEW_CONTAINER_CLS

alias of dantro.containers.xrdatactr.XrDataContainer

LDG_ALLOW_DEEP_SELECTION = True

LDG_DIMS = ()

LDG_EXTRACT_COORDS_FROM = 'data'

LDG_COORDS_ATTR_PREFIX = 'ext_coords__'

LDG_COORDS_MODE_ATTR_PREFIX = 'ext_coords_mode__'

LDG_COORDS_MODE_DEFAULT = 'scalar'

LDG_STRICT_ATTR_CHECKING = False

LDG_COORDS_SEPARATOR_IN_NAME = ';'

__init__(*args, dims: Tuple[str] = None, allow_deep_selection: bool = None, **kwargs)

Initialize a LabelledDataGroup

Parameters

• *args – Passed on to OrderedDataGroup

• dims (TDims, optional) – The dimensions associated with this group. If not given, will use those defined in the LDG_DIMS class variable. These can not be changed afterwards!

• allow_deep_selection (bool, optional) – Whether to allow deep selection. If not given, will use the LDG_ALLOW_DEEP_SELECTION class variable’s value. Behaviour can be changed via the property of the same name.

• **kwargs – Passed on to OrderedDataGroup

property dims

The names of the group-level dimensions this group manages.

It _may_ contain dimensions that overlap with dimension names from the members; this is intentional.

property ndim

The rank of the space covered by the group-level dimensions.

property coords

Returns a dict-like container of group-level coordinates.

The coordinates are calculated by iterating over all members and aggregating their individual coordinates. Once the member map is available, information is retrieved from there rather than recalculating it.

property shape

Return the shape of the space covered by the group-level dimensions.

This will be calculated from the available coordinates. Once the member map is available, information is retrieved from there rather than recalculating it.

property allow_deep_selection

Whether deep selection is allowed.

property member_map

Returns an array that represents the space that the members of this group span, where each value (i.e. a specific coordinate combination) is the name of the corresponding member of this group.

Upon first call, this is computed here. If members are added, it is tried to accomodate them in there; if not possible, the cache will be invalidated.

The member map _may_ include empty strings, i.e. coordinate combinations that are not covered by any member. Also, they can contain duplicate names, as one member can cover multiple coordinates.

Note

The member map is invalidated when new members are added that can not be accomodated in it. It will be recalculated when needed.

property member_map_available

Whether the member map is available yet.

isel(indexers: dict = None, *, drop: bool = False, combination_method: str = 'try_concat', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with an index-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Parameters

• indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick indices. As xr.DataArray.sel, uses pandas-like indexing, i.e.: slices include the terminal value.

• drop (bool, optional) – Drop coordinate variables instead of making them scalar.

• combination_method (str, optional) –

  How to combine group-level data with member-level data. Can be:

  • concat: Concatenate. This can preserve the dtype, but

    requires that no data is missing.

  • merge: Merge, using xarray.merge. This leads to a

    type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

  • try_concat: Try concatenation, fall back to merging

    if that was unsuccessful.

• deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

• **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

sel(indexers: dict = None, *, method: str = None, tolerance: float = None, drop: bool = False, combination_method: str = 'try_concat', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with a coordinate-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Parameters

• indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick labels. As xr.DataArray.sel, uses pandas-like indexing, i.e.: slices include the terminal value.

• method (str, optional) – Method to use for inexact matches

• tolerance (float, optional) – Maximum (absolute) distance between original and given label for inexact matches.

• drop (bool, optional) – Drop coordinate variables instead of making them scalar.

• combination_method (str, optional) –

  How to combine group-level data with member-level data. Can be:

  • concat: Concatenate. This can preserve the dtype, but

    requires that no data is missing.

  • merge: Merge, using xarray.merge. This leads to a

    type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

  • try_concat: Try concatenation, fall back to merging

    if that was unsuccessful.

• deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

• **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

_get_coords_of(obj: dantro.abc.AbstractDataContainer, *, mode=None) → Dict[str, Sequence[TCoord]]

Extract the coordinates for the given object using the dantro.utils.coords.extract_coords function.

Parameters

• obj (AbstractDataContainer) – The object to get the coordinates of.

• mode (None, optional) – By which coordiante extraction mode to get the coordinates from the object. Can be attrs, name, data or anything else specified in ~`dantro.utils.coords.extract_coords`.

Returns

The extracted coordinates

Return type

TCoordsDict

_add_container_callback(cont: dantro.abc.AbstractDataContainer) → None

Called by the base class after adding a container, this method checks whether the member map needs to be invalidated or whether the new container can be accomodated in it.

If it can be accomodated, the member map will be adjusted such that for all coordinates associated with the given cont, the member map points to the newly added container.

Parameters

cont (AbstractDataContainer) – The newly added container

Returns

Description

Return type

None

_parse_indexers(indexers: dict, *, allow_deep: bool, **indexers_kwargs) → Tuple[dict, dict]

Parses the given indexer arguments and split them into indexers for the selection of group members and deep selection.

Parameters

• indexers (dict) – The indexers dict, may be empty

• allow_deep (bool) – Whether to allow deep selection

• **indexers_kwargs – Additional indexers

Returns

(shallow indexers, deep indexers)

Return type

Tuple[dict, dict]

Raises

ValueError – If deep indexers were given but deep selection was not enabled

_combine(cont_names: xarray.core.dataarray.DataArray, *, combination_method: str, deep_indexers: dict, by_index: bool, **sel_kwargs) → xarray.core.dataset.Dataset

Combine the given objects by the specified method. If deep indexers are given, apply the deep indexing on each of the members.

This method receives a labelled array of container names, on which the selection already took place. The aim is now to align the objects these names refer to, including their coordinates, and thereby construct an array that contains both the dimensions given by the cont_names array and each members’ data dimensions.

Available combination methods are based either on xarray.merge operations or xarray.concat along each dimension. For both these combination methods, the members of this group need to be prepared such that the operation can be applied, i.e.: they need to already be in an array capable of that operation and they need to directly or indirectly preserve coordinate information.

For that purpose, an object-array is constructed that has the same shape as the given cont_names. As the xarray.Dataset and xarray.DataArray types have issues with handling array-like objects in object arrays, this is done via a numpy.ndarray.

Parameters

• cont_names (xr.DataArray) – The pre-selected member map object, i.e. a labelled array containing names of the desired members that are to be combined.

• combination_method (str) – How to combine them: concat, try_concat, or merge. Concatenation will allow preserving the dtype of the underlying data.

• deep_indexers (dict) – Whether any further indexing is to take place before combination.

• by_index (bool) – Whether the deep indexing should take place by index; if False, will use label-based selection.

• **sel_kwargs – Passed on to .sel or .isel.

Returns

The data of the members from cont_names, combined

using the given combination method.

Return type

xr.Dataset

Raises

• ValueError – Invalid combination method

• KeyError – In concat mode, upon missing members.

_ALLOWED_CONT_TYPES = None

_ATTRS_CLS

alias of dantro.base.BaseDataAttrs

_COND_TREE_CONDENSE_THRESH = 10

_COND_TREE_MAX_LEVEL = 10

_LockDataMixin__locked = False

_MutableMapping__marker = <object object>

_NEW_GROUP_CLS = None

_STORAGE_CLS

alias of collections.OrderedDict

__contains__(cont: Union[str, dantro.base.BaseDataContainer]) → bool

Whether the given container is in this group or not.

Parameters

cont (Union[str, BaseDataContainer]) – The name of the container or an object reference.

Returns

Whether the given container is in this group.

Return type

bool

__delitem__(key: str) → None

Deletes an item from the group

__format__(spec_str: str) → str

Creates a formatted string from the given specification.

Invokes further methods which are prefixed by _format_.

__getitem__(key: Union[str, List[str]])

Returns the container in this group with the given name.

Parameters

key (Union[str, List[str]]) – The object to retrieve. If this is a path, will recurse down until at the end.

Returns

The object at key

Raises

KeyError – If no such key can be found

__iter__()

Returns an iterator over the OrderedDict

__len__() → int

The length of the data.

__repr__() → str

Same as __str__

__setitem__(key: Union[str, List[str]], val: dantro.base.BaseDataContainer) → None

This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!

Parameters

• key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.

• val (BaseDataContainer) – The value to set

Returns

None

Raises

ValueError – If trying to add an element to this group, which should be done via the add method.

__sizeof__() → int

Returns the size of the data (in bytes) stored in this container’s data and its attributes.

Note that this value is approximate. It is computed by calling the sys.getsizeof function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.

Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.

For more information, see the documentation of sys.getsizeof:

https://docs.python.org/3/library/sys.html#sys.getsizeof

__str__() → str

An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.

_abc_impl = <_abc_data object>

_add_container(cont, *, overwrite: bool)

Private helper method to add a container to this group.

_add_container_to_data(cont) → None

Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …

NOTE This method should NEVER be called on its own, but only via the

_add_container method, which takes care of properly linking the container that is to be added.

NOTE After adding, the container need be reachable under its .name!

Parameters

cont – The container to add

_attrs = None

_check_cont(cont) → None

Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.

This is not expected to return, but can raise errors, if something did not work out as expected.

Parameters

cont – The container to check

_check_data(data: Any) → None

This method can be used to check the data provided to this container

It is called before the data is stored in the __init__ method and should raise an exception or create a warning if the data is not as desired.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

NOTE The CheckDataMixin provides a generalised implementation of this

method to perform some type checks and react to unexpected types.

Parameters

data (Any) – The data to check

_check_name(new_name: str) → None

Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Parameters

new_name (str) – The new name, which is to be checked.

classmethod _combine_by_merge(dsets: numpy.ndarray) → xarray.core.dataset.Dataset

Combine the given datasets by merging using xarray.merge.

Parameters

dsets (np.ndarray) – The object-dtype array of xr.Datasets that are to be combined.

Returns

All datasets, aligned and combined via xarray.merge

Return type

xr.Dataset

_format_cls_name() → str

A __format__ helper function: returns the class name

_format_info() → str

A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!

_format_logstr() → str

A __format__ helper function: returns the log string, a combination of class name and name

_format_name() → str

A __format__ helper function: returns the name

_format_path() → str

A __format__ helper function: returns the path to this container

_format_tree() → str

Returns the default tree representation of this group by invoking the .tree property

_format_tree_condensed() → str

Returns the default tree representation of this group by invoking the .tree property

_ipython_key_completions_() → List[str]

For ipython integration, return a list of available keys

_link_child(*, new_child: dantro.base.BaseDataContainer, old_child: dantro.base.BaseDataContainer = None)

Links the new_child to this class, unlinking the old one.

This method should be called from any method that changes which items are associated with this group.

_lock_hook()

Invoked upon locking.

_tree_repr(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]

Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.

Parameters

• level (int, optional) – The depth within the tree

• max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.

• info_fstr (str, optional) – The format string for the info string

• info_ratio (float, optional) – The width ratio of the whole line width that the info string takes

• condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.

• total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.

Returns

The (multi-line) tree representation of

this group. If this method was invoked with level == 0, a string will be returned; otherwise, a list of strings will be returned.

Return type

Union[str, List[str]]

_unlink_child(child: dantro.base.BaseDataContainer)

Unlink a child from this class.

This method should be called from any method that removes an item from this group, be it through deletion or through

_unlock_hook()

Invoked upon unlocking.

add(*conts, overwrite: bool = False)

Add the given containers to this group.

property attrs

The container attributes.

property classname

Returns the name of this DataContainer-derived class

clear() → None. Remove all items from D.

property data

The stored data.

get(key, default=None)

Return the container at key, or default if container with name key is not available.

items()

Returns an iterator over the (name, data container) tuple of this group.

keys()

Returns an iterator over the container names in this group.

lock()

Locks the data of this object

property locked

Whether this object is locked

property logstr

Returns the classname and name of this object; a combination often used in logging…

property name

The name of this DataContainer-derived object.

new_container(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new container of class Cls and adds it at the given path relative to this group.

Parameters

• path (Union[str, list]) – Where to add the container. Note that the intermediates of this path need to already exist.

• Cls (type, optional) – The class of the container to add. If None, the _NEW_CONTAINER_CLS class variable’s value is used; if not given, this will raise a ValueError.

• **kwargs – kwargs to pass on to Cls.__init__

Returns

the created container

Return type

Cls

Raises

• KeyError – When intermediate groups to path are missing

• TypeError – When the given Cls is invalid

new_group(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new group at the given path.

Parameters

• path (Union[str, list]) – The path to create the group at. Note that the whole intermediate path needs to already exist.

• Cls (type, optional) – If given, use this type to create the group. If not given, uses the class specified in the _NEW_GROUP_CLS class variable or, as last resort, the type of this instance.

• **kwargs – Passed on to Cls.__init__

Returns

the created group

Return type

Cls

Raises

TypeError – For the given class not being derived from BaseDataGroup

property parent

The associated parent of this container or group

property path

The path to get to this container or group from some root path

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

raise_if_locked(*, prefix: str = None)

Raises an exception if this object is locked; does nothing otherwise

recursive_update(other)

Recursively updates the contents of this data group with the entries of the given data group

NOTE This will create shallow copies of those elements in other that are added to this object.

Parameters

other (BaseDataGroup) – The group to update with

Raises

TypeError – If other was of invalid type

setdefault(key, default=None)

This method is not supported for a data group

property tree

Returns the default (full) tree representation of this group

property tree_condensed

Returns the condensed tree representation of this group. Uses the _COND_TREE_* prefixed class attributes as parameters.

unlock()

Unlocks the data of this object

update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Returns an iterator over the containers in this group.

classmethod _combine_by_concatenation(dsets: numpy.ndarray, *, dims: Tuple[str]) → xarray.core.dataset.Dataset

Combine the given datasets by concatenation using xarray.concat and subsequent application along all dimensions specified in dims.

Parameters

• dsets (np.ndarray) – The object-dtype array of xr.Dataset objects that are to be combined by concatenation.

• dims (TDims) – The dimension names corresponding to _all_ the dimensions of the dsets array.

Returns

The dataset resulting from the concatenation

Return type

xr.Dataset