dantro.groups.time_series module

Implements LabelledDataGroup specializations for time series

class dantro.groups.time_series.TimeSeriesGroup(*args, dims: Tuple[str] = None, mode: str = None, allow_deep_selection: bool = None, **kwargs)[source]

Bases: dantro.groups.labelled.LabelledDataGroup, dantro.groups.ordered.IndexedDataGroup

A time-series group assumes that each stored member refers to one point in time, where the name is to be interpreted as the time coordinate.

For more information on selection methods, see:

LDG_DIMS = ('time',)
LDG_EXTRACT_COORDS_FROM = 'name'
LDG_ALLOW_DEEP_SELECTION = True
LDG_COORDS_ATTR_PREFIX = 'ext_coords__'
LDG_COORDS_MODE_ATTR_PREFIX = 'ext_coords_mode__'
LDG_COORDS_MODE_DEFAULT = 'scalar'
LDG_COORDS_SEPARATOR_IN_NAME = ';'
LDG_STRICT_ATTR_CHECKING = False
_ALLOWED_CONT_TYPES = None
_ATTRS_CLS

alias of dantro.base.BaseDataAttrs

_COLLECTIVE_SELECT_THRESHOLD = 1.8
_COND_TREE_CONDENSE_THRESH = 10
_COND_TREE_MAX_LEVEL = 10
_DirectInsertionModeMixin__in_direct_insertion_mode = False
_IndexedDataGroup__last_keys = None
_LockDataMixin__locked = False
_MutableMapping__marker = <object object>
_NEW_CONTAINER_CLS

alias of dantro.containers.xrdatactr.XrDataContainer

_NEW_GROUP_CLS

alias of dantro.groups.ordered.OrderedDataGroup

_STORAGE_CLS

alias of dantro.utils.ordereddict.IntOrderedDict

__contains__(key: Union[str, int]) → bool

Adjusts the parent method to allow checking for integers

__delitem__(key: Union[str, int])

Adjusts the parent method to allow item deletion by integer key

__eq__(other) → bool

Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the _data and _attrs attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.

If types do not match exactly, NotImplemented is returned, thus referring the comparison to the other side of the ==.

__format__(spec_str: str) → str

Creates a formatted string from the given specification.

Invokes further methods which are prefixed by _format_.

__getitem__(key: Union[str, int])

Adjusts the parent method to allow integer key item access

__init__(*args, dims: Tuple[str] = None, mode: str = None, allow_deep_selection: bool = None, **kwargs)

Initialize a LabelledDataGroup

Parameters
  • *args – Passed on to OrderedDataGroup

  • dims (TDims, optional) – The dimensions associated with this group. If not given, will use those defined in the LDG_DIMS class variable. These can not be changed afterwards!

  • mode (str, optional) – By which coordinate extraction mode to get the coordinates from the group members. Can be attrs, name, data or anything else specified in extract_coords().

  • allow_deep_selection (bool, optional) – Whether to allow deep selection. If not given, will use the LDG_ALLOW_DEEP_SELECTION class variable’s value. Behaviour can be changed via the property of the same name.

  • **kwargs – Passed on to OrderedDataGroup

__iter__()

Returns an iterator over the OrderedDict

__len__() → int

The number of members in this group.

__repr__() → str

Same as __str__

__setitem__(key: Union[str, int])

Adjusts the parent method to allow item setting by integer key

__sizeof__() → int

Returns the size of the data (in bytes) stored in this container’s data and its attributes.

Note that this value is approximate. It is computed by calling the sys.getsizeof function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.

Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.

For more information, see the documentation of sys.getsizeof:

__str__() → str

An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.

_abc_impl = <_abc_data object>
_add_container(cont, *, overwrite: bool)

Private helper method to add a container to this group.

_add_container_callback(cont: dantro.abc.AbstractDataContainer) → None

Called by the base class after adding a container, this method checks whether the member map needs to be invalidated or whether the new container can be accomodated in it.

If it can be accomodated, the member map will be adjusted such that for all coordinates associated with the given cont, the member map points to the newly added container.

Parameters

cont (AbstractDataContainer) – The newly added container

_add_container_to_data(cont) → None

Adds a container to the underlying integer-ordered dictionary.

Unlike the parent method, this uses insert() in order to provide hints regarding the insertion position. It is optimised for insertion in ascending order.

_attrs = None
_check_cont(cont) → None

Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.

This is not expected to return, but can raise errors, if something did not work out as expected.

Parameters

cont – The container to check

_check_data(data: Any) → None

This method can be used to check the data provided to this container

It is called before the data is stored in the __init__ method and should raise an exception or create a warning if the data is not as desired.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Note

The CheckDataMixin provides a generalised implementation of this method to perform some type checks and react to unexpected types.

Parameters

data (Any) – The data to check

_check_name(new_name: str) → None

Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Parameters

new_name (str) – The new name, which is to be checked.

classmethod _combine_by_concatenation(dsets: numpy.ndarray, *, dims: Tuple[str]) → xarray.core.dataset.Dataset

Combine the given datasets by concatenation using xarray.concat and subsequent application along all dimensions specified in dims.

Parameters
  • dsets (np.ndarray) – The object-dtype array of xr.Dataset objects that are to be combined by concatenation.

  • dims (TDims) – The dimension names corresponding to _all_ the dimensions of the dsets array.

Returns

The dataset resulting from the concatenation

Return type

xr.Dataset

classmethod _combine_by_merge(dsets: numpy.ndarray) → xarray.core.dataset.Dataset

Combine the given datasets by merging using xarray.merge.

Parameters

dsets (np.ndarray) – The object-dtype array of xr.Datasets that are to be combined.

Returns

All datasets, aligned and combined via xarray.merge

Return type

xr.Dataset

_direct_insertion_mode(*, enabled: bool = True)

A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the with_direct_insertion() property will return true.

This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering, _enter_direct_insertion_mode() is called. After exiting, _exit_direct_insertion_mode() is called.

Parameters

enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.

_enter_direct_insertion_mode()

Called after entering direct insertion mode; can be overwritten to attach additional behaviour.

_exit_direct_insertion_mode()

Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.

_format_cls_name() → str

A __format__ helper function: returns the class name

_format_info() → str

A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!

_format_logstr() → str

A __format__ helper function: returns the log string, a combination of class name and name

_format_name() → str

A __format__ helper function: returns the name

_format_path() → str

A __format__ helper function: returns the path to this container

_format_tree() → str

Returns the default tree representation of this group by invoking the .tree property

_format_tree_condensed() → str

Returns the default tree representation of this group by invoking the .tree property

_get_cont(name: str, *, combination_method: str) → Optional[dantro.containers.xrdatactr.XrDataContainer]

Retrieve the container from the group. If no container could be found, returns None, which denotes that further processing should be skipped.

Parameters
  • name (str) – Name of the container to be extracted

  • combination_method (str) – How the container data will be combined

Returns

The extracted container

Return type

Union[XrDataContainer, None]

Raises

ItemAccessError – If combination_method == "concat", on invalid container name.

_get_coords_of(obj: dantro.abc.AbstractDataContainer) → Dict[str, Sequence[TCoord]]

Extract the coordinates for the given object using the extract_coords() function.

Parameters

obj (AbstractDataContainer) – The object to get the coordinates of.

Returns

The extracted coordinates

Return type

TCoordsDict

_ipython_key_completions_() → List[int]

For ipython integration, return a list of available keys.

Unlike the BaseDataGroup method, which returns a list of strings, this returns a list of integers.

Links the new_child to this class, unlinking the old one.

This method should be called from any method that changes which items are associated with this group.

_lock_hook()

Invoked upon locking.

_parse_indexers(indexers: dict, *, allow_deep: bool, **indexers_kwargs) → Tuple[dict, dict]

Parses the given indexer arguments and split them into indexers for the selection of group members and deep selection.

Parameters
  • indexers (dict) – The indexers dict, may be empty

  • allow_deep (bool) – Whether to allow deep selection

  • **indexers_kwargs – Additional indexers

Returns

(shallow indexers, deep indexers)

Return type

Tuple[dict, dict]

Raises

ValueError – If deep indexers were given but deep selection was not enabled

_parse_key(key: Union[str, int]) → str

Makes sure a key is a string

_process_cont(cont, *, coords, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Process the given container and coordinates into a data array; this applies selection along container dimensions that overlap with the group dimensions as well as deep selection.

Parameters
  • cont – The container to be processed

  • coords – The DataArrayCoordinates of the given container in the preselected member map.

  • shallow_indexers (dict) – Indexers that were used to preselect the member map.

  • deep_indexers (dict) – Indexers to be applied to the container

  • by_index (bool) – Whether to select by index

  • drop (bool) – Whether to drop coordinate variables instead of making them scalar.

  • **sel_kwargs – Passed to .sel.

Returns

The processed container data

Return type

xr.DataArray

Raises

ValueError – In name mode, on conflicting non-dimension container coordinates.

_select(*, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Preselect the member map (if needed) and designate a suitable method for further processing and selection based on the given combination method and indexers.

If possible, take shortcuts when selecting all data or when selecting data from a single group member.

Parameters
  • combination_method (str) – How to combine the member data.

  • shallow_indexers (dict) – Indexers to be applied on the group-level.

  • deep_indexers (dict) – Indexers to be applied on the member-level only.

  • by_index (bool) – Whether to select by index.

  • drop (bool) – Whether to drop coordinate variables instead of making them scalar.

  • **sel_kwargs – Passed to .sel.

Returns

The selected data.

Return type

xr.DataArray

Raises

ValueError – On invalid combination_method.

_select_all_merge() → xarray.core.dataarray.DataArray

Select all group data by directly merging all containers. This circumvents building the member map. This might fail, e.g. if there are conflicting or duplicate coordinates.

_select_generic(cont_names: xarray.core.dataarray.DataArray, *, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Select data from group members using the given indexers and combine it via the specified method. If deep indexers are given, apply the deep indexing on each of the members.

This method receives a labelled array of container names, on which the selection already took place. The aim is now to align the objects these names refer to, including their coordinates, and thereby construct an array that contains both the dimensions given by the cont_names array and each members’ data dimensions.

Available combination methods are based either on xarray.merge operations or xarray.concat along each dimension. For both these combination methods, the members of this group need to be prepared such that the operation can be applied, i.e.: they need to already be in an array capable of that operation and they need to directly or indirectly preserve coordinate information.

For that purpose, an object-array is constructed holding the processed member data. As the xarray.Dataset and xarray.DataArray types have issues with handling array-like objects in object arrays, this is done via a numpy.ndarray.

Parameters
  • cont_names (xr.DataArray) – The pre-selected member map object, i.e. a labelled array containing names of the desired members that are to be combined.

  • combination_method (str) – How to combine them: concat, try_concat, or merge. Concatenation will allow preserving the dtype of the underlying data.

  • shallow_indexers (dict) – Indexer arguments that were used for the group member selection.

  • deep_indexers (dict) – Indexer arguments for deep selection to be done before combination.

  • by_index (bool) – Whether the deep indexing should take place by index; if False, will use label-based selection.

  • **sel_kwargs – Passed on to .sel.

Returns

The selected data of the members from cont_names,

combined using the given combination method.

Return type

xr.Dataset

Raises
  • ValueError – On conflicting coordinate information on group-level and member-level.

  • KeyError – In concat mode, upon missing members.

_select_single(cont_names: xarray.core.dataarray.DataArray, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Select data from a single group member. Expects the preselected member map to contain only a single valid container name.

_tree_repr(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]

Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.

Parameters
  • level (int, optional) – The depth within the tree

  • max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.

  • info_fstr (str, optional) – The format string for the info string

  • info_ratio (float, optional) – The width ratio of the whole line width that the info string takes

  • condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.

  • total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.

Returns

The (multi-line) tree representation of

this group. If this method was invoked with level == 0, a string will be returned; otherwise, a list of strings will be returned.

Return type

Union[str, List[str]]

Unlink a child from this class.

This method should be called from any method that removes an item from this group, be it through deletion or through

_unlock_hook()

Invoked upon unlocking.

add(*conts, overwrite: bool = False)

Add the given containers to this group.

property allow_deep_selection

Whether deep selection is allowed.

property attrs

The container attributes.

property classname

Returns the name of this DataContainer-derived class

clear()

Clears all containers from this group.

This is done by unlinking all children and then overwriting _data with an empty _STORAGE_CLS object.

property coords

Returns a dict-like container of group-level coordinate values keyed by dimension.

property data

The stored data.

property dims

The names of the group-level dimensions this group manages.

It _may_ contain dimensions that overlap with dimension names from the members; this is intentional.

get(key, default=None)

Return the container at key, or default if container with name key is not available.

isel(indexers: dict = None, *, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with an index-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Note

For data combination (via any combination_method) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using the respective coordinates. See the xarray documentation for more information about coordinates.

Parameters
  • indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick indices. As xr.DataArray.isel, uses pandas-like indexing, i.e.: slices do not include the terminal value.

  • drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.

  • combination_method (str, optional) –

    How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:

    • concat: Concatenate. This can preserve the dtype, but requires that no data is missing.

    • merge: Merge, using xarray.merge. This leads to a type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

    • try_concat: Try concatenation, fall back to merging if that was unsuccessful.

    • auto: Automatically deduce suitably combination method. Use merge if data is non-integer type and try_concat otherwise.

    Note

    Selecting all data (by not passing any indexers) can be significantly faster using the merge combination method than using the concat method.

  • deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

  • **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

items()

Returns an iterator over the (name, data container) tuple of this group.

key_at_idx(idx: int) → str

Get a key by its index within the container. Can be negative.

Parameters

idx (int) – The index within the member sequence

Returns

The desired key

Return type

str

Raises

IndexError – Index out of range

keys()

Returns an iterator over the container names in this group.

keys_as_int() → Generator[int, None, None]

Returns an iterator over keys as integer values

lock()

Locks the data of this object

property locked

Whether this object is locked

property logstr

Returns the classname and name of this object

property member_map

Returns an array that represents the space that the members of this group span, where each value (i.e. a specific coordinate combination) is the name of the corresponding member of this group.

Upon first call, this is computed here. If members are added, it is tried to accomodate them in there; if not possible, the cache will be invalidated.

The member map _may_ include empty strings, i.e. coordinate combinations that are not covered by any member. Also, they can contain duplicate names, as one member can cover multiple coordinates.

Note

The member map is invalidated when new members are added that can not be accomodated in it. It will be recalculated when needed.

property member_map_available

Whether the member map is available yet.

property name

The name of this DataContainer-derived object.

property ndim

The rank of the space covered by the group-level dimensions.

new_container(path: Union[str, List[str]], *, Cls: type = None, **kwargs)

Creates a new container of class Cls and adds it at the given path relative to this group.

If needed, intermediate groups are automatically created.

Parameters
  • path (Union[str, List[str]]) – Where to add the container.

  • Cls (type, optional) – The class of the container to add. If None, the _NEW_CONTAINER_CLS class variable’s value is used.

  • **kwargs – passed on to Cls.__init__

Returns

the created container

Return type

Cls

Raises
  • ValueError – If neither the Cls argument nor the class variable _NEW_CONTAINER_CLS were set or if path was empty.

  • TypeError – When Cls is not compatible to the data tree

new_group(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new group at the given path.

Parameters
  • path (Union[str, list]) – The path to create the group at. Note that the whole intermediate path needs to already exist.

  • Cls (type, optional) – If given, use this type to create the group. If not given, uses the class specified in the _NEW_GROUP_CLS class variable or, as last resort, the type of this instance.

  • **kwargs – Passed on to Cls.__init__

Returns

the created group

Return type

Cls

Raises

TypeError – For the given class not being derived from BaseDataGroup

property parent

The associated parent of this container or group

property path

The path to get to this container or group from some root path

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

raise_if_locked(*, prefix: str = None)

Raises an exception if this object is locked; does nothing otherwise

recursive_update(other, *, overwrite: bool = True)

Recursively updates the contents of this data group with the entries of the given data group

Note

This will create shallow copies of those elements in other that are added to this object.

Parameters
  • other (BaseDataGroup) – The group to update with

  • overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.

Raises

TypeError – If other was of invalid type

sel(indexers: dict = None, *, method: str = None, tolerance: float = None, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with a coordinate-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Note

For data combination (via any combination_method) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using the respective coordinates. See the xarray documentation for more information about coordinates.

Parameters
  • indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick labels. As xr.DataArray.sel, uses pandas-like indexing, i.e.: slices include the terminal value.

  • method (str, optional) – Method to use for inexact matches

  • tolerance (float, optional) – Maximum (absolute) distance between original and given label for inexact matches.

  • drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.

  • combination_method (str, optional) –

    How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:

    • concat: Concatenate. This can preserve the dtype, but requires that no data is missing.

    • merge: Merge, using xarray.merge. This leads to a type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

    • try_concat: Try concatenation, fall back to merging if that was unsuccessful.

    • auto: Automatically deduce suitably combination method. Use merge if data is non-integer type and try_concat otherwise.

    Note

    Selecting all data (by not passing any indexers) can be significantly faster using the merge combination method than using the concat method.

  • deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

  • **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

setdefault(key, default=None)

This method is not supported for a data group

property shape

Return the shape of the space covered by the group-level dimensions.

property tree

Returns the default (full) tree representation of this group

property tree_condensed

Returns the condensed tree representation of this group. Uses the _COND_TREE_* prefixed class attributes as parameters.

unlock()

Unlocks the data of this object

update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Returns an iterator over the containers in this group.

property with_direct_insertion

Whether the class this mixin is mixed into is currently in direct insertion mode.

class dantro.groups.time_series.HeterogeneousTimeSeriesGroup(*args, dims: Tuple[str] = None, mode: str = None, allow_deep_selection: bool = None, **kwargs)[source]

Bases: dantro.groups.time_series.TimeSeriesGroup

This extends the TimeSeriesGroup by configuring it such that it retrieves its coordinates not from the name of the members contained in it but from _their_ data.

It still manages only the time dimension, which is now overlapping with the time dimension in the members of this group. However, the py:class:~dantro.groups.labelled.LabelledDataGroup can handle this overlap and provides a uniform selection interface that allows combining this heterogeneously stored data.

This becomes especially useful in cases where the members of this group store data with the following properties:

  • Potentially different coordiantes than the coordinates of other members of the group.

  • Containing time information for more than a single time coordinate

  • No guarantee for overlaps between time dimension or any other dimension.

As such it is suitable to work with data that represents ensembles that frequently change not only their size but also their identifying labels. Additionally, it supports them not being stored in regular intervals but only upon a change in coordinates.

LDG_EXTRACT_COORDS_FROM = 'data'
LDG_ALLOW_DEEP_SELECTION = True
LDG_COORDS_ATTR_PREFIX = 'ext_coords__'
LDG_COORDS_MODE_ATTR_PREFIX = 'ext_coords_mode__'
LDG_COORDS_MODE_DEFAULT = 'scalar'
LDG_COORDS_SEPARATOR_IN_NAME = ';'
LDG_DIMS = ('time',)
LDG_STRICT_ATTR_CHECKING = False
_ALLOWED_CONT_TYPES = None
_ATTRS_CLS

alias of dantro.base.BaseDataAttrs

_COLLECTIVE_SELECT_THRESHOLD = 1.8
_COND_TREE_CONDENSE_THRESH = 10
_COND_TREE_MAX_LEVEL = 10
_DirectInsertionModeMixin__in_direct_insertion_mode = False
_IndexedDataGroup__last_keys = None
_LockDataMixin__locked = False
_MutableMapping__marker = <object object>
_NEW_CONTAINER_CLS

alias of dantro.containers.xrdatactr.XrDataContainer

_NEW_GROUP_CLS

alias of dantro.groups.ordered.OrderedDataGroup

_STORAGE_CLS

alias of dantro.utils.ordereddict.IntOrderedDict

__contains__(key: Union[str, int]) → bool

Adjusts the parent method to allow checking for integers

__delitem__(key: Union[str, int])

Adjusts the parent method to allow item deletion by integer key

__eq__(other) → bool

Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the _data and _attrs attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.

If types do not match exactly, NotImplemented is returned, thus referring the comparison to the other side of the ==.

__format__(spec_str: str) → str

Creates a formatted string from the given specification.

Invokes further methods which are prefixed by _format_.

__getitem__(key: Union[str, int])

Adjusts the parent method to allow integer key item access

__init__(*args, dims: Tuple[str] = None, mode: str = None, allow_deep_selection: bool = None, **kwargs)

Initialize a LabelledDataGroup

Parameters
  • *args – Passed on to OrderedDataGroup

  • dims (TDims, optional) – The dimensions associated with this group. If not given, will use those defined in the LDG_DIMS class variable. These can not be changed afterwards!

  • mode (str, optional) – By which coordinate extraction mode to get the coordinates from the group members. Can be attrs, name, data or anything else specified in extract_coords().

  • allow_deep_selection (bool, optional) – Whether to allow deep selection. If not given, will use the LDG_ALLOW_DEEP_SELECTION class variable’s value. Behaviour can be changed via the property of the same name.

  • **kwargs – Passed on to OrderedDataGroup

__iter__()

Returns an iterator over the OrderedDict

__len__() → int

The number of members in this group.

__repr__() → str

Same as __str__

__setitem__(key: Union[str, int])

Adjusts the parent method to allow item setting by integer key

__sizeof__() → int

Returns the size of the data (in bytes) stored in this container’s data and its attributes.

Note that this value is approximate. It is computed by calling the sys.getsizeof function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.

Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.

For more information, see the documentation of sys.getsizeof:

__str__() → str

An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.

_abc_impl = <_abc_data object>
_add_container(cont, *, overwrite: bool)

Private helper method to add a container to this group.

_add_container_callback(cont: dantro.abc.AbstractDataContainer) → None

Called by the base class after adding a container, this method checks whether the member map needs to be invalidated or whether the new container can be accomodated in it.

If it can be accomodated, the member map will be adjusted such that for all coordinates associated with the given cont, the member map points to the newly added container.

Parameters

cont (AbstractDataContainer) – The newly added container

_add_container_to_data(cont) → None

Adds a container to the underlying integer-ordered dictionary.

Unlike the parent method, this uses insert() in order to provide hints regarding the insertion position. It is optimised for insertion in ascending order.

_attrs = None
_check_cont(cont) → None

Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.

This is not expected to return, but can raise errors, if something did not work out as expected.

Parameters

cont – The container to check

_check_data(data: Any) → None

This method can be used to check the data provided to this container

It is called before the data is stored in the __init__ method and should raise an exception or create a warning if the data is not as desired.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Note

The CheckDataMixin provides a generalised implementation of this method to perform some type checks and react to unexpected types.

Parameters

data (Any) – The data to check

_check_name(new_name: str) → None

Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Parameters

new_name (str) – The new name, which is to be checked.

classmethod _combine_by_concatenation(dsets: numpy.ndarray, *, dims: Tuple[str]) → xarray.core.dataset.Dataset

Combine the given datasets by concatenation using xarray.concat and subsequent application along all dimensions specified in dims.

Parameters
  • dsets (np.ndarray) – The object-dtype array of xr.Dataset objects that are to be combined by concatenation.

  • dims (TDims) – The dimension names corresponding to _all_ the dimensions of the dsets array.

Returns

The dataset resulting from the concatenation

Return type

xr.Dataset

classmethod _combine_by_merge(dsets: numpy.ndarray) → xarray.core.dataset.Dataset

Combine the given datasets by merging using xarray.merge.

Parameters

dsets (np.ndarray) – The object-dtype array of xr.Datasets that are to be combined.

Returns

All datasets, aligned and combined via xarray.merge

Return type

xr.Dataset

_direct_insertion_mode(*, enabled: bool = True)

A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the with_direct_insertion() property will return true.

This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering, _enter_direct_insertion_mode() is called. After exiting, _exit_direct_insertion_mode() is called.

Parameters

enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.

_enter_direct_insertion_mode()

Called after entering direct insertion mode; can be overwritten to attach additional behaviour.

_exit_direct_insertion_mode()

Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.

_format_cls_name() → str

A __format__ helper function: returns the class name

_format_info() → str

A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!

_format_logstr() → str

A __format__ helper function: returns the log string, a combination of class name and name

_format_name() → str

A __format__ helper function: returns the name

_format_path() → str

A __format__ helper function: returns the path to this container

_format_tree() → str

Returns the default tree representation of this group by invoking the .tree property

_format_tree_condensed() → str

Returns the default tree representation of this group by invoking the .tree property

_get_cont(name: str, *, combination_method: str) → Optional[dantro.containers.xrdatactr.XrDataContainer]

Retrieve the container from the group. If no container could be found, returns None, which denotes that further processing should be skipped.

Parameters
  • name (str) – Name of the container to be extracted

  • combination_method (str) – How the container data will be combined

Returns

The extracted container

Return type

Union[XrDataContainer, None]

Raises

ItemAccessError – If combination_method == "concat", on invalid container name.

_get_coords_of(obj: dantro.abc.AbstractDataContainer) → Dict[str, Sequence[TCoord]]

Extract the coordinates for the given object using the extract_coords() function.

Parameters

obj (AbstractDataContainer) – The object to get the coordinates of.

Returns

The extracted coordinates

Return type

TCoordsDict

_ipython_key_completions_() → List[int]

For ipython integration, return a list of available keys.

Unlike the BaseDataGroup method, which returns a list of strings, this returns a list of integers.

Links the new_child to this class, unlinking the old one.

This method should be called from any method that changes which items are associated with this group.

_lock_hook()

Invoked upon locking.

_parse_indexers(indexers: dict, *, allow_deep: bool, **indexers_kwargs) → Tuple[dict, dict]

Parses the given indexer arguments and split them into indexers for the selection of group members and deep selection.

Parameters
  • indexers (dict) – The indexers dict, may be empty

  • allow_deep (bool) – Whether to allow deep selection

  • **indexers_kwargs – Additional indexers

Returns

(shallow indexers, deep indexers)

Return type

Tuple[dict, dict]

Raises

ValueError – If deep indexers were given but deep selection was not enabled

_parse_key(key: Union[str, int]) → str

Makes sure a key is a string

_process_cont(cont, *, coords, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Process the given container and coordinates into a data array; this applies selection along container dimensions that overlap with the group dimensions as well as deep selection.

Parameters
  • cont – The container to be processed

  • coords – The DataArrayCoordinates of the given container in the preselected member map.

  • shallow_indexers (dict) – Indexers that were used to preselect the member map.

  • deep_indexers (dict) – Indexers to be applied to the container

  • by_index (bool) – Whether to select by index

  • drop (bool) – Whether to drop coordinate variables instead of making them scalar.

  • **sel_kwargs – Passed to .sel.

Returns

The processed container data

Return type

xr.DataArray

Raises

ValueError – In name mode, on conflicting non-dimension container coordinates.

_select(*, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Preselect the member map (if needed) and designate a suitable method for further processing and selection based on the given combination method and indexers.

If possible, take shortcuts when selecting all data or when selecting data from a single group member.

Parameters
  • combination_method (str) – How to combine the member data.

  • shallow_indexers (dict) – Indexers to be applied on the group-level.

  • deep_indexers (dict) – Indexers to be applied on the member-level only.

  • by_index (bool) – Whether to select by index.

  • drop (bool) – Whether to drop coordinate variables instead of making them scalar.

  • **sel_kwargs – Passed to .sel.

Returns

The selected data.

Return type

xr.DataArray

Raises

ValueError – On invalid combination_method.

_select_all_merge() → xarray.core.dataarray.DataArray

Select all group data by directly merging all containers. This circumvents building the member map. This might fail, e.g. if there are conflicting or duplicate coordinates.

_select_generic(cont_names: xarray.core.dataarray.DataArray, *, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Select data from group members using the given indexers and combine it via the specified method. If deep indexers are given, apply the deep indexing on each of the members.

This method receives a labelled array of container names, on which the selection already took place. The aim is now to align the objects these names refer to, including their coordinates, and thereby construct an array that contains both the dimensions given by the cont_names array and each members’ data dimensions.

Available combination methods are based either on xarray.merge operations or xarray.concat along each dimension. For both these combination methods, the members of this group need to be prepared such that the operation can be applied, i.e.: they need to already be in an array capable of that operation and they need to directly or indirectly preserve coordinate information.

For that purpose, an object-array is constructed holding the processed member data. As the xarray.Dataset and xarray.DataArray types have issues with handling array-like objects in object arrays, this is done via a numpy.ndarray.

Parameters
  • cont_names (xr.DataArray) – The pre-selected member map object, i.e. a labelled array containing names of the desired members that are to be combined.

  • combination_method (str) – How to combine them: concat, try_concat, or merge. Concatenation will allow preserving the dtype of the underlying data.

  • shallow_indexers (dict) – Indexer arguments that were used for the group member selection.

  • deep_indexers (dict) – Indexer arguments for deep selection to be done before combination.

  • by_index (bool) – Whether the deep indexing should take place by index; if False, will use label-based selection.

  • **sel_kwargs – Passed on to .sel.

Returns

The selected data of the members from cont_names,

combined using the given combination method.

Return type

xr.Dataset

Raises
  • ValueError – On conflicting coordinate information on group-level and member-level.

  • KeyError – In concat mode, upon missing members.

_select_single(cont_names: xarray.core.dataarray.DataArray, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) → xarray.core.dataarray.DataArray

Select data from a single group member. Expects the preselected member map to contain only a single valid container name.

_tree_repr(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]

Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.

Parameters
  • level (int, optional) – The depth within the tree

  • max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.

  • info_fstr (str, optional) – The format string for the info string

  • info_ratio (float, optional) – The width ratio of the whole line width that the info string takes

  • condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.

  • total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.

Returns

The (multi-line) tree representation of

this group. If this method was invoked with level == 0, a string will be returned; otherwise, a list of strings will be returned.

Return type

Union[str, List[str]]

Unlink a child from this class.

This method should be called from any method that removes an item from this group, be it through deletion or through

_unlock_hook()

Invoked upon unlocking.

add(*conts, overwrite: bool = False)

Add the given containers to this group.

property allow_deep_selection

Whether deep selection is allowed.

property attrs

The container attributes.

property classname

Returns the name of this DataContainer-derived class

clear()

Clears all containers from this group.

This is done by unlinking all children and then overwriting _data with an empty _STORAGE_CLS object.

property coords

Returns a dict-like container of group-level coordinate values keyed by dimension.

property data

The stored data.

property dims

The names of the group-level dimensions this group manages.

It _may_ contain dimensions that overlap with dimension names from the members; this is intentional.

get(key, default=None)

Return the container at key, or default if container with name key is not available.

isel(indexers: dict = None, *, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with an index-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Note

For data combination (via any combination_method) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using the respective coordinates. See the xarray documentation for more information about coordinates.

Parameters
  • indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick indices. As xr.DataArray.isel, uses pandas-like indexing, i.e.: slices do not include the terminal value.

  • drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.

  • combination_method (str, optional) –

    How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:

    • concat: Concatenate. This can preserve the dtype, but requires that no data is missing.

    • merge: Merge, using xarray.merge. This leads to a type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

    • try_concat: Try concatenation, fall back to merging if that was unsuccessful.

    • auto: Automatically deduce suitably combination method. Use merge if data is non-integer type and try_concat otherwise.

    Note

    Selecting all data (by not passing any indexers) can be significantly faster using the merge combination method than using the concat method.

  • deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

  • **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

items()

Returns an iterator over the (name, data container) tuple of this group.

key_at_idx(idx: int) → str

Get a key by its index within the container. Can be negative.

Parameters

idx (int) – The index within the member sequence

Returns

The desired key

Return type

str

Raises

IndexError – Index out of range

keys()

Returns an iterator over the container names in this group.

keys_as_int() → Generator[int, None, None]

Returns an iterator over keys as integer values

lock()

Locks the data of this object

property locked

Whether this object is locked

property logstr

Returns the classname and name of this object

property member_map

Returns an array that represents the space that the members of this group span, where each value (i.e. a specific coordinate combination) is the name of the corresponding member of this group.

Upon first call, this is computed here. If members are added, it is tried to accomodate them in there; if not possible, the cache will be invalidated.

The member map _may_ include empty strings, i.e. coordinate combinations that are not covered by any member. Also, they can contain duplicate names, as one member can cover multiple coordinates.

Note

The member map is invalidated when new members are added that can not be accomodated in it. It will be recalculated when needed.

property member_map_available

Whether the member map is available yet.

property name

The name of this DataContainer-derived object.

property ndim

The rank of the space covered by the group-level dimensions.

new_container(path: Union[str, List[str]], *, Cls: type = None, **kwargs)

Creates a new container of class Cls and adds it at the given path relative to this group.

If needed, intermediate groups are automatically created.

Parameters
  • path (Union[str, List[str]]) – Where to add the container.

  • Cls (type, optional) – The class of the container to add. If None, the _NEW_CONTAINER_CLS class variable’s value is used.

  • **kwargs – passed on to Cls.__init__

Returns

the created container

Return type

Cls

Raises
  • ValueError – If neither the Cls argument nor the class variable _NEW_CONTAINER_CLS were set or if path was empty.

  • TypeError – When Cls is not compatible to the data tree

new_group(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new group at the given path.

Parameters
  • path (Union[str, list]) – The path to create the group at. Note that the whole intermediate path needs to already exist.

  • Cls (type, optional) – If given, use this type to create the group. If not given, uses the class specified in the _NEW_GROUP_CLS class variable or, as last resort, the type of this instance.

  • **kwargs – Passed on to Cls.__init__

Returns

the created group

Return type

Cls

Raises

TypeError – For the given class not being derived from BaseDataGroup

property parent

The associated parent of this container or group

property path

The path to get to this container or group from some root path

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

raise_if_locked(*, prefix: str = None)

Raises an exception if this object is locked; does nothing otherwise

recursive_update(other, *, overwrite: bool = True)

Recursively updates the contents of this data group with the entries of the given data group

Note

This will create shallow copies of those elements in other that are added to this object.

Parameters
  • other (BaseDataGroup) – The group to update with

  • overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.

Raises

TypeError – If other was of invalid type

sel(indexers: dict = None, *, method: str = None, tolerance: float = None, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) → xarray.core.dataarray.DataArray

Return a new labelled xr.DataArray with a coordinate-selected subset of members of this group.

If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.

Note

For data combination (via any combination_method) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using the respective coordinates. See the xarray documentation for more information about coordinates.

Parameters
  • indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick labels. As xr.DataArray.sel, uses pandas-like indexing, i.e.: slices include the terminal value.

  • method (str, optional) – Method to use for inexact matches

  • tolerance (float, optional) – Maximum (absolute) distance between original and given label for inexact matches.

  • drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.

  • combination_method (str, optional) –

    How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:

    • concat: Concatenate. This can preserve the dtype, but requires that no data is missing.

    • merge: Merge, using xarray.merge. This leads to a type conversion to float64, but allows members being missing or coordinates not fully filling the available space.

    • try_concat: Try concatenation, fall back to merging if that was unsuccessful.

    • auto: Automatically deduce suitably combination method. Use merge if data is non-integer type and try_concat otherwise.

    Note

    Selecting all data (by not passing any indexers) can be significantly faster using the merge combination method than using the concat method.

  • deep (bool, optional) – Whether to allow deep indexing, i.e.: that indexers may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. If None, will use the value returned by the allow_deep_selection property.

  • **indexers_kwargs – Additional indexers

Returns

The selected data, potentially a combination of data

on group level and member-level data.

Return type

xr.DataArray

setdefault(key, default=None)

This method is not supported for a data group

property shape

Return the shape of the space covered by the group-level dimensions.

property tree

Returns the default (full) tree representation of this group

property tree_condensed

Returns the condensed tree representation of this group. Uses the _COND_TREE_* prefixed class attributes as parameters.

unlock()

Unlocks the data of this object

update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Returns an iterator over the containers in this group.

property with_direct_insertion

Whether the class this mixin is mixed into is currently in direct insertion mode.