dantro.groups.pspgrp module

In this module, BaseDataContainer specializations that make use of features from the paramspace package are implemented.

class dantro.groups.pspgrp.ParamSpaceStateGroup(*, name: str, containers: list = None, attrs=None)[source]

Bases: dantro.groups.ordered.OrderedDataGroup

A ParamSpaceStateGroup is a member group of the ParamSpaceGroup.

While its own name need be interpretable as a positive integer (enforced in the enclosing ParamSpaceGroup but also here), it can hold members with any name.

_NEW_GROUP_CLS

alias of dantro.groups.ordered.OrderedDataGroup

_check_name(name: str) → None[source]

Called by __init__ and overwritten here to check the name.

property coords

Retrieves the coordinates of this group within the parameter space described by the ParamSpaceGroup this group is enclosed in.

Returns

The coordinates of this group, keys being dimension names and

values being the coordinate values for this group.

Return type

dict

_ALLOWED_CONT_TYPES = None
_ATTRS_CLS

alias of dantro.base.BaseDataAttrs

_COND_TREE_CONDENSE_THRESH = 10
_COND_TREE_MAX_LEVEL = 10
_LockDataMixin__locked = False
_MutableMapping__marker = <object object>
_NEW_CONTAINER_CLS = None
_STORAGE_CLS

alias of collections.OrderedDict

__contains__(cont: Union[str, dantro.base.BaseDataContainer]) → bool

Whether the given container is in this group or not.

Parameters

cont (Union[str, BaseDataContainer]) – The name of the container or an object reference.

Returns

Whether the given container is in this group.

Return type

bool

__delitem__(key: str) → None

Deletes an item from the group

__format__(spec_str: str) → str

Creates a formatted string from the given specification.

Invokes further methods which are prefixed by _format_.

__getitem__(key: Union[str, List[str]])

Returns the container in this group with the given name.

Parameters

key (Union[str, List[str]]) – The object to retrieve. If this is a path, will recurse down until at the end.

Returns

The object at key

Raises

KeyError – If no such key can be found

__init__(*, name: str, containers: list = None, attrs=None)

Initialize a BaseDataGroup, which can store other containers and attributes.

Parameters

  • name (str) – The name of this data container

  • containers (list, optional) – The containers that are to be stored as members of this group. If given, these are added one by one using the .add method.

  • attrs (None, optional) – A mapping that is stored as attributes

__iter__()

Returns an iterator over the OrderedDict

__len__() → int

The length of the data.

__repr__() → str

Same as __str__

__setitem__(key: Union[str, List[str]], val: dantro.base.BaseDataContainer) → None

This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!

Parameters

  • key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.

  • val (BaseDataContainer) – The value to set

Returns

None

Raises

ValueError – If trying to add an element to this group, which should be done via the add method.

__sizeof__() → int

Returns the size of the data (in bytes) stored in this container’s data and its attributes.

Note that this value is approximate. It is computed by calling the sys.getsizeof function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.

Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.

For more information, see the documentation of sys.getsizeof:

__str__() → str

An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.

_abc_impl = <_abc_data object>
_add_container(cont, *, overwrite: bool)

Private helper method to add a container to this group.

_add_container_callback(cont) → None

Called after a container was added.

_add_container_to_data(cont) → None

Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …

NOTE This method should NEVER be called on its own, but only via the

_add_container method, which takes care of properly linking the container that is to be added.

NOTE After adding, the container need be reachable under its .name!

Parameters

cont – The container to add

_attrs = None
_check_cont(cont) → None

Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.

This is not expected to return, but can raise errors, if something did not work out as expected.

Parameters

cont – The container to check

_check_data(data: Any) → None

This method can be used to check the data provided to this container

It is called before the data is stored in the __init__ method and should raise an exception or create a warning if the data is not as desired.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

NOTE The CheckDataMixin provides a generalised implementation of this

method to perform some type checks and react to unexpected types.

Parameters

data (Any) – The data to check

_format_cls_name() → str

A __format__ helper function: returns the class name

_format_info() → str

A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!

_format_logstr() → str

A __format__ helper function: returns the log string, a combination of class name and name

_format_name() → str

A __format__ helper function: returns the name

_format_path() → str

A __format__ helper function: returns the path to this container

_format_tree() → str

Returns the default tree representation of this group by invoking the .tree property

_format_tree_condensed() → str

Returns the default tree representation of this group by invoking the .tree property

_ipython_key_completions_() → List[str]

For ipython integration, return a list of available keys

Links the new_child to this class, unlinking the old one.

This method should be called from any method that changes which items are associated with this group.

_lock_hook()

Invoked upon locking.

_tree_repr(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]

Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.

Parameters

  • level (int, optional) – The depth within the tree

  • max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.

  • info_fstr (str, optional) – The format string for the info string

  • info_ratio (float, optional) – The width ratio of the whole line width that the info string takes

  • condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.

  • total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.

Returns

The (multi-line) tree representation of

this group. If this method was invoked with level == 0, a string will be returned; otherwise, a list of strings will be returned.

Return type

Union[str, List[str]]

Unlink a child from this class.

This method should be called from any method that removes an item from this group, be it through deletion or through

_unlock_hook()

Invoked upon unlocking.

add(*conts, overwrite: bool = False)

Add the given containers to this group.

property attrs

The container attributes.

property classname

Returns the name of this DataContainer-derived class

clear() → None. Remove all items from D.
property data

The stored data.

get(key, default=None)

Return the container at key, or default if container with name key is not available.

items()

Returns an iterator over the (name, data container) tuple of this group.

keys()

Returns an iterator over the container names in this group.

lock()

Locks the data of this object

property locked

Whether this object is locked

property logstr

Returns the classname and name of this object; a combination often used in logging…

property name

The name of this DataContainer-derived object.

new_container(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new container of class Cls and adds it at the given path relative to this group.

Parameters

  • path (Union[str, list]) – Where to add the container. Note that the intermediates of this path need to already exist.

  • Cls (type, optional) – The class of the container to add. If None, the _NEW_CONTAINER_CLS class variable’s value is used; if not given, this will raise a ValueError.

  • **kwargs – kwargs to pass on to Cls.__init__

Returns

the created container

Return type

Cls

Raises

  • KeyError – When intermediate groups to path are missing

  • TypeError – When the given Cls is invalid

new_group(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new group at the given path.

Parameters

  • path (Union[str, list]) – The path to create the group at. Note that the whole intermediate path needs to already exist.

  • Cls (type, optional) – If given, use this type to create the group. If not given, uses the class specified in the _NEW_GROUP_CLS class variable or, as last resort, the type of this instance.

  • **kwargs – Passed on to Cls.__init__

Returns

the created group

Return type

Cls

Raises

TypeError – For the given class not being derived from BaseDataGroup

property parent

The associated parent of this container or group

property path

The path to get to this container or group from some root path

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

raise_if_locked(*, prefix: str = None)

Raises an exception if this object is locked; does nothing otherwise

recursive_update(other)

Recursively updates the contents of this data group with the entries of the given data group

NOTE This will create shallow copies of those elements in other that are added to this object.

Parameters

other (BaseDataGroup) – The group to update with

Raises

TypeError – If other was of invalid type

setdefault(key, default=None)

This method is not supported for a data group

property tree

Returns the default (full) tree representation of this group

property tree_condensed

Returns the condensed tree representation of this group. Uses the _COND_TREE_* prefixed class attributes as parameters.

unlock()

Unlocks the data of this object

update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Returns an iterator over the containers in this group.

class dantro.groups.pspgrp.ParamSpaceGroup(*, name: str, pspace: paramspace.paramspace.ParamSpace = None, containers: list = None, **kwargs)[source]

Bases: dantro.mixins.indexing.PaddedIntegerItemAccessMixin, dantro.groups.ordered.IndexedDataGroup

The ParamSpaceGroup is associated with a ParamSpace object and the loaded results of an iteration over this parameter space.

Thus, the groups that are stored in the ParamSpaceGroup need all relate to a state of the parameter space, identified by a zero-padded string name. In fact, this group allows no other kinds of groups stored inside.

To make access to a specific state easier, it allows accessing a state by its state number as integer.

_PSPGRP_PSPACE_ATTR_NAME = 'pspace'
_PSPGRP_TRANSFORMATOR = None
_NEW_GROUP_CLS

alias of ParamSpaceStateGroup

_ALLOWED_CONT_TYPES = (<class 'dantro.groups.pspgrp.ParamSpaceStateGroup'>,)
__init__(*, name: str, pspace: paramspace.paramspace.ParamSpace = None, containers: list = None, **kwargs)[source]

Initialize a OrderedDataGroup from the list of given containers.

Parameters

  • name (str) – The name of this group.

  • pspace (ParamSpace, optional) – Can already pass a ParamSpace object here.

  • containers (list, optional) – A list of containers to add, which need to be ParamSpaceStateGroup objects.

  • **kwargs – Further initialisation kwargs, e.g. attrs …

property pspace

Reads the entry named _PSPGRP_PSPACE_ATTR_NAME in .attrs and returns a ParamSpace object, if available there.

Returns

The associated parameter space, or None,

if there is none associated yet.

Return type

Union[ParamSpace, None]

_ATTRS_CLS

alias of dantro.base.BaseDataAttrs

_COND_TREE_CONDENSE_THRESH = 10
_COND_TREE_MAX_LEVEL = 10
_LockDataMixin__locked = False
_MutableMapping__marker = <object object>
_NEW_CONTAINER_CLS = None
_PADDED_INT_FSTR = None
_PADDED_INT_KEY_WIDTH = None
_PADDED_INT_MAX_VAL = None
_PADDED_INT_STRICT_CHECKING = True
_STORAGE_CLS

alias of dantro.utils.ordereddict.IntOrderedDict

__contains__(key: Union[str, int]) → bool

Adjusts the parent method to allow checking for integers

__delitem__(key: Union[str, int])

Adjusts the parent method to allow item deletion by integer key

__format__(spec_str: str) → str

Creates a formatted string from the given specification.

Invokes further methods which are prefixed by _format_.

__getitem__(key: Union[str, int])

Adjusts the parent method to allow integer key item access

__iter__()

Returns an iterator over the OrderedDict

__len__() → int

The length of the data.

__repr__() → str

Same as __str__

__setitem__(key: Union[str, List[str]], val: dantro.base.BaseDataContainer) → None

This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!

Parameters

  • key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.

  • val (BaseDataContainer) – The value to set

Returns

None

Raises

ValueError – If trying to add an element to this group, which should be done via the add method.

__sizeof__() → int

Returns the size of the data (in bytes) stored in this container’s data and its attributes.

Note that this value is approximate. It is computed by calling the sys.getsizeof function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.

Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.

For more information, see the documentation of sys.getsizeof:

__str__() → str

An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.

_abc_impl = <_abc_data object>
_add_container(cont, *, overwrite: bool)

Private helper method to add a container to this group.

_add_container_callback(cont) → None

Called after a container was added.

_add_container_to_data(cont) → None

Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …

NOTE This method should NEVER be called on its own, but only via the

_add_container method, which takes care of properly linking the container that is to be added.

NOTE After adding, the container need be reachable under its .name!

Parameters

cont – The container to add

_attrs = None
_check_cont(cont: dantro.abc.AbstractDataContainer) → None

This method is invoked when adding a member to a group and makes sure the name of the added group is correctly zero-padded.

Also, upon first call, communicates the zero padded integer key width, i.e.: the length of the container name, to the PaddedIntegerItemAccessMixin.

Parameters

cont – The member container to add

Returns

None: No return value needed

_check_data(data: Any) → None

This method can be used to check the data provided to this container

It is called before the data is stored in the __init__ method and should raise an exception or create a warning if the data is not as desired.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

NOTE The CheckDataMixin provides a generalised implementation of this

method to perform some type checks and react to unexpected types.

Parameters

data (Any) – The data to check

_check_name(new_name: str) → None

Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.

This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().

Parameters

new_name (str) – The new name, which is to be checked.

_format_cls_name() → str

A __format__ helper function: returns the class name

_format_info() → str

A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!

_format_logstr() → str

A __format__ helper function: returns the log string, a combination of class name and name

_format_name() → str

A __format__ helper function: returns the name

_format_path() → str

A __format__ helper function: returns the path to this container

_format_tree() → str

Returns the default tree representation of this group by invoking the .tree property

_format_tree_condensed() → str

Returns the default tree representation of this group by invoking the .tree property

_ipython_key_completions_() → List[int]

For ipython integration, return a list of available keys.

Unlike the BaseDataGroup method, which returns a list of strings, this returns a list of integers.

Links the new_child to this class, unlinking the old one.

This method should be called from any method that changes which items are associated with this group.

_lock_hook()

Invoked upon locking.

_parse_key(key: Union[str, int]) → str

Parse a potentially integer key to a zero-padded string

_tree_repr(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]

Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.

Parameters

  • level (int, optional) – The depth within the tree

  • max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.

  • info_fstr (str, optional) – The format string for the info string

  • info_ratio (float, optional) – The width ratio of the whole line width that the info string takes

  • condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.

  • total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.

Returns

The (multi-line) tree representation of

this group. If this method was invoked with level == 0, a string will be returned; otherwise, a list of strings will be returned.

Return type

Union[str, List[str]]

Unlink a child from this class.

This method should be called from any method that removes an item from this group, be it through deletion or through

_unlock_hook()

Invoked upon unlocking.

add(*conts, overwrite: bool = False)

Add the given containers to this group.

property attrs

The container attributes.

property classname

Returns the name of this DataContainer-derived class

clear() → None. Remove all items from D.
property data

The stored data.

get(key, default=None)

Return the container at key, or default if container with name key is not available.

items()

Returns an iterator over the (name, data container) tuple of this group.

key_at_idx(idx: int) → str

Get a key by its index within the container. Can be negative.

Parameters

idx (int) – The index within the member sequence

Returns

The desired key

Return type

str

Raises

IndexError – Index out of range

keys()

Returns an iterator over the container names in this group.

keys_as_int() → Generator[int, None, None]

Returns an iterator over keys as integer values

lock()

Locks the data of this object

property locked

Whether this object is locked

property logstr

Returns the classname and name of this object; a combination often used in logging…

property name

The name of this DataContainer-derived object.

new_container(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new container of class Cls and adds it at the given path relative to this group.

Parameters

  • path (Union[str, list]) – Where to add the container. Note that the intermediates of this path need to already exist.

  • Cls (type, optional) – The class of the container to add. If None, the _NEW_CONTAINER_CLS class variable’s value is used; if not given, this will raise a ValueError.

  • **kwargs – kwargs to pass on to Cls.__init__

Returns

the created container

Return type

Cls

Raises

  • KeyError – When intermediate groups to path are missing

  • TypeError – When the given Cls is invalid

new_group(path: Union[str, list], *, Cls: type = None, **kwargs)

Creates a new group at the given path.

Parameters

  • path (Union[str, list]) – The path to create the group at. Note that the whole intermediate path needs to already exist.

  • Cls (type, optional) – If given, use this type to create the group. If not given, uses the class specified in the _NEW_GROUP_CLS class variable or, as last resort, the type of this instance.

  • **kwargs – Passed on to Cls.__init__

Returns

the created group

Return type

Cls

Raises

TypeError – For the given class not being derived from BaseDataGroup

property only_default_data_present

Returns true if only data for the default point in parameter space is available in this group.

property padded_int_key_width

Returns the width of the zero-padded integer key or None, if it is not already specified.

property parent

The associated parent of this container or group

property path

The path to get to this container or group from some root path

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

raise_if_locked(*, prefix: str = None)

Raises an exception if this object is locked; does nothing otherwise

recursive_update(other)

Recursively updates the contents of this data group with the entries of the given data group

NOTE This will create shallow copies of those elements in other that are added to this object.

Parameters

other (BaseDataGroup) – The group to update with

Raises

TypeError – If other was of invalid type

setdefault(key, default=None)

This method is not supported for a data group

property tree

Returns the default (full) tree representation of this group

property tree_condensed

Returns the condensed tree representation of this group. Uses the _COND_TREE_* prefixed class attributes as parameters.

unlock()

Unlocks the data of this object

update([E, ]**F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Returns an iterator over the containers in this group.

select(*, field: Union[str, List[str]] = None, fields: Dict[str, List[str]] = None, subspace: dict = None, method: str = 'concat', idx_as_label: bool = False, base_path: str = None, **kwargs) → xarray.core.dataset.Dataset[source]

Selects a multi-dimensional slab of this ParamSpaceGroup and the specified fields and returns them bundled into an xarray.Dataset with labelled dimensions and coordinates.

Parameters

  • field (Union[str, List[str]], optional) – The field of data to select. Should be path or a list of strings that points to an entry in the data tree. To select multiple fields, do not pass this argument but use the fields argument.

  • fields (Dict[str, List[str]], optional) – A dict specifying the fields that are to be loaded into the dataset. Keys will be the names of the resulting variables, while values should specify the path to the field in the data tree. Thus, they can be strings, lists of strings or dicts with the path key present. In the latter case, a dtype can be specified via the dtype key in the dict.

  • subspace (dict, optional) – Selector for a subspace of the parameter space. Adheres to the ParamSpace.activate_subspace signature.

  • method (str, optional) –

    How to combine the selected datasets.

    • concat: concatenate sequentially along all parameter space dimensions. This can preserve the data type but it does not work if one data point is missing.

    • merge: merge always works, even if data points are missing, but will convert all dtypes to float.

  • idx_as_label (bool, optional) – If true, adds the trivial indices as labels for those dimensions where coordinate labels were not extractable from the loaded field. This allows merging for data with different extends in an unlabelled dimension.

  • base_path (str, optional) – If given, path specifications for each field can be seen as relative to this path

  • **kwargs – Passed along either to xr.concat or xr.merge, depending on the method argument.

Raises

  • KeyError – Description

  • ValueError – Raised in multiple scenarios: - If no ParamSpace was associated with this group - For wrong argument values - If the data to select cannot be extracted with the given argument values - Exceptions passed on from xarray

Returns

The selected hyperslab of the parameter space, holding

the desired fields.

Return type

xr.Dataset