dantro.groups package
Contents
dantro.groups package#
The groups sub-package implements BaseDataGroup
specializations.
isort:skip_file
Submodules#
dantro.groups._registry module#
Implements a registry for dantro group types based on
ObjectRegistry
.
- class GroupRegistry[source]#
Bases:
dantro._registry.ObjectRegistry
- _EXPECTED_TYPE: Optional[Union[tuple, type]] = (<class 'type'>,)#
If set, will check for expected types
- _register_via_decorator(obj: type, name: Optional[str] = None, **kws)[source]#
Performs the registration operations when the decorator is used to register an object.
- __contains__(obj_or_key: Union[Any, str]) bool #
Whether the given argument is part of the keys or values of this registry.
- _decorator(arg: Optional[Union[Any, str]] = None, /, **kws)#
Method that can be used as a decorator for registering objects with this registry.
- Parameters
arg (Union[Any, str], optional) – The name that should be used or the object that is to be added. If not a string, this refers to the
@is_container
call syntax**kws – Passed to
register()
- _determine_name(obj: Any, *, name: Optional[str]) str #
Determines the object name, using a potentially given name
- items()#
- keys()#
- register(obj: Any, name: Optional[str] = None, *, skip_existing: Optional[bool] = None, overwrite_existing: Optional[bool] = None) str #
Adds an entry to the registry.
- Parameters
obj (Any) – The object to add to the registry.
name (Optional[str], optional) – The name to use. If not given, will deduce a name from the given object.
skip_existing (bool, optional) – Whether to skip registration if an object of that name already exists. If None, the classes default behavior (see
_SKIP
) is used.overwrite_existing (bool, optional) – Whether to overwrite an entry if an object with that name already exists. If None, the classes default behavior (see
_OVERWRITE
) is used.
- values()#
- GROUPS = <dantro.groups._registry.GroupRegistry object>#
The dantro data group registry object.
- register_group(Cls: type, name: str, *, skip_existing: bool = False, overwrite_existing: bool = True) None [source]#
Adds an entry to the shared group registry.
- Parameters
Cls (type) – The class that is to be registered as a group.
name (str) – The name to use for registration.
skip_existing (bool, optional) – Whether to skip registration if the group name is already registered. This suppresses the ValueError raised on existing group name.
overwrite_existing (bool, optional) – Whether to overwrite a potentially already existing group of the same name. If set, this takes precedence over
skip_existing
.
- is_group(arg: Optional[Union[type, str]] = None, /, **kws)[source]#
Decorator for registering groups with the container type registry.
As an alternative to
register_group()
, this decorator can be used to register a container right where its defined:from dantro.groups import BaseDataGroup, is_group # Group name deduced from class name @is_group class MyDataGroup(BaseDataGroup): # ... do stuff here ... pass # Custom group name @is_group("my_group") class MyDataGroup(BaseDataGroup): # ... do stuff here ... pass # Overwriting a registered group of the same name @is_group("my_group", overwrite_existing=True) class MyDataGroup(BaseDataGroup): # ... do stuff here ... pass
dantro.groups.dirpath module#
A data group that represents a directory path and holds the corresponding file paths as members
- class DirectoryGroup(*args, name: str, dirpath: Optional[Union[str, Path]] = None, strict: bool = True, parent: Optional[DirectoryGroup] = None, **kwargs)[source]#
Bases:
dantro.groups.ordered.OrderedDataGroup
A group that maps to a directory path in the file system.
- _NEW_CONTAINER_CLS#
alias of
dantro.containers.path.PathContainer
- __init__(*args, name: str, dirpath: Optional[Union[str, Path]] = None, strict: bool = True, parent: Optional[DirectoryGroup] = None, **kwargs)[source]#
Sets up a DirectoryGroup instance that holds the path to a certain directory.
By default, this group only allows members to be
PathContainer
orDirectoryGroup
instances in order to maintain an association to the filesystem directory represented by this group.Note
It is not actually checked whether the path points to a directory.
- Parameters
*args – Passed to parent init
dirpath (Union[str, Path], optional) – A path compatible to
pathlib.Path
. This path need not exist in the file system but it should point to a directory. If not given at initialization, it should be set afterwards.strict (bool, optional) – If not True, will allow members of this group to be of any kind of dantro object.
**kwargs – Passed to parent init
- _ALLOWED_CONT_TYPES: tuple = ('self', <class 'dantro.containers.path.PathContainer'>)#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_GROUP_CLS: type = None#
Which class to use when creating a new group via
new_group()
. If None, the type of the current instance is used for the new group.
- _STORAGE_CLS#
alias of
collections.OrderedDict
- __contains__(cont: Union[str, AbstractDataContainer]) bool #
Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __getitem__(key: Union[str, List[str]]) AbstractDataContainer #
Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
- __iter__()#
Returns an iterator over the OrderedDict
- __setitem__(key: Union[str, List[str]], val: BaseDataContainer) None #
This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont: AbstractDataContainer) None #
Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- property fs_path: Path#
Returns the filesystem path associated with this group by reading the corresponding data attribute (by default:
fs_path
).Note
The path this object represents may or may not exist.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
dantro.groups.graph module#
In this module, the GraphGroup
is
implemented, which provides an interface between hierarchically stored data
and the creation of graph objects using the networkx package and the therein
implemented networkx.Graph
classes.
See The GraphGroup for more information.
- class GraphGroup(*args, **kwargs)[source]#
Bases:
dantro.base.BaseDataGroup
The GraphGroup class manages groups of graph data containers and provides the possibility to create networkx graph objects using the data inside this group.
See The GraphGroup for more information.
- _ALLOWED_CONT_TYPES: Optional[tuple] = (<class 'dantro.containers.xr.XrDataContainer'>, <class 'dantro.groups.labelled.LabelledDataGroup'>)#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _GG_node_container = 'nodes'#
- _GG_edge_container = 'edges'#
- _GG_attr_directed = 'directed'#
- _GG_attr_parallel = 'parallel'#
- _GG_attr_edge_container_is_transposed = 'edge_container_is_transposed'#
- _GG_attr_keep_dim = 'keep_dim'#
- _GG_WARN_UPON_BAD_ALIGN = True#
- __init__(*args, **kwargs)[source]#
Initialize a GraphGroup.
- Parameters
*args – passed to
dantro.base.BaseDataGroup.__init__()
**kwargs – passed to
dantro.base.BaseDataGroup.__init__()
- property node_container#
Returns the associated node container of this graph group
- property edge_container#
Returns the associated edge container of this graph group
- property default_keep_dim#
The default dimensions not to be squeezed during data selection as specified in the respective group attribute.
- _get_item_or_pmap(key: Union[str, List[str]])[source]#
Returns the object accessible via
key
. Apart from allowing to retrieve objects in this group, the method additionally allows to access data stored in property maps.
- _get_data_at(*, data: Union[XrDataContainer, LabelledDataGroup], sel: dict = None, isel: dict = None, at_time: int = None, at_time_idx: int = None, keep_dim=None) Union[DataArray, XrDataContainer] [source]#
Returns a
xarray.DataArray
containing the data specified via the selectorssel
andisel
. Any dimension of size 1 is removed from the selected data.Warning
Any invalid key in
sel
andisel
is ignored silently.- Parameters
data (Union[XrDataContainer, LabelledDataGroup]) – Data to select from.
sel (dict, optional) – Dict of coordinate values keyed by dimensions, passed to
data.sel
. Used to select data via index label. May be given together withisel
if no key exists in both.isel (dict, optional) – Dict of indexes keyed by dimensions, passed to
data.isel
. Used to select data via index. May be given together withsel
if no key exists in both.at_time (int, optional) – Select along
time
dimension via index label. Translated tosel = dict(time=at_time)
, potentially overwriting an existingtime
entry.at_time_idx (int, optional) – Select along
time
dimension via index. Translated toisel = dict(time=at_time_idx)
, potentially overwriting an existingtime
entry.keep_dim (optional) – Iterable containing names of the dimensions that can not be squeezed.
- Returns
The selected data
- Return type
- Raises
ValueError – On keys that exist in both
sel
andisel
- _prepare_edge_data(*, edges, max_tuple_size: int)[source]#
Prepares the edge data. Depending on the
_GG_attr_edge_container_is_transposed
class attribute, the edge data is transposed or not. If the attribute does not exist, the data is transposed only if the correct shape could unambiguously be deduced.- Parameters
edges – The edge data stored in a 2-dimensional container
max_tuple_size (int) – The maximum allowed edge tuple size (4 for
networkx.MultiGraph
, else 3). Used if the correct shape is tried to be deduced automatically.
- Returns
The edge data, possibly transposed
- Raises
TypeError – Edge data is not 2-dimensional
- _check_alignment(*, ent, prop)[source]#
Checks the alignment of property data and entity (node or edge) data. If
self._GG_WARN_UPON_BAD_ALIGN
is True, warn on possible pitfalls.- Parameters
ent – The entity (node or edge) data
prop – The property data
- register_property_map(key: str, data)[source]#
Registers a new property map. It allows for the given data to be accessed internally by the specified key.
- Parameters
key (str) – The key via which the registered data will be available
data – The data to be mapped. If the given data is not an allowed container type, an attempt is made to construct an
XrDataContainer
with the data. Only if this operation fails, will property map registration fail.
- Raises
ValueError – On invalid key
- create_graph(*, directed: bool = None, parallel_edges: bool = None, node_props: list = None, edge_props: list = None, sel: dict = None, isel: dict = None, at_time: int = None, at_time_idx: int = None, align: bool = False, keep_dim=None, **graph_kwargs) Graph [source]#
Create a networkx
networkx.Graph
(or a more specialized graph type) object from the node and edge data associated with this graph group. Optionally, node and edge properties can be added from data stored or registered in the graph group. The coordinates for the selected or squeezed dimensions of the node, edge, and property data are stored as graph attributes ing.graph
.Note
Any pre-selection specified by
sel
,isel
,at_time
, orat_time_idx
will be applied to the node data, edge data, as well as any given property data.Warning
Any invalid key in
sel
andisel
is ignored silently (see_get_data_at()
).- Parameters
directed (bool, optional) – If true, the graph will be directed. If not given, the value given by the group attribute with name
_GG_attr_directed
is used instead.parallel_edges (bool, optional) – If true, the graph will allow parallel edges. If not given, the value is tried to be read from the group attribute with name
_GG_attr_parallel
.node_props (list, optional) – List of names specifying the containers that contain the node property data.
edge_props (list, optional) – List of names specifying the containers that contain the edge property data.
sel (dict, optional) – Dict of coordinate values keyed by dimensions, passed to
_get_data_at()
. Used to select data via index label.isel (dict, optional) – Dict of indexes keyed by dimensions, passed to
_get_data_at()
. Used to select data via index.at_time (int, optional) – Select along
time
dimension via index label. Translated tosel = dict(time=at_time)
.at_time_idx (int, optional) – Select along
time
dimension via index. Translated toisel = dict(time=at_time_idx)
.align (bool, optional) – If True, the property data is aligned with the node/edge data using
xarray.align
(default: False). The indexes of the<node/edge>_container
are used for each dimension. If the class variable_GG_WARN_UPON_BAD_ALIGN
is True, warn upon missing values or if no re-ordering was done. Any dimension of size 1 is squeezed and thus alignment (viaalign=True
) will have no effect on such dimensions.keep_dim (optional) – Iterable containing names of the dimensions that can not be squeezed. Passed on to
_get_data_at()
.**graph_kwargs – Passed to the constructor of the respective networkx graph object.
- Returns
The networkx graph object. Depending on the provided information, one of the following graph objects is created:
networkx.Graph
,networkx.DiGraph
,networkx.MultiGraph
,networkx.MultiDiGraph
.
- set_node_property(*, g: Graph, name: str, data=None, align: bool = False, keep_dim=None, **selector)[source]#
Sets a property to every node in Graph
g
that is also in thenode_container
of the graph group. The coordinates for the selected or squeezed dimensions of the property data are stored as Graph attributes (ing.graph
).- Parameters
g (Graph) – The networkx graph object
name (str) – If
data
isNone
,name
must specify the container within the graph group that contains the property values, or be valid key inproperty_maps
.name
is used as the name for the property in the graph object, potentially overwriting an existing property.data (None, optional) – If given, load node properties directly from
data
. If the given data is not an allowed container type, an attempt is made to construct anXrDataContainer
with the data. Only if this operation fails, the node property setting will fail.align (bool, optional) – If True, the property data is aligned with the node data using
xarray.align
. The indexes of thenode_container
are used for each dimension. If the class variable_GG_WARN_UPON_BAD_ALIGN
is True, warn upon missing values or if no re-ordering was done. Any dimension of size 1 is squeezed and thus alignment (viaalign=True
) will have no effect on such dimensions.keep_dim (optional) – Iterable containing names of the dimensions that can not be squeezed. Passed on to
_get_data_at()
.**selector – Specifies the selection applied to both node data and property data. Passed on to
_get_data_at()
. Use thesel
(isel
) dict to select data via coordinate value (index).
- Raises
ValueError – Lenght mismatch of the selected property and node data
- set_edge_property(*, g: Graph, name: str, data=None, align: bool = False, keep_dim=None, **selector)[source]#
Sets a property to every edge in Graph
g
that is also in theedge_container
of the graph group. The coordinates for the selected or squeezed dimensions of the property data are stored as Graph attributes (ing.graph
).- Parameters
g (Graph) – The networkx graph object
name (str) – If
data
isNone
,name
must specify the container within the graph group that contains the property values, or be valid key inproperty_maps
.name
is used as the name for the property in the graph object, potentially overwriting an existing property.data (None, optional) – If given, load edge properties directly from
data
. If the given data is not an allowed container type, an attempt is made to construct anXrDataContainer
with the data. Only if this operation fails, the edge property setting will fail.align (bool, optional) – If True, the property data is aligned with the edge data using
xarray.align
. The indexes of theedge_container
are used for each dimension. If the class variable_GG_WARN_UPON_BAD_ALIGN
is True, warn upon missing values or if no re-ordering was done. Any dimension of size 1 is squeezed and thus alignment (viaalign=True
) will have no effect on such dimensions.keep_dim (optional) – Iterable containing names of the dimensions that can not be squeezed. Passed on to
_get_data_at()
.**selector – Specifies the selection applied to both edge data and property data. Passed on to
_get_data_at()
. Use thesel
(isel
) dict to select data via coordinate value (index).
- Raises
ValueError – Lenght mismatch of the selected property and edge data
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS: type = None#
Which class to use for creating a new container via call to the
new_container()
method. If None, the type needs to be specified explicitly in the method call.
- _NEW_GROUP_CLS: type = None#
Which class to use when creating a new group via
new_group()
. If None, the type of the current instance is used for the new group.
- __contains__(cont: Union[str, AbstractDataContainer]) bool #
Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __getitem__(key: Union[str, List[str]]) AbstractDataContainer #
Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
- __iter__()#
Returns an iterator over the OrderedDict
- __setitem__(key: Union[str, List[str]], val: BaseDataContainer) None #
This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont: AbstractDataContainer) None #
Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
dantro.groups.labelled module#
Implements the LabelledDataGroup, which allows to handle groups and containers that can be associated with further coordinates.
This imitates the xarray selection interface and provides a uniform interface to select data from these groups. Most importantly, it allows to combine all the data of one group, allowing to conveniently work with heterogeneously stored data.
- class LabelledDataGroup(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)[source]#
Bases:
dantro.groups.ordered.OrderedDataGroup
A group that assumes that the members it contains can be labelled with dimension names and coordinates.
Such a group has the great benefit to provide a selection interface that works fully on the dimension labels and coordinates and can cooperate with the xarray selection interface, i.e. the
sel
andisel
methods.- _NEW_CONTAINER_CLS#
alias of
dantro.containers.xr.XrDataContainer
- LDG_ALLOW_DEEP_SELECTION = True#
- LDG_DIMS = ()#
- LDG_EXTRACT_COORDS_FROM = 'data'#
- LDG_COORDS_ATTR_PREFIX = 'ext_coords__'#
- LDG_COORDS_MODE_ATTR_PREFIX = 'ext_coords_mode__'#
- LDG_COORDS_MODE_DEFAULT = 'scalar'#
- LDG_STRICT_ATTR_CHECKING = False#
- LDG_COORDS_SEPARATOR_IN_NAME = ';'#
- _COLLECTIVE_SELECT_THRESHOLD = 1.8#
- __init__(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)[source]#
Initialize a LabelledDataGroup
- Parameters
*args – Passed on to
OrderedDataGroup
dims (TDims, optional) – The dimensions associated with this group. If not given, will use those defined in the
LDG_DIMS
class variable. These can not be changed afterwards!mode (str, optional) – By which coordinate extraction mode to get the coordinates from the group members. Can be
attrs
,name
,data
or anything else specified inextract_coords()
.allow_deep_selection (bool, optional) – Whether to allow deep selection. If not given, will use the
LDG_ALLOW_DEEP_SELECTION
class variable’s value. Behaviour can be changed via the property of the same name.**kwargs – Passed on to
OrderedDataGroup
- property dims: Tuple[str]#
The names of the group-level dimensions this group manages.
It _may_ contain dimensions that overlap with dimension names from the members; this is intentional.
- property coords: Dict[str, List[dantro.utils.coords.TCoord]]#
Returns a dict-like container of group-level coordinate values keyed by dimension.
- property member_map: DataArray#
Returns an array that represents the space that the members of this group span, where each value (i.e. a specific coordinate combination) is the name of the corresponding member of this group.
Upon first call, this is computed here. If members are added, it is tried to accomodate them in there; if not possible, the cache will be invalidated.
The member map _may_ include empty strings, i.e. coordinate combinations that are not covered by any member. Also, they can contain duplicate names, as one member can cover multiple coordinates.
Note
The member map is invalidated when new members are added that can not be accomodated in it. It will be recalculated when needed.
- isel(indexers: dict = None, *, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) DataArray [source]#
Return a new labelled
xarray.DataArray
with an index-selected subset of members of this group.If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.
Note
For data combination (via any
combination_method
) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using xarray’sxarray.align()
function and the respective coordinates. See the xarray documentation for more information about coordinates.- Parameters
indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick indices. As
xarray.DataArray.isel()
, uses pandas-like indexing, i.e.: slices do not include the terminal value.drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.
combination_method (str, optional) –
How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:
concat
: Concatenate. This can preserve the dtype, but requires that no data is missing.merge
: Merge, usingxarray.merge()
. This leads to a type conversion tofloat64
, but allows members being missing or coordinates not fully filling the available space.try_concat
: Try concatenation, fall back to merging if that was unsuccessful.auto
: Automatically deduce suitably combination method. Usemerge
if data is non-integer type andtry_concat
otherwise.
Note
Selecting all data (by not passing any
indexers
) can be significantly faster using themerge
combination method than using theconcat
method.deep (bool, optional) – Whether to allow deep indexing, i.e.: that
indexers
may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. IfNone
, will use the value returned by theallow_deep_selection
property.**indexers_kwargs – Additional indexers
- Returns
- The selected data, potentially a combination of
data on group level and member-level data.
- Return type
- sel(indexers: dict = None, *, method: str = None, tolerance: float = None, drop: bool = False, combination_method: str = 'auto', deep: bool = None, **indexers_kwargs) DataArray [source]#
Return a new labelled
xarray.DataArray
with a coordinate-selected subset of members of this group.If deep selection is activated, those indexers that are not available in the group-managed dimensions are looked up in the members of this group.
Note
For data combination (via any
combination_method
) dimensions that differ in size across group members have to be labelled, such that arrays can be aligned using xarray’sxarray.align()
function and the respective coordinates. See the xarray documentation for more information about coordinates.- Parameters
indexers (dict, optional) – A dict with keys matching dimensions and values given by scalars, slices or arrays of tick labels. As
xarray.DataArray.sel()
, uses pandas-like indexing, i.e.: slices include the terminal value.method (str, optional) – Method to use for inexact matches
tolerance (float, optional) – Maximum (absolute) distance between original and given label for inexact matches.
drop (bool, optional) – Whether to drop coordinate variables instead of making them scalar.
combination_method (str, optional) –
How to combine group-level data with member-level data. Ignored if data from a single group member is selected, i.e. no data has to be combined. Can be:
concat
: Concatenate. This can preserve the dtype, but requires that no data is missing.merge
: Merge, usingxarray.merge()
. This leads to a type conversion tofloat64
, but allows members being missing or coordinates not fully filling the available space.try_concat
: Try concatenation, fall back to merging if that was unsuccessful.auto
: Automatically deduce suitably combination method. Usemerge
if data is non-integer type andtry_concat
otherwise.
Note
Selecting all data (by not passing any
indexers
) can be significantly faster using themerge
combination method than using theconcat
method.deep (bool, optional) – Whether to allow deep indexing, i.e.: that
indexers
may contain dimensions that don’t refer to group- level dimensions but to dimensions that are only availble among the member data. IfNone
, will use the value returned by theallow_deep_selection
property.**indexers_kwargs – Additional indexers
- Returns
- The selected data, potentially a combination of
data on group level and member-level data.
- Return type
- _get_coords_of(obj: AbstractDataContainer) Dict[str, Sequence[dantro.utils.coords.TCoord]] [source]#
Extract the coordinates for the given object using the
extract_coords()
function.- Parameters
obj (AbstractDataContainer) – The object to get the coordinates of.
- Returns
The extracted coordinates
- Return type
TCoordsDict
- _add_container_callback(cont: AbstractDataContainer) None [source]#
Called by the base class after adding a container, this method checks whether the member map needs to be invalidated or whether the new container can be accomodated in it.
If it can be accomodated, the member map will be adjusted such that for all coordinates associated with the given
cont
, the member map points to the newly added container.- Parameters
cont (AbstractDataContainer) – The newly added container
- _parse_indexers(indexers: dict, *, allow_deep: bool, **indexers_kwargs) Tuple[dict, dict] [source]#
Parses the given indexer arguments and split them into indexers for the selection of group members and deep selection.
- Parameters
- Returns
(shallow indexers, deep indexers)
- Return type
- Raises
ValueError – If deep indexers were given but deep selection was not enabled
- _get_cont(name: str, *, combination_method: str) Optional[XrDataContainer] [source]#
Retrieve the container from the group. If no container could be found, returns None, which denotes that further processing should be skipped.
- Parameters
- Returns
The extracted container
- Return type
Union[XrDataContainer, None]
- Raises
ItemAccessError – If
combination_method == "concat"
, on invalid container name.
- _process_cont(cont, *, coords, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) DataArray [source]#
Process the given container and coordinates into a data array; this applies selection along container dimensions that overlap with the group dimensions as well as deep selection.
- Parameters
cont – The container to be processed
coords – The DataArrayCoordinates of the given container in the preselected member map.
shallow_indexers (dict) – Indexers that were used to preselect the member map.
deep_indexers (dict) – Indexers to be applied to the container
by_index (bool) – Whether to select by index
drop (bool) – Whether to drop coordinate variables instead of making them scalar.
**sel_kwargs – Passed to
sel()
.
- Returns
The processed container data
- Return type
- Raises
ValueError – In
name
mode, on conflicting non-dimension container coordinates.
- _select(*, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) DataArray [source]#
Preselect the member map (if needed) and designate a suitable method for further processing and selection based on the given combination method and indexers.
If possible, take shortcuts when selecting all data or when selecting data from a single group member.
- Parameters
combination_method (str) – How to combine the member data.
shallow_indexers (dict) – Indexers to be applied on the group-level.
deep_indexers (dict) – Indexers to be applied on the member-level only.
by_index (bool) – Whether to select by index.
drop (bool) – Whether to drop coordinate variables instead of making them scalar.
**sel_kwargs – Passed to
sel()
.
- Returns
The selected data.
- Return type
- Raises
ValueError – On invalid
combination_method
.
- _select_single(cont_names: DataArray, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) DataArray [source]#
Select data from a single group member. Expects the preselected member map to contain only a single valid container name.
- _select_all_merge() DataArray [source]#
Select all group data by directly merging all containers. This circumvents building the member map. This might fail, e.g. if there are conflicting or duplicate coordinates.
- _ALLOWED_CONT_TYPES: Optional[tuple] = None#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_GROUP_CLS: type = None#
Which class to use when creating a new group via
new_group()
. If None, the type of the current instance is used for the new group.
- _STORAGE_CLS#
alias of
collections.OrderedDict
- __contains__(cont: Union[str, AbstractDataContainer]) bool #
Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __getitem__(key: Union[str, List[str]]) AbstractDataContainer #
Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
- __iter__()#
Returns an iterator over the OrderedDict
- __setitem__(key: Union[str, List[str]], val: BaseDataContainer) None #
This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont: AbstractDataContainer) None #
Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _select_generic(cont_names: DataArray, *, combination_method: str, shallow_indexers: dict, deep_indexers: dict, by_index: bool, drop: bool, **sel_kwargs) DataArray [source]#
Select data from group members using the given indexers and combine it via the specified method. If deep indexers are given, apply the deep indexing on each of the members.
This method receives a labelled array of container names, on which the selection already took place. The aim is now to align the objects these names refer to, including their coordinates, and thereby construct an array that contains both the dimensions given by the
cont_names
array and each members’ data dimensions.Available combination methods are based either on
xarray.merge()
operations orxarray.concat()
along each dimension. For both these combination methods, the members of this group need to be prepared such that the operation can be applied, i.e.: they need to already be in an array capable of that operation and they need to directly or indirectly preserve coordinate information.For that purpose, an object-array is constructed holding the processed member data. As the
xarray.Dataset
andxarray.DataArray
types have issues with handling array-like objects in object arrays, this is done via anumpy.ndarray
.- Parameters
cont_names (DataArray) – The pre-selected member map object, i.e. a labelled array containing names of the desired members that are to be combined.
combination_method (str) – How to combine them:
concat
,try_concat
, ormerge
. Concatenation will allow preserving the dtype of the underlying data.shallow_indexers (dict) – Indexer arguments that were used for the group member selection.
deep_indexers (dict) – Indexer arguments for deep selection to be done before combination.
by_index (bool) – Whether the deep indexing should take place by index; if False, will use label-based selection.
**sel_kwargs – Passed on to
sel()
.
- Returns
- The selected data of the members from
cont_names
, combined using the given combination method.
- Return type
- Raises
ValueError – On conflicting coordinate information on group-level and member-level.
KeyError – In
concat
mode, upon missing members.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
- classmethod _combine_by_merge(dsets: ndarray) Dataset [source]#
Combine the given datasets by merging using xarray’s
xarray.merge()
.- Parameters
dsets (ndarray) – The
object
-dtype array ofxarray.Dataset
objects that are to be combined.- Returns
- All datasets, aligned and combined via
- Return type
- classmethod _combine_by_concatenation(dsets: ndarray, *, dims: Tuple[str]) Dataset [source]#
Combine the given datasets by concatenation using xarray’s
xarray.concat()
and subsequent application along all dimensions specified indims
.- Parameters
dsets (ndarray) – The object-dtype array of
xarray.Dataset
objects that are to be combined by concatenation.dims (TDims) – The dimension names corresponding to all the dimensions of the
dsets
array.
- Returns
The dataset resulting from the concatenation
- Return type
dantro.groups.ordered module#
In this module, the BaseDataGroup is specialized for holding members in a specific order.
- class OrderedDataGroup(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)[source]#
Bases:
dantro.base.BaseDataGroup
,collections.abc.MutableMapping
The OrderedDataGroup class manages groups of data containers, preserving the order in which they were added to this group.
It uses an
collections.OrderedDict
to associate containers with this group.- _STORAGE_CLS#
alias of
collections.OrderedDict
- _ALLOWED_CONT_TYPES: Optional[tuple] = None#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS: type = None#
Which class to use for creating a new container via call to the
new_container()
method. If None, the type needs to be specified explicitly in the method call.
- _NEW_GROUP_CLS: type = None#
Which class to use when creating a new group via
new_group()
. If None, the type of the current instance is used for the new group.
- __contains__(cont: Union[str, AbstractDataContainer]) bool #
Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __getitem__(key: Union[str, List[str]]) AbstractDataContainer #
Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
- __init__(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)#
Initialize a BaseDataGroup, which can store other containers and attributes.
- Parameters
name (str) – The name of this data container
containers (list, optional) – The containers that are to be stored as members of this group. If given, these are added one by one using the .add method.
attrs (None, optional) – A mapping that is stored as attributes
parent (AbstractDataGroup, optional) – If known, the parent group, which can be used to extract information during initialization. Note that linking occurs only after the group was added to the parent group, i.e. after initialization finished.
- __iter__()#
Returns an iterator over the OrderedDict
- __setitem__(key: Union[str, List[str]], val: BaseDataContainer) None #
This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont: AbstractDataContainer) None #
Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
- class IndexedDataGroup(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)[source]#
Bases:
dantro.mixins.indexing.IntegerItemAccessMixin
,dantro.groups.ordered.OrderedDataGroup
The IndexedDataGroup class holds members that are of the same type and have names that can directly be interpreted as positive integers.
Especially, this group maintains the correct order of members according to integer ordering.
To speed up element insertion, this group keeps track of recently added container names, which are then used as hints for subsequent insertions.
Note
Albeit the members of this group being ordered, item access still refers to the names of the members, not their index within the sequence!
Warning
With the underlying ordering mechanism of
KeyOrderedDict
, the performance of this data structure is sensitive to the insertion order of elements.It is fastest for in-order insertions, where the complexity per insertion is constant (regardless of whether insertion order is ascending or descending). For out-of-order insertions, the whole key map may have to be searched, in which case the complexity scales with the number of elements in this group.
Hint
If experiencing trouble with the performance of this data structure, sort elements before adding them to this group.
- _STORAGE_CLS#
- _NEW_GROUP_CLS#
- key_at_idx(idx: int) str [source]#
Get a key by its index within the container. Can be negative.
- Parameters
idx (int) – The index within the member sequence
- Returns
The desired key
- Return type
- Raises
IndexError – Index out of range
- _add_container_to_data(cont) None [source]#
Adds a container to the underlying integer-ordered dictionary.
Unlike the parent method, this uses
insert()
in order to provide hints regarding the insertion position. It is optimised for insertion in ascending order.
- _ipython_key_completions_() List[int] [source]#
For ipython integration, return a list of available keys.
Unlike the BaseDataGroup method, which returns a list of strings, this returns a list of integers.
- _ALLOWED_CONT_TYPES: Optional[tuple] = None#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS: type = None#
Which class to use for creating a new container via call to the
new_container()
method. If None, the type needs to be specified explicitly in the method call.
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __init__(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)#
Initialize a BaseDataGroup, which can store other containers and attributes.
- Parameters
name (str) – The name of this data container
containers (list, optional) – The containers that are to be stored as members of this group. If given, these are added one by one using the .add method.
attrs (None, optional) – A mapping that is stored as attributes
parent (AbstractDataGroup, optional) – If known, the parent group, which can be used to extract information during initialization. Note that linking occurs only after the group was added to the parent group, i.e. after initialization finished.
- __iter__()#
Returns an iterator over the OrderedDict
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
dantro.groups.psp module#
This module implements BaseDataContainer
specializations that make use of features from the
paramspace package, in particular the
ParamSpace
class.
- class ParamSpaceStateGroup(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)[source]#
Bases:
dantro.groups.ordered.OrderedDataGroup
A ParamSpaceStateGroup is meant to be used as a member group of the
ParamSpaceGroup
.While its own name need be interpretable as a positive integer (enforced in the enclosing
ParamSpaceGroup
but also here), it can hold members with any name.- _NEW_GROUP_CLS#
- property coords: dict#
Retrieves the coordinates of this group within the parameter space described by the
ParamSpaceGroup
this group is enclosed in.- Returns
- The coordinates of this group, keys being dimension names and
values being the coordinate values for this group.
- Return type
- _ALLOWED_CONT_TYPES: Optional[tuple] = None#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS: type = None#
Which class to use for creating a new container via call to the
new_container()
method. If None, the type needs to be specified explicitly in the method call.
- _STORAGE_CLS#
alias of
collections.OrderedDict
- __contains__(cont: Union[str, AbstractDataContainer]) bool #
Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __getitem__(key: Union[str, List[str]]) AbstractDataContainer #
Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
- __init__(*, name: str, containers: Optional[list] = None, attrs=None, parent: Optional[AbstractDataGroup] = None)#
Initialize a BaseDataGroup, which can store other containers and attributes.
- Parameters
name (str) – The name of this data container
containers (list, optional) – The containers that are to be stored as members of this group. If given, these are added one by one using the .add method.
attrs (None, optional) – A mapping that is stored as attributes
parent (AbstractDataGroup, optional) – If known, the parent group, which can be used to extract information during initialization. Note that linking occurs only after the group was added to the parent group, i.e. after initialization finished.
- __iter__()#
Returns an iterator over the OrderedDict
- __setitem__(key: Union[str, List[str]], val: BaseDataContainer) None #
This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont: AbstractDataContainer) None #
Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
- class ParamSpaceGroup(*, name: str, pspace: Optional[ParamSpace] = None, containers: Optional[list] = None, **kwargs)[source]#
Bases:
dantro.mixins.indexing.PaddedIntegerItemAccessMixin
,dantro.groups.ordered.IndexedDataGroup
The ParamSpaceGroup is associated with a
paramspace.paramspace.ParamSpace
object and the loaded results of an iteration over this parameter space.Thus, the groups that are stored in the ParamSpaceGroup need all relate to a state of the parameter space, identified by a zero-padded string name. In fact, this group allows no other kinds of groups stored inside.
To make access to a specific state easier, it allows accessing a state by its state number as integer.
- _PSPGRP_PSPACE_ATTR_NAME = 'pspace'#
- _PSPGRP_TRANSFORMATOR = None#
- _NEW_GROUP_CLS#
- _ALLOWED_CONT_TYPES: Optional[tuple] = (<class 'dantro.groups.psp.ParamSpaceStateGroup'>,)#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- __init__(*, name: str, pspace: Optional[ParamSpace] = None, containers: Optional[list] = None, **kwargs)[source]#
Initialize a OrderedDataGroup from the list of given containers.
- Parameters
name (str) – The name of this group.
pspace (ParamSpace, optional) – Can already pass a ParamSpace object here.
containers (list, optional) – A list of containers to add, which need to be
ParamSpaceStateGroup
objects.**kwargs – Further initialisation kwargs, e.g.
attrs
…
- property pspace: Optional[ParamSpace]#
Reads the entry named
_PSPGRP_PSPACE_ATTR_NAME
in.attrs
and returns aParamSpace
object, if available there.- Returns
- The associated
parameter space, or None, if there is none associated yet.
- Return type
Union[ParamSpace, None]
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS: type = None#
Which class to use for creating a new container via call to the
new_container()
method. If None, the type needs to be specified explicitly in the method call.
- _PADDED_INT_FSTR: str = None#
The format string to generate a padded integer; deduced upon first call
- _PADDED_INT_KEY_WIDTH: int = None#
The number of digits of the padded string representing the integer
- _PADDED_INT_MAX_VAL: int = None#
The allowed maximum value of an integer key; checked only in strict mode
- _PADDED_INT_STRICT_CHECKING: bool = True#
Whether to use strict checking when parsing keys, i.e. check that the range of keys is valid and an error is thrown when an integer key was given that cannot be represented consistently by a padded string of the determined key width.
- _STORAGE_CLS#
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __iter__()#
Returns an iterator over the OrderedDict
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_to_data(cont) None #
Adds a container to the underlying integer-ordered dictionary.
Unlike the parent method, this uses
insert()
in order to provide hints regarding the insertion position. It is optimised for insertion in ascending order.
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont: AbstractDataContainer) None #
This method is invoked when adding a member to a group and makes sure the name of the added group is correctly zero-padded.
Also, upon first call, communicates the zero padded integer key width, i.e.: the length of the container name, to the PaddedIntegerItemAccessMixin.
- Parameters
cont – The member container to add
- Returns
None: No return value needed
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- _determine_container_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new container.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_CONTAINER_CLS
class variable. If a string, tries to extract it from the class variable_DATA_CONTAINER_CLASSES
dict. Otherwise, assumes this is already a type.- Returns
The container class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_group_type(Cls: Union[type, str]) type #
Helper function to determine the type to use for a new group.
- Parameters
Cls (Union[type, str]) – If None, uses the
_NEW_GROUP_CLS
class variable. If that one is not set, usestype(self)
. If a string, tries to extract it from the class variable_DATA_GROUP_CLASSES
dict. Otherwise, assumesCls
is already a type.- Returns
The group class to use
- Return type
- Raises
ValueError – If the string class name was not registered
AttributeError – If no default class variable was set
- _determine_type(T: Union[type, str], *, default: type, registry: Dict[str, type]) type #
Helper function to determine a type by name, falling back to a default type or looking it up from a dict-like registry if it is a string.
- _direct_insertion_mode(*, enabled: bool = True)#
A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
- _enter_direct_insertion_mode()#
Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
- _exit_direct_insertion_mode()#
Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
- _format_info() str #
A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
- _format_logstr() str #
A __format__ helper function: returns the log string, a combination of class name and name
- _format_tree() str #
Returns the default tree representation of this group by invoking the .tree property
- _format_tree_condensed() str #
Returns the default tree representation of this group by invoking the .tree property
- _ipython_key_completions_() List[int] #
For ipython integration, return a list of available keys.
Unlike the BaseDataGroup method, which returns a list of strings, this returns a list of integers.
- _link_child(*, new_child: BaseDataContainer, old_child: Optional[BaseDataContainer] = None)#
Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
- _lock_hook()#
Invoked upon locking.
- _tree_repr(*, level: int = 0, max_level: Optional[int] = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Optional[Union[int, Callable[[int, int], int]]] = None, total_item_count: int = 0) Union[str, List[str]] #
Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
- _unlink_child(child: BaseDataContainer)#
Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
- _unlock_hook()#
Invoked upon unlocking.
- property attrs#
The container attributes.
- clear()#
Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
- property data#
The stored data.
- get(key, default=None)#
Return the container at key, or default if container with name key is not available.
- items()#
Returns an iterator over the (name, data container) tuple of this group.
- key_at_idx(idx: int) str #
Get a key by its index within the container. Can be negative.
- Parameters
idx (int) – The index within the member sequence
- Returns
The desired key
- Return type
- Raises
IndexError – Index out of range
- keys()#
Returns an iterator over the container names in this group.
- lock()#
Locks the data of this object
- new_container(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, _target_is_group: bool = False, **kwargs) BaseDataContainer #
Creates a new container of type
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
Cls (Union[type, str], optional) – The type of the target container (or group) that is to be added. If None, will use the type set in
_NEW_CONTAINER_CLS
class variable. If a string is given, the type is looked up in the container type registry.GroupCls (Union[type, str], optional) – Like
Cls
but used for intermediate group types only._target_is_group (bool, optional) – Internally used variable. If True, will look up the
Cls
type via_determine_group_type()
instead of_determine_container_type()
.**kwargs – passed on to
Cls.__init__
- Returns
The created container of type
Cls
- Return type
- new_group(path: Union[str, List[str]], *, Cls: Optional[Union[type, str]] = None, GroupCls: Optional[Union[type, str]] = None, **kwargs) BaseDataGroup #
Creates a new group at the given path.
- Parameters
path (Union[str, List[str]]) – The path to create the group at. If necessary, intermediate paths will be created.
Cls (Union[type, str], optional) –
If given, use this type to create the target group. If not given, uses the class specified in the
_NEW_GROUP_CLS
class variable or (if a string) the one from the group type registry.Note
This argument is evaluated at each segment of the
path
by the corresponding object in the tree. Subsequently, the types need to be available at the desiredGroupCls (Union[type, str], optional) – Like
Cls
, but this applies only to the creation of intermediate groups.**kwargs – Passed on to
Cls.__init__
- Returns
The created group of type
Cls
- Return type
- property only_default_data_present: bool#
Returns true if only data for the default point in parameter space is available in this group.
- property padded_int_key_width: Optional[int]#
Returns the width of the zero-padded integer key or None, if it is not already specified.
- property parent#
The associated parent of this container or group
- pop(k[, d]) v, remove specified key and return the corresponding value. #
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair #
as a 2-tuple; but raise KeyError if D is empty.
- raise_if_locked(*, prefix: Optional[str] = None)#
Raises an exception if this object is locked; does nothing otherwise
- recursive_update(other, *, overwrite: bool = True)#
Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
- setdefault(key, default=None)#
This method is not supported for a data group
- property tree_condensed: str#
Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
- unlock()#
Unlocks the data of this object
- update([E, ]**F) None. Update D from mapping/iterable E and F. #
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
- values()#
Returns an iterator over the containers in this group.
- property with_direct_insertion: bool#
Whether the class this mixin is mixed into is currently in direct insertion mode.
- __locked#
Whether the data is regarded as locked. Note name-mangling here.
- __in_direct_insertion_mode#
A name-mangled state flag that determines the state of the object.
- select(*, field: Union[str, List[str]] = None, fields: Dict[str, List[str]] = None, subspace: dict = None, method: str = 'concat', idx_as_label: bool = False, base_path: str = None, **kwargs) Dataset [source]#
Selects a multi-dimensional slab of this ParamSpaceGroup and the specified fields and returns them bundled into an
xarray.Dataset
with labelled dimensions and coordinates.- Parameters
field (Union[str, List[str]], optional) – The field of data to select. Should be path or a list of strings that points to an entry in the data tree. To select multiple fields, do not pass this argument but use the fields argument.
fields (Dict[str, List[str]], optional) – A dict specifying the fields that are to be loaded into the dataset. Keys will be the names of the resulting variables, while values should specify the path to the field in the data tree. Thus, they can be strings, lists of strings or dicts with the path key present. In the latter case, a dtype can be specified via the dtype key in the dict.
subspace (dict, optional) – Selector for a subspace of the parameter space. Adheres to the parameter space’s
activate_subspace()
signature.method (str, optional) –
How to combine the selected datasets.
concat
: concatenate sequentially along all parameter space dimensions. This can preserve the data type but it does not work if one data point is missing.merge
: merge always works, even if data points are missing, but will convert all dtypes to float.
idx_as_label (bool, optional) – If true, adds the trivial indices as labels for those dimensions where coordinate labels were not extractable from the loaded field. This allows merging for data with different extends in an unlabelled dimension.
base_path (str, optional) – If given,
path
specifications for each field can be seen as relative to this path**kwargs – Passed along either to xr.concat or xr.merge, depending on the
method
argument.
- Raises
KeyError – On invalid state key.
ValueError – Raised in multiple scenarios: If no
ParamSpace
was associated with this group, for wrong argument values, if the data to select cannot be extracted with the given argument values, exceptions passed on from xarray.
- Returns
- The selected hyperslab of the parameter space,
holding the desired fields.
- Return type
dantro.groups.time_series module#
Implements LabelledDataGroup
specializations for time series data.
- class TimeSeriesGroup(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)[source]#
Bases:
dantro.groups.labelled.LabelledDataGroup
,dantro.groups.ordered.IndexedDataGroup
A time-series group assumes that each stored member refers to one point in time, where the name is to be interpreted as the time coordinate.
For more information on selection methods, see:
- LDG_DIMS = ('time',)#
Expected dimension names. There is only one dimension in a TimeSeriesGroup:
time
- LDG_EXTRACT_COORDS_FROM = 'name'#
Where to extract time coordinates from. Here, the container name is expected to be the time coordinate.
- LDG_ALLOW_DEEP_SELECTION = True#
- LDG_COORDS_ATTR_PREFIX = 'ext_coords__'#
- LDG_COORDS_MODE_ATTR_PREFIX = 'ext_coords_mode__'#
- LDG_COORDS_MODE_DEFAULT = 'scalar'#
- LDG_COORDS_SEPARATOR_IN_NAME = ';'#
- LDG_STRICT_ATTR_CHECKING = False#
- _ALLOWED_CONT_TYPES: Optional[tuple] = None#
The types that are allowed to be stored in this group. If None, all types derived from the dantro base classes are allowed. This applies to both containers and groups that are added to this group.
Hint
To add the type of the current object, add a string entry
self
to the tuple. This will be resolved totype(self)
at invocation.
- _ATTRS_CLS#
alias of
dantro.base.BaseDataAttrs
- _COLLECTIVE_SELECT_THRESHOLD = 1.8#
- _COND_TREE_CONDENSE_THRESH = 10#
Condensed tree representation threshold parameter
- _COND_TREE_MAX_LEVEL = 10#
Condensed tree representation maximum level
- _DATA_CONTAINER_CLASSES: Dict[str, type] = None#
Mapping from strings to available data container types. Used in string-based lookup of container types in
new_container()
.
- _DATA_GROUP_CLASSES: Dict[str, type] = None#
Mapping from strings to available data group types. Used in string-based lookup of group types in
new_group()
.
- _NEW_CONTAINER_CLS#
alias of
dantro.containers.xr.XrDataContainer
- _NEW_GROUP_CLS#
- _STORAGE_CLS#
- __eq__(other) bool #
Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
- __format__(spec_str: str) str #
Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
- __init__(*args, dims: Optional[Tuple[str]] = None, mode: Optional[str] = None, allow_deep_selection: Optional[bool] = None, **kwargs)#
Initialize a LabelledDataGroup
- Parameters
*args – Passed on to
OrderedDataGroup
dims (TDims, optional) – The dimensions associated with this group. If not given, will use those defined in the
LDG_DIMS
class variable. These can not be changed afterwards!mode (str, optional) – By which coordinate extraction mode to get the coordinates from the group members. Can be
attrs
,name
,data
or anything else specified inextract_coords()
.allow_deep_selection (bool, optional) – Whether to allow deep selection. If not given, will use the
LDG_ALLOW_DEEP_SELECTION
class variable’s value. Behaviour can be changed via the property of the same name.**kwargs – Passed on to
OrderedDataGroup
- __iter__()#
Returns an iterator over the OrderedDict
- __sizeof__() int #
Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof()
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
- __str__() str #
An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
- _abc_impl = <_abc_data object>#
- _add_container_callback(cont: AbstractDataContainer) None #
Called by the base class after adding a container, this method checks whether the member map needs to be invalidated or whether the new container can be accomodated in it.
If it can be accomodated, the member map will be adjusted such that for all coordinates associated with the given
cont
, the member map points to the newly added container.- Parameters
cont (AbstractDataContainer) – The newly added container
- _add_container_to_data(cont) None #
Adds a container to the underlying integer-ordered dictionary.
Unlike the parent method, this uses
insert()
in order to provide hints regarding the insertion position. It is optimised for insertion in ascending order.
- _attrs = None#
The attribute that data attributes will be stored to
- _check_cont(cont) None #
Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
- _check_data(data: Any) None #
This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
- _check_name(new_name: str) None #
Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
- classmethod _combine_by_concatenation(dsets: ndarray, *, dims: Tuple[str]) Dataset #
Combine the given datasets by concatenation using xarray’s
xarray.concat()
and subsequent application along all dimensions specified indims
.- Parameters
dsets (ndarray) – The object-dtype array of
xarray.Dataset
objects that are to be combined by concatenation.dims (TDims) – The dimension names corresponding to all the dimensions of the
dsets
array.
- Returns
The dataset resulting from the concatenation
- Return type
- classmethod _combine_by_merge(dsets: ndarray) Dataset #
Combine the given datasets by merging using xarray’s
xarray.merge()
.- Parameters
dsets (ndarray) – The
object
-dtype array ofxarray.Dataset
objects that are to be combined.- Returns
- All datasets, aligned and combined via
- Return type