dantro.data_mngr module¶
This module implements the DataManager class, the root of the data tree.
-
dantro.data_mngr.
fmt_time
(seconds)¶
-
dantro.data_mngr.
_load_file_wrapper
(filepath: str, *, dm: dantro.data_mngr.DataManager, loader: str, **kwargs) → Tuple[dantro.base.BaseDataGroup, str][source]¶ A wrapper around
_load_file()
that is used for parallel loading via multiprocessing.Pool. It takes care of resolving the loader function and instantiating the file- loading method.This function needs to be on the module scope such that it is pickleable. For that reason, loader resolution also takes place here, because pickling the load function may be problematic.
- Parameters
filepath (str) – The path of the file to load data from
dm (DataManager) – The DataManager instance to resolve the loader from
loader (str) – The namer of the loader
**kwargs – Any further loading arguments.
- Returns
- The return value of
- Return type
Tuple[BaseDataContainer, str]
-
dantro.data_mngr.
_parse_parallel_opts
(files: List[str], *, enabled: bool = True, processes: int = None, min_files: int = 2, min_total_size: int = None) → int[source]¶ Parser function for the parallel file loading options dict
- Parameters
files (List[str]) – List of files that are to be loaded
enabled (bool, optional) – Whether to use parallel loading. If True, the threshold arguments will still need to be fulfilled.
processes (int, optional) – The number of processors to use; if this is a negative integer, will deduce from available CPU count.
min_files (int, optional) – If there are fewer files to load than this number, will not use parallel loading.
min_total_size (int, optional) – If the total file size is smaller than this file size (in bytes), will not use parallel loading.
- Returns
- number of processes to use. Will return 1 if loading should not
happen in parallel. Additionally, this number will never be larger than the number of files in order to prevent unnecessary processes.
- Return type
int
-
class
dantro.data_mngr.
DataManager
(data_dir: str, *, name: str = None, load_cfg: Union[dict, str] = None, out_dir: Union[str, bool] = '_output/{timestamp:}', out_dir_kwargs: dict = None, create_groups: List[Union[str, dict]] = None, condensed_tree_params: dict = None, default_tree_cache_path: str = None)[source]¶ Bases:
dantro.groups.ordered.OrderedDataGroup
The DataManager is the root of a data tree, coupled to a specific data directory.
It handles the loading of data and can be used for interactive work with the data.
-
_BASE_LOAD_CFG
= None¶
-
_DEFAULT_GROUPS
= None¶
-
_DATA_GROUP_DEFAULT_CLS
¶
-
_DATA_GROUP_CLASSES
= None¶
-
_DEFAULT_TREE_CACHE_PATH
= '.tree_cache.d3'¶
-
__init__
(data_dir: str, *, name: str = None, load_cfg: Union[dict, str] = None, out_dir: Union[str, bool] = '_output/{timestamp:}', out_dir_kwargs: dict = None, create_groups: List[Union[str, dict]] = None, condensed_tree_params: dict = None, default_tree_cache_path: str = None)[source]¶ Initializes a DataManager for the specified data directory.
- Parameters
data_dir (str) – the directory the data can be found in. If this is a relative path, it is considered relative to the current working directory.
name (str, optional) – which name to give to the DataManager. If no name is given, the data directories basename will be used
load_cfg (Union[dict, str], optional) – The base configuration used for loading data. If a string is given, assumes it to be the path to a YAML file and loads it using the
load_yml()
function. If None is given, it can still be supplied to theload()
method later on.out_dir (Union[str, bool], optional) – where output is written to. If this is given as a relative path, it is considered relative to the
data_dir
. A formatting operation with the keystimestamp
andname
is performed on this, where the latter is the name of the data manager. If set to False, no output directory is created.out_dir_kwargs (dict, optional) – Additional arguments that affect how the output directory is created.
create_groups (List[Union[str, dict]], optional) – If given, these groups will be created after initialization. If the list entries are strings, the default group class will be used; if they are dicts, the name key specifies the name of the group and the Cls key specifies the type. If a string is given instead of a type, the lookup happens from the
_DATA_GROUP_CLASSES
variable.condensed_tree_params (dict, optional) – If given, will set the parameters used for the condensed tree representation. Available options:
max_level
andcondense_thresh
, where the latter may be a callable. Seedantro.base.BaseDataGroup._tree_repr()
for more information.default_tree_cache_path (str, optional) – The path to the default tree cache file. If not given, uses the value from the class variable
_DEFAULT_TREE_CACHE_PATH
. Whichever value was chosen is then prepared using the_parse_file_path()
method, which regards relative paths as being relative to the associated data directory.
-
_init_dirs
(*, data_dir: str, out_dir: Union[str, bool], timestamp: float = None, timefstr: str = '%y%m%d-%H%M%S', exist_ok: bool = False) → Dict[str, str][source]¶ Initializes the directories managed by this DataManager and returns a dictionary that stores the absolute paths to these directories.
If they do not exist, they will be created.
- Parameters
data_dir (str) – the directory the data can be found in. If this is a relative path, it is considered relative to the current working directory.
out_dir (Union[str, bool]) – where output is written to. If this is given as a relative path, it is considered relative to the data directory. A formatting operation with the keys
timestamp
andname
is performed on this, where the latter is the name of the data manager. If set to False, no output directory is created.timestamp (float, optional) – If given, use this time to generate the date format string key. If not, uses the current time.
timefstr (str, optional) – Format string to use for generating the string representation of the current timestamp
exist_ok (bool, optional) – Whether the output directory may exist. Note that it only makes sense to set this to True if you can be sure that there will be no file conflicts! Otherwise the errors will just occur at a later stage.
- Returns
- The directory paths registered under certain keys,
e.g.
data
andout
.
- Return type
Dict[str, str]
-
property
hashstr
¶ The hash of a DataManager is computed from its name and the coupled data directory, which are regarded as the relevant parts. While other parts of the DataManager are not invariant, it is characterized most by the directory it is associated with.
As this is a string-based hash, it is not implemented as the __hash__ magic method but as a separate property.
- WARNING Changing how the hash is computed for the DataManager will
invalidate all TransformationDAG caches.
-
property
tree_cache_path
¶ Absolute path to the default tree cache file
-
property
tree_cache_exists
¶ Whether the tree cache file exists
-
load_from_cfg
(*, load_cfg: dict = None, update_load_cfg: dict = None, exists_action: str = 'raise', print_tree: Union[bool, str] = False) → None[source]¶ Load multiple data entries using the specified load configuration.
- Parameters
load_cfg (dict, optional) – The load configuration to use. If not given, the one specified during initialization is used.
update_load_cfg (dict, optional) – If given, it is used to update the load configuration recursively
exists_action (str, optional) – The behaviour upon existing data. Can be:
raise
(default),skip
,skip_nowarn
,overwrite
,overwrite_nowarn
. With the*_nowarn
values, no warning is given if an entry already existed.print_tree (Union[bool, str], optional) – If True, the full tree representation of the DataManager is printed after the data was loaded. If
'condensed'
, the condensed tree will be printed.
- Raises
TypeError – Raised if a given configuration entry was of invalid type, i.e. not a dict
-
load
(entry_name: str, *, loader: str, enabled: bool = True, glob_str: Union[str, List[str]], base_path: str = None, target_group: str = None, target_path: str = None, print_tree: Union[bool, str] = False, load_as_attr: bool = False, parallel: Union[bool, dict] = False, **load_params) → None[source]¶ Performs a single load operation.
- Parameters
entry_name (str) – Name of this entry; will also be the name of the created group or container, unless
target_basename
is givenloader (str) – The name of the loader to use
enabled (bool, optional) – Whether the load operation is enabled. If not, simply returns without loading any data or performing any further checks.
glob_str (Union[str, List[str]]) – A glob string or a list of glob strings by which to identify the files within
data_dir
that are to be loaded using the given loader functionbase_path (str, optional) – The base directory to concatenate the glob string to; if None, will use the DataManager’s data directory. With this option, it becomes possible to load data from a path outside the associated data directory.
target_group (str, optional) – If given, the files to be loaded will be stored in this group. This may only be given if the argument target_path is not given.
target_path (str, optional) – The path to write the data to. This can be a format string. It is evaluated for each file that has been matched. If it is not given, the content is loaded to a group with the name of this entry at the root level. Available keys are:
basename
,match
(ifpath_regex
is used, see**load_params
)print_tree (Union[bool, str], optional) – If True, the full tree representation of the DataManager is printed after the data was loaded. If
'condensed'
, the condensed tree will be printed.load_as_attr (bool, optional) – If True, the loaded entry will be added not as a new DataContainer or DataGroup, but as an attribute to an (already existing) object at
target_path
. The name of the attribute will be theentry_name
.parallel (Union[bool, dict]) –
If True, data is loaded in parallel. If a dict, can supply more options:
enabled
: whether to use parallel loadingprocesses
: how many processes to use; if None, will use as many as are available. For negative integers, will useos.cpu_count() + processes
processes.min_files
: if given, will fall back to non-parallel loading if fewer than the given number of files were matched byglob_str
min_size
: if given, specifies the minimum total size of all matched files (in bytes) below which to fall back to non-parallel loading
Note that a single file will never be loaded in parallel and there will never be more processes used than files that were selected to be loaded. Parallel loading incurs a constant overhead and is typically only speeding up data loading if the task is CPU-bound. Also, it requires the data tree to be fully serializable.
**load_params –
Further loading parameters, all optional. These are evaluated by
_load()
.- ignore (list):
The exact file names in this list will be ignored during loading. Paths are seen as elative to the data directory of the data manager.
- required (bool):
If True, will raise an error if no files were found. Default: False.
- path_regex (str):
This pattern can be used to match the path of the file that is being loaded. The match result is available to the format string under the
match
key.- exists_action (str):
The behaviour upon existing data. Can be:
raise
(default),skip
,skip_nowarn
,overwrite
,overwrite_nowarn
. With*_nowarn
values, no warning is given if an entry already existed. Note that this is ignored when theload_as_attr
argument is given.- unpack_data (bool, optional):
If True, and
load_as_attr
is active, not the DataContainer or DataGroup itself will be stored in the attribute, but the content of its.data
attribute.- progress_indicator (bool):
Whether to print a progress indicator or not. Default: True
- any further kwargs:
passed on to the loader function
- Returns
None
- Raises
ValueError – Upon invalid combination of
target_group
andtarget_path
arguments
-
_load
(*, target_path: str, loader: str, glob_str: Union[str, List[str]], load_as_attr: Optional[str], base_path: str = None, ignore: List[str] = None, required: bool = False, path_regex: str = None, exists_action: str = 'raise', unpack_data: bool = False, progress_indicator: bool = True, parallel: Union[bool, dict] = False, **loader_kwargs) → Tuple[int, int][source]¶ Helper function that loads a data entry to the specified path.
- Parameters
target_path (str) – The path to load the result of the loader to. This can be a format string; it is evaluated for each file. Available keys are: basename, match (if
path_regex
is given)loader (str) – The loader to use
glob_str (Union[str, List[str]]) – A glob string or a list of glob strings to match files in the data directory
load_as_attr (Union[str, None]) – If a string, the entry will be loaded into the object at
target_path
under a new attribute with this name.base_path (str, optional) – The base directory to concatenate the glob string to; if None, will use the DataManager’s data directory. With this option, it becomes possible to load data from a path outside the associated data directory.
ignore (List[str], optional) – The exact file names in this list will be ignored during loading. Paths are seen as relative to the data directory.
required (bool, optional) – If True, will raise an error if no files were found or if loading of a file failed.
path_regex (str, optional) – The regex applied to the relative path of the files that were found. It is used to generate the name of the target container. If not given, the basename is used.
exists_action (str, optional) – The behaviour upon existing data. Can be:
raise
(default),skip
,skip_nowarn
,overwrite
,overwrite_nowarn
. With*_nowarn
values, no warning is given if an entry already existed. Note that this is ignored ifload_as_attr
is given.unpack_data (bool, optional) – If True, and
load_as_attr
is active, not the DataContainer or DataGroup itself will be stored in the attribute, but the content of its.data
attribute.progress_indicator (bool, optional) – Whether to print a progress indicator or not
parallel (Union[bool, dict]) –
If True, data is loaded in parallel. If a dict, can supply more options:
enabled
: whether to use parallel loadingprocesses
: how many processes to use; if None, will use as many as are available. For negative integers, will useos.cpu_count() + processes
processes.min_files
: if given, will fall back to non-parallel loading if fewer than the given number of files were matched byglob_str
min_size
: if given, specifies the minimum total size of all matched files (in bytes) below which to fall back to non-parallel loading
Note that a single file will never be loaded in parallel and there will never be more processes used than files that were selected to be loaded. Parallel loading incurs a constant overhead and is typically only speeding up data loading if the task is CPU-bound. Also, it requires the data tree to be fully serializable.
**loader_kwargs – passed on to the loader function
- Raises
ValueError – Bad
path_regex
- Returns
- Tuple of number of files that matched the glob
strings, including those that may have been skipped, and number of successfully loaded and stored entries
- Return type
Tuple[int, int]
-
_load_file
(filepath: str, *, loader: str, load_func: Callable, target_path: str, path_sre: str, load_as_attr: str, TargetCls: type, required: bool, **loader_kwargs) → Tuple[Union[None, dantro.base.BaseDataContainer], List[str]][source]¶ Loads the data of a single file into a dantro object and returns the loaded object (or None) and the parsed target path key sequence.
-
_resolve_loader
(loader: str) → Tuple[Callable, type][source]¶ Resolves the loader function and returns a 2-tuple containing the load function and the declared dantro target type to load data to.
-
_create_files_list
(*, glob_str: Union[str, List[str]], ignore: List[str], base_path: str = None, required: bool = False, sort: bool = False) → List[str][source]¶ Create the list of file paths to load from.
Internally, this uses a set, thus ensuring that the paths are unique. The set is converted to a list before returning.
- Parameters
glob_str (Union[str, List[str]]) – The glob pattern or a list of glob patterns
ignore (List[str]) – The list of files to ignore
base_path (str, optional) – The base path for the glob pattern; use data directory, if not given.
required (bool, optional) – Will lead to an error being raised if no files could be matched
sort (bool, optional) – If true, sorts the list before returning
- Returns
the file paths to load
- Return type
list
- Raises
MissingDataError – If no files could be matched
RequiredDataMissingError – If no files could be matched but were required.
-
_prepare_target_path
(target_path: str, *, filepath: str, path_sre=None) → List[str][source]¶ Prepare the target path
-
_skip_path
(path: str, *, exists_action: str) → bool[source]¶ Check whether a given path exists and — depending on the
exists_action
– decides whether to skip this path or not.- Parameters
path (str) – The path to check for existence.
exists_action (str) – The behaviour upon existing data. Can be:
raise
,skip
,skip_nowarn
,overwrite
,overwrite_nowarn
. The*_nowarn
arguments suppress the warning.
- Returns
Whether to skip this path
- Return type
bool
- Raises
ExistingDataError – Raised when exists_action == ‘raise’
ValueError – Raised for invalid exists_action value
-
_store_object
(obj: Union[dantro.base.BaseDataGroup, dantro.base.BaseDataContainer], *, target_path: List[str], as_attr: Optional[str], unpack_data: bool, exists_action: str) → bool[source]¶ Store the given
obj
at the suppliedtarget_path
.Note that this will automatically overwrite, assuming that all checks have been made prior to the call to this function.
- Parameters
obj (Union[BaseDataGroup, BaseDataContainer]) – Object to store
target_path (List[str]) – The path to store the object at
as_attr (Union[str, None]) – If a string, store the object in the attributes of the container or group at target_path
unpack_data (bool) – Description
exists_action (str) – Description
- Returns
- Whether storing was successful. May be False in case the
target path already existed and
exists_action
specifies that it is to be skipped, or if the object was None.
- Return type
bool
- Raises
ExistingDataError – If non-group-like data already existed at that path
RequiredDataMissingError – If storing as attribute was selected but there was no object at the given target_path
-
_contains_group
(path: Union[str, List[str]], *, base_group: dantro.base.BaseDataGroup = None) → bool[source]¶ Recursively checks if the given path is available _and_ a group.
- Parameters
path (Union[str, List[str]]) – The path to check.
base_group (BaseDataGroup) – The group to start from. If not given, will use self.
- Returns
Whether the path points to a group
- Return type
bool
-
_create_groups
(path: Union[str, List[str]], *, base_group: dantro.base.BaseDataGroup = None, GroupCls: Union[type, str] = None, exist_ok: bool = True)[source]¶ Recursively create groups for the given path. Unlike new_group, this also creates the groups at the intermediate paths.
- Parameters
path (Union[str, List[str]]) – The path to create groups along
base_group (BaseDataGroup, optional) – The group to start from. If not given, uses self.
GroupCls (Union[type, str], optional) – The class to use for creating the groups or None if the _DATA_GROUP_DEFAULT_CLS is to be used. If a string is given, lookup happens from the _DATA_GROUPS_CLASSES variable.
exist_ok (bool, optional) – Whether it is ok that groups along the path already exist. These might also be of different type. Default: True
- Raises
ExistingDataError – If not exist_ok
ExistingGroupError – If not exist_ok and a group already exists
-
_determine_group_class
(Cls: Union[type, str]) → type[source]¶ Helper function to determine the type of a group from an argument.
- Parameters
Cls (Union[type, str]) – If None, uses the _DATA_GROUP_DEFAULT_CLS. If a string, tries to extract it from the _DATA_GROUP_CLASSES class variable. Otherwise, assumes this is already a type.
- Returns
The group class to use
- Return type
type
- Raises
KeyError – If the string class name was not registered
ValueError – If no _DATA_GROUP_CLASSES variable was populated
-
_parse_file_path
(path: str, *, default_ext=None) → str[source]¶ Parses a file path: if it is a relative path, makes it relative to the associated data directory. If a default extension is specified and the path does not contain one, that extension is added.
This helper method is used as part of dumping and storing the data tree, i.e. in the
dump()
andrestore()
methods.
-
dump
(*, path: str = None, **dump_kwargs) → str[source]¶ Dumps the data tree to a new file at the given path, creating any necessary intermediate data directories.
For restoring, use
restore()
.- Parameters
path (str, optional) – The path to store this file at. If this is not given, use the default tree cache path that was set up during initialization. If it is given and a relative path, it is assumed relative to the data directory. If the path does not end with an extension, the
.d3
(read: “data tree”) extension is automatically added.**dump_kwargs – Passed on to
pkl.dump
- Returns
The path that was used for dumping the tree file
- Return type
str
-
restore
(*, from_path: str = None, merge: bool = False, **load_kwargs)[source]¶ Restores the data tree from a dump.
For dumping, use
dump()
.- Parameters
from_path (str, optional) – The path to restore this DataManager from. If it is not given, uses the default tree cache path that was set up at initialization. If it is a relative path, it is assumed relative to the data directory. Take care to add the corresponding file extension.
merge (bool, optional) – If True, uses a recursive update to merge the current tree with the restored tree. If False, uses
clear()
to clear the current tree and then re-populates it with the restored tree.**load_kwargs – Passed on to
pkl.load
- Raises
FileNotFoundError – If no file is found at the (expanded) path.
-
_ALLOWED_CONT_TYPES
= None¶
-
_ATTRS_CLS
¶ alias of
dantro.base.BaseDataAttrs
-
_COND_TREE_CONDENSE_THRESH
= 10¶
-
_COND_TREE_MAX_LEVEL
= 10¶
-
_DirectInsertionModeMixin__in_direct_insertion_mode
= False¶
-
_LockDataMixin__locked
= False¶
-
_MutableMapping__marker
= <object object>¶
-
_NEW_CONTAINER_CLS
= None¶
-
_NEW_GROUP_CLS
= None¶
-
_STORAGE_CLS
¶ alias of
collections.OrderedDict
-
__contains__
(cont: Union[str, dantro.abc.AbstractDataContainer]) → bool¶ Whether the given container is in this group or not.
If this is a data tree object, it will be checked whether this specific instance is part of the group, using
is
-comparison.Otherwise, assumes that
cont
is a valid argument to the__getitem__()
method (a key or key sequence) and tries to access the item at that path, returningTrue
if this succeeds andFalse
if not.Lookup complexity is that of item lookup (scalar) for both name and object lookup.
- Parameters
cont (Union[str, AbstractDataContainer]) – The name of the container, a path, or an object to check via identity comparison.
- Returns
- Whether the given container object is part of this group or
whether the given path is accessible from this group.
- Return type
bool
-
__delitem__
(key: str) → None¶ Deletes an item from the group
-
__eq__
(other) → bool¶ Evaluates equality by making the following comparisons: identity, strict type equality, and finally: equality of the
_data
and_attrs
attributes, i.e. the private attribute. This ensures that comparison does not trigger any downstream effects like resolution of proxies.If types do not match exactly,
NotImplemented
is returned, thus referring the comparison to the other side of the==
.
-
__format__
(spec_str: str) → str¶ Creates a formatted string from the given specification.
Invokes further methods which are prefixed by
_format_
.
-
__getitem__
(key: Union[str, List[str]]) → dantro.abc.AbstractDataContainer¶ Looks up the given key and returns the corresponding item.
This supports recursive relative lookups in two ways:
By supplying a path as a string that includes the path separator. For example,
foo/bar/spam
walks down the tree along the given path segments.By directly supplying a key sequence, i.e. a list or tuple of key strings.
With the last path segment, it is possible to access an element that is no longer part of the data tree; successive lookups thus need to use the interface of the corresponding leaf object of the data tree.
Absolute lookups, i.e. from path
/foo/bar
, are not possible!Lookup complexity is that of the underlying data structure: for groups based on dict-like storage containers, lookups happen in constant time.
Note
This method aims to replicate the behavior of POSIX paths.
Thus, it can also be used to access the element itself or the parent element: Use
.
to refer to this object and..
to access this object’sparent
.- Parameters
key (Union[str, List[str]]) – The name of the object to retrieve or a path via which it can be found in the data tree.
- Returns
- The object at
key
, which concurs to the dantro tree interface.
- The object at
- Return type
- Raises
ItemAccessError – If no object could be found at the given
key
or if an absolute lookup, starting with/
, was attempted.
-
__iter__
()¶ Returns an iterator over the OrderedDict
-
__len__
() → int¶ The number of members in this group.
-
__repr__
() → str¶ Same as __str__
-
__setitem__
(key: Union[str, List[str]], val: dantro.base.BaseDataContainer) → None¶ This method is used to allow access to the content of containers of this group. For adding an element to this group, use the add method!
- Parameters
key (Union[str, List[str]]) – The key to which to set the value. If this is a path, will recurse down to the lowest level. Note that all intermediate keys need to be present.
val (BaseDataContainer) – The value to set
- Returns
None
- Raises
ValueError – If trying to add an element to this group, which should be done via the add method.
-
__sizeof__
() → int¶ Returns the size of the data (in bytes) stored in this container’s data and its attributes.
Note that this value is approximate. It is computed by calling the
sys.getsizeof
function on the data, the attributes, the name and some caching attributes that each dantro data tree class contains. Importantly, this is not a recursive algorithm.Also, derived classes might implement further attributes that are not taken into account either. To be more precise in a subclass, create a specific __sizeof__ method and invoke this parent method additionally.
For more information, see the documentation of
sys.getsizeof
:
-
__str__
() → str¶ An info string, that describes the object. This invokes the formatting helpers to show the log string (type and name) as well as the info string of this object.
-
_abc_impl
= <_abc_data object>¶
-
_add_container
(cont, *, overwrite: bool)¶ Private helper method to add a container to this group.
-
_add_container_callback
(cont) → None¶ Called after a container was added.
-
_add_container_to_data
(cont: dantro.abc.AbstractDataContainer) → None¶ Performs the operation of adding the container to the _data. This can be used by subclasses to make more elaborate things while adding data, e.g. specify ordering …
- NOTE This method should NEVER be called on its own, but only via the
_add_container method, which takes care of properly linking the container that is to be added.
NOTE After adding, the container need be reachable under its .name!
- Parameters
cont – The container to add
-
_attrs
= None¶
-
_check_cont
(cont) → None¶ Can be used by a subclass to check a container before adding it to this group. Is called by _add_container before checking whether the object exists or not.
This is not expected to return, but can raise errors, if something did not work out as expected.
- Parameters
cont – The container to check
-
_check_data
(data: Any) → None¶ This method can be used to check the data provided to this container
It is called before the data is stored in the
__init__
method and should raise an exception or create a warning if the data is not as desired.This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using
super()
.Note
The
CheckDataMixin
provides a generalised implementation of this method to perform some type checks and react to unexpected types.- Parameters
data (Any) – The data to check
-
_check_name
(new_name: str) → None¶ Called from name.setter and can be used to check the name that the container is supposed to have. On invalid name, this should raise.
This method can be subclassed to implement more specific behaviour. To propagate the parent classes’ behaviour the subclassed method should always call its parent method using super().
- Parameters
new_name (str) – The new name, which is to be checked.
-
_direct_insertion_mode
(*, enabled: bool = True)¶ A context manager that brings the class this mixin is used in into direct insertion mode. While in that mode, the
with_direct_insertion()
property will return true.This context manager additionally invokes two callback functions, which can be specialized to perform certain operations when entering or exiting direct insertion mode: Before entering,
_enter_direct_insertion_mode()
is called. After exiting,_exit_direct_insertion_mode()
is called.- Parameters
enabled (bool, optional) – whether to actually use direct insertion mode. If False, will yield directly without setting the toggle. This is equivalent to a null-context.
-
_enter_direct_insertion_mode
()¶ Called after entering direct insertion mode; can be overwritten to attach additional behaviour.
-
_exit_direct_insertion_mode
()¶ Called before exiting direct insertion mode; can be overwritten to attach additional behaviour.
-
_format_cls_name
() → str¶ A __format__ helper function: returns the class name
-
_format_info
() → str¶ A __format__ helper function: returns an info string that is used to characterize this object. Does NOT include name and classname!
-
_format_logstr
() → str¶ A __format__ helper function: returns the log string, a combination of class name and name
-
_format_name
() → str¶ A __format__ helper function: returns the name
-
_format_path
() → str¶ A __format__ helper function: returns the path to this container
-
_format_tree
() → str¶ Returns the default tree representation of this group by invoking the .tree property
-
_format_tree_condensed
() → str¶ Returns the default tree representation of this group by invoking the .tree property
-
_ipython_key_completions_
() → List[str]¶ For ipython integration, return a list of available keys
-
_link_child
(*, new_child: dantro.base.BaseDataContainer, old_child: dantro.base.BaseDataContainer = None)¶ Links the new_child to this class, unlinking the old one.
This method should be called from any method that changes which items are associated with this group.
-
_lock_hook
()¶ Invoked upon locking.
-
_tree_repr
(*, level: int = 0, max_level: int = None, info_fstr='<{:cls_name,info}>', info_ratio: float = 0.6, condense_thresh: Union[int, Callable[[int, int], int]] = None, total_item_count: int = 0) → Union[str, List[str]]¶ Recursively creates a multi-line string tree representation of this group. This is used by, e.g., the _format_tree method.
- Parameters
level (int, optional) – The depth within the tree
max_level (int, optional) – The maximum depth within the tree; recursion is not continued beyond this level.
info_fstr (str, optional) – The format string for the info string
info_ratio (float, optional) – The width ratio of the whole line width that the info string takes
condense_thresh (Union[int, Callable[[int, int], int]], optional) – If given, this specifies the threshold beyond which the tree view for the current element becomes condensed by hiding the output for some elements. The minimum value for this is 3, indicating that there should be at most 3 lines be generated from this level (excluding the lines coming from recursion), i.e.: two elements and one line for indicating how many values are hidden. If a smaller value is given, this is silently brought up to 3. Half of the elements are taken from the beginning of the item iteration, the other half from the end. If given as integer, that number is used. If a callable is given, the callable will be invoked with the current level, number of elements to be added at this level, and the current total item count along this recursion branch. The callable should then return the number of lines to be shown for the current element.
total_item_count (int, optional) – The total number of items already created in this recursive tree representation call. Passed on between recursive calls.
- Returns
- The (multi-line) tree representation of
this group. If this method was invoked with
level == 0
, a string will be returned; otherwise, a list of strings will be returned.
- Return type
Union[str, List[str]]
-
_unlink_child
(child: dantro.base.BaseDataContainer)¶ Unlink a child from this class.
This method should be called from any method that removes an item from this group, be it through deletion or through
-
_unlock_hook
()¶ Invoked upon unlocking.
-
add
(*conts, overwrite: bool = False)¶ Add the given containers to this group.
-
property
attrs
¶ The container attributes.
-
property
classname
¶ Returns the name of this DataContainer-derived class
-
clear
()¶ Clears all containers from this group.
This is done by unlinking all children and then overwriting
_data
with an empty_STORAGE_CLS
object.
-
property
data
¶ The stored data.
-
get
(key, default=None)¶ Return the container at key, or default if container with name key is not available.
-
items
()¶ Returns an iterator over the (name, data container) tuple of this group.
-
keys
()¶ Returns an iterator over the container names in this group.
-
lock
()¶ Locks the data of this object
-
property
locked
¶ Whether this object is locked
-
property
logstr
¶ Returns the classname and name of this object
-
property
name
¶ The name of this DataContainer-derived object.
-
new_container
(path: Union[str, List[str]], *, Cls: type = None, **kwargs)¶ Creates a new container of class
Cls
and adds it at the given path relative to this group.If needed, intermediate groups are automatically created.
- Parameters
path (Union[str, List[str]]) – Where to add the container.
Cls (type, optional) – The class of the container to add. If None, the
_NEW_CONTAINER_CLS
class variable’s value is used.**kwargs – passed on to
Cls.__init__
- Returns
the created container
- Return type
Cls
- Raises
ValueError – If neither the
Cls
argument nor the class variable_NEW_CONTAINER_CLS
were set or ifpath
was empty.TypeError – When
Cls
is not compatible to the data tree
-
new_group
(path: str, *, Cls: Union[type, str] = None, **kwargs)[source]¶ Creates a new group at the given path.
This is a slightly advanced version of the new_group method of the BaseDataGroup. It not only adjusts the default type, but also allows more ways how to specify the type of the group to create.
- Parameters
path (str) – Where to create the group. Note that the intermediates of this path need to already exist.
Cls (Union[type, str], optional) – If given, use this type to create the group. If a string is given, resolves the type from the _DATA_GROUP_CLASSES class variable. If None, uses the default data group type of the data manager.
**kwargs – Passed on to Cls.__init__
- Returns
the created group
- Return type
Cls
-
property
parent
¶ The associated parent of this container or group
-
property
path
¶ The path to get to this container or group from some root path
-
pop
(k[, d]) → v, remove specified key and return the corresponding value.¶ If key is not found, d is returned if given, otherwise KeyError is raised.
-
popitem
() → (k, v), remove and return some (key, value) pair¶ as a 2-tuple; but raise KeyError if D is empty.
-
raise_if_locked
(*, prefix: str = None)¶ Raises an exception if this object is locked; does nothing otherwise
-
recursive_update
(other, *, overwrite: bool = True)¶ Recursively updates the contents of this data group with the entries of the given data group
Note
This will create shallow copies of those elements in
other
that are added to this object.- Parameters
other (BaseDataGroup) – The group to update with
overwrite (bool, optional) – Whether to overwrite already existing object. If False, a conflict will lead to an error being raised and the update being stopped.
- Raises
TypeError – If
other
was of invalid type
-
setdefault
(key, default=None)¶ This method is not supported for a data group
-
property
tree
¶ Returns the default (full) tree representation of this group
-
property
tree_condensed
¶ Returns the condensed tree representation of this group. Uses the
_COND_TREE_*
prefixed class attributes as parameters.
-
unlock
()¶ Unlocks the data of this object
-
update
([E, ]**F) → None. Update D from mapping/iterable E and F.¶ If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
-
values
()¶ Returns an iterator over the containers in this group.
-
property
with_direct_insertion
¶ Whether the class this mixin is mixed into is currently in direct insertion mode.
-