dantro._dag_utils module

Private low-level helper classes and functions for the DAG framework

NOTE This is imported by dantro.tools to register classes with YAML.

class dantro._dag_utils.Placeholder(data: Any)[source]

Bases: object

A generic placeholder class for use in the DAG framework.

Objects of this class or derived classes are yaml-representable and thus hashable after a parent object created a YAML representation. In addition, the __hash__ method can be used to generate a hash from the string representation.

__init__(data: Any)[source]

Initialize a Placeholder by storing its payload

__eq__(other) → bool[source]

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

yaml_tag = '!dag_placeholder'
classmethod from_yaml(constructor, node)[source]

Construct a Placeholder from a scalar YAML node

classmethod to_yaml(representer, node)[source]

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.ResultPlaceholder(data: Any)[source]

Bases: dantro._dag_utils.Placeholder

A placeholder class for a data transformation result

yaml_tag = '!dag_result'
property result_name

The name of the transformation result this is a placeholder for

__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

__init__(data: Any)

Initialize a Placeholder by storing its payload

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

dantro._dag_utils.resolve_placeholders(d: dict, *, dag: TransformationDAG, Cls: type = <class 'dantro._dag_utils.ResultPlaceholder'>, **compute_kwargs) → dict[source]

Recursively replaces placeholder objects throughout the given dict.

Computes TransformationDAG results and replaces the placeholder objects with entries from the results dict, thereby making it possible to compute configuration values using results of the data transformation framework <dag_framework>, for example as done in the plotting framework; see Using data transformation results in the plot configuration.

Warning

While this function has a return value, it resolves the placeholders in-place, such that the given d will be mutated even if the return value is ignored on the calling site.

Parameters
  • d (dict) – The object to replace placeholders in. Will recursively walk through all dict- and list-like objects to find placeholders.

  • dag (TransformationDAG) – The data transformation tree to resolve the placeholders’ results from.

  • Cls (type, optional) – The expected type of the placeholders.

  • **compute_kwargs – Passed on to compute().

class dantro._dag_utils.PositionalArgument(pos: int)[source]

Bases: dantro._dag_utils.Placeholder

A PositionalArgument is a placeholder that holds as payload a positional argument’s position. This is used, e.g., for meta-operation specification.

yaml_tag = '!arg'
__init__(pos: int)[source]

Initialize from an integer, also accepting int-convertibles

property position
__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.KeywordArgument(name: str)[source]

Bases: dantro._dag_utils.Placeholder

A KeywordArgument is a placeholder that holds as payload the name of a keyword argument. This is used, e.g., for meta-operation specification.

yaml_tag = '!kwarg'
__init__(name: str)[source]

Initialize by storing the keyword argument name

property name
__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.DAGReference(ref: str)[source]

Bases: dantro._dag_utils.Placeholder

The DAGReference class is the base class of all DAG reference objects. It extends the generic Placeholder class with the ability to resolve references within a TransformationDAG.

yaml_tag = '!dag_ref'
__init__(ref: str)[source]

Initialize a DAGReference object from a hash.

property ref

The associated reference of this object

_resolve_ref(*, dag: TransformationDAG) → str[source]

Return the hash reference; for the base class, the data is already the hash reference, so no DAG is needed. Derived classes _might_ need the DAG to resolve their reference hash.

convert_to_ref(*, dag: TransformationDAG) → DAGReference[source]

Create a new object that is a hash ref to the same object this tag refers to.

resolve_object(*, dag: TransformationDAG) → Any[source]

Resolve the object by looking up the reference in the DAG’s object database.

__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.DAGTag(name: str)[source]

Bases: dantro._dag_utils.DAGReference

A DAGTag object stores a name of a tag, which serves as a named reference to some object in the DAG.

yaml_tag = '!dag_tag'
__init__(name: str)[source]

Initialize a DAGTag object, storing the specified field name

property name

The name of the tag within the DAG that this object references

_resolve_ref(*, dag: TransformationDAG) → str[source]

Return the hash reference by looking up the tag in the DAG

__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

convert_to_ref(*, dag: TransformationDAG) → DAGReference

Create a new object that is a hash ref to the same object this tag refers to.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

property ref

The associated reference of this object

resolve_object(*, dag: TransformationDAG) → Any

Resolve the object by looking up the reference in the DAG’s object database.

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.DAGMetaOperationTag(name: str)[source]

Bases: dantro._dag_utils.DAGTag

A DAGMetaOperationTag stores a name of a tag, just as DAGTag, but can only be used inside a meta-operation. When resolving this tag’s reference, the target is looked up from the stack of the TransformationDAG.

yaml_tag = '!mop_tag'
SPLIT_STR = '::'
__init__(name: str)[source]

Initialize the DAGMetaOperationTag object.

The name needs to be of the <meta-operation name>::<tag name> pattern and thereby include information on the name of the meta-operation this tag is used in.

_resolve_ref(*, dag: TransformationDAG) → str[source]

Return the hash reference by looking it up in the reference stacks of the specified TransformationDAG. The last entry always refers to the currently active meta-operation.

classmethod make_name(meta_operation: str, *, tag: str) → str[source]

Given a meta-operation name and a tag name, generates the name of this meta-operation tag.

classmethod from_names(meta_operation: str, *, tag: str) → dantro._dag_utils.DAGMetaOperationTag[source]

Generates a DAGMetaOperationTag using the names of a meta-operation and the name of a tag.

__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

convert_to_ref(*, dag: TransformationDAG) → DAGReference

Create a new object that is a hash ref to the same object this tag refers to.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

property name

The name of the tag within the DAG that this object references

property ref

The associated reference of this object

resolve_object(*, dag: TransformationDAG) → Any

Resolve the object by looking up the reference in the DAG’s object database.

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.DAGNode(idx: int)[source]

Bases: dantro._dag_utils.DAGReference

A DAGNode is a reference by the index within the DAG’s node list.

yaml_tag = '!dag_node'
__init__(idx: int)[source]

Initialize a DAGNode object with a node index.

Parameters

idx (int) – The idx value to set this reference to. Can also be a negative value, in which case the node list is traversed from the back.

Raises

TypeError – On invalid type (not int-convertible)

property idx

The idx to the referenced node within the DAG’s node list

_resolve_ref(*, dag: TransformationDAG) → str[source]

Return the hash reference by looking up the node index in the DAG

__eq__(other) → bool

Only objects with exactly the same type and data are regarded as equal; specifically, this makes instances of subclasses always unequal to instances of this base class.

convert_to_ref(*, dag: TransformationDAG) → DAGReference

Create a new object that is a hash ref to the same object this tag refers to.

classmethod from_yaml(constructor, node)

Construct a Placeholder from a scalar YAML node

property ref

The associated reference of this object

resolve_object(*, dag: TransformationDAG) → Any

Resolve the object by looking up the reference in the DAG’s object database.

classmethod to_yaml(representer, node)

Create a YAML representation of a Placeholder, carrying only the _data attribute over…

As YAML expects scalar data to be str-like, a type cast is done. The subclasses that rely on certain argument types should take care that their __init__ method can parse arguments that are str-like.

class dantro._dag_utils.DAGObjects[source]

Bases: object

An objects database for the DAG framework.

It uses a flat dict containing (hash, object ref) pairs. The interface is slightly restricted compared to a regular dict; especially, item deletion is not made available.

Objects are added to the database via the add_object method. They need to have a hashstr property, which returns a hash string deterministically representing the object; note that this is not equivalent to the Python builtin hash() function which invokes the __hash__ magic method.

__init__()[source]

Initialize an empty objects database

__str__() → str[source]

A human-readable string representation of the object database

add_object(obj, *, custom_hash: str = None) → str[source]

Add an object to the object database, storing it under its hash.

Note that the object cannot be just any object that is hashable but it needs to return a string-based hash via the hashstr property. This is a dantro DAG framework-internal interface.

Also note that the object will NOT be added if an object with the same hash is already present. The object itself is of no importance, only the returned hash is.

Parameters
  • obj – Some object that has the hashstr property, i.e. is hashable as required by the DAG interface

  • custom_hash (str, optional) – A custom hash to use instead of the hash extracted from obj. Can only be given when obj does not have a hashstr property.

Returns

The hash string of the given object. If a custom hash string

was given, it is also the return value

Return type

str

Raises
  • TypeError – When attempting to pass custom_hash while obj has a hashstr property

  • ValueError – If the given custom_hash already exists.

__getitem__(key: str) → object[source]

Return the object associated with the given hash

__len__() → int[source]

Returns the number of objects in the objects database

__contains__(key: str) → bool[source]

Whether the given hash refers to an object in this database

keys()[source]
values()[source]
items()[source]
dantro._dag_utils.parse_dag_minimal_syntax(params: Union[str, dict]) → dict[source]

Parses the minimal syntax parameters, effectively translating a string- like argument to a dict with the string specified as the operation key.

dantro._dag_utils.parse_dag_syntax(*, operation: str = None, args: list = None, kwargs: dict = None, tag: str = None, with_previous_result: bool = False, salt: int = None, file_cache: dict = None, ignore_hooks: bool = False, allow_failure: Union[bool, str] = None, fallback: Any = None, **ops) → dict[source]

Given the parameters of a transform operation, possibly in a shorthand notation, returns a dict with normalized content by expanding the shorthand notation.

Keys that will always be available in the resulting dict:

operation, args, kwargs, tag.

Parameters
  • operation (str, optional) – Which operation to carry out; can only be specified if there is no ops argument.

  • args (list, optional) – Positional arguments for the operation; can only be specified if there is no ops argument.

  • kwargs (dict, optional) – Keyword arguments for the operation; can only be specified if there is no ops argument.

  • tag (str, optional) – The tag to attach to this transformation

  • with_previous_result (bool, optional) – Whether the result of the previous transformation is to be used as first positional argument of this transformation.

  • salt (int, optional) – A salt to the Transformation object, thereby changing its hash.

  • file_cache (dict, optional) – File cache parameters

  • ignore_hooks (bool, optional) – If True, there will be no lookup in the operation hooks.

  • allow_failure (Union[bool, str], optional) – Whether this Transformation allows failure during computation.

  • fallback (Any, optional) – The fallback value to use in case of failure

  • **ops – The operation that is to be carried out. May contain one and only one operation.

Returns

The normalized dict of transform parameters.

Return type

dict

Raises

ValueError – For invalid notation, e.g. unambiguous specification of arguments or the operation.

dantro._dag_utils.op_hook_expression(operation, *args, **kwargs) → Tuple[str, list, dict][source]

An operation hook for the expression operation, attempting to auto-detect which symbols are specified in the given expression. From those, DAGTag objects are created, making it more convenient to specify an expression that is based on other DAG tags.

The detected symbols are added to the kwargs.symbols, if no symbol of the same name is already explicitly defined there.

This hook accepts as positional arguments both the (expr,) form and the (prev_node, expr) form, making it more robust when the with_previous_result flag was set.

If the expression contains the prev or previous_result symbols, the corresponding DAGNode will be added to the symbols additionally.

For more information on operation hooks, see DAG Syntax Operation Hooks.