Specializing dantro Classes
This page shows a few examples of how to specialize
dantro classes to your liking.
This step is an important aspect of adapting dantro to work with the data structures that you are frequently using, which is beneficial for good integration into your workflow.
The code snippets shown on this page are implemented as test cases to assert that they function as intended.
To have a look at the full source code used in the examples below, you can
download the relevant file or view it online.
Note that the integration into the test framework requires some additional code in those files, e.g. to generate dummy data.
As an example, let’s look at the implementation of the
MutableSequenceContainer, a container that is meant to store mutable sequences:
# Import the python abstract base class we want to adhere to from collections.abc import MutableSequence # Import base container class and the mixins we would like to use from dantro.base import BaseDataContainer from dantro.mixins import CheckDataMixin, CollectionMixin, ItemAccessMixin class MutableSequenceContainer(CheckDataMixin, ItemAccessMixin, CollectionMixin, BaseDataContainer, MutableSequence): """The MutableSequenceContainer stores sequence-like mutable data"""
The steps to arrive at this point are as follows:
collections.abc python module is also used by python to specify the interfaces for python-internal classes.
In the documentation it says that the
MutableSequence inherits from
Sequence and has the following abstract methods:
As we want the resulting container to adhere to this interface, we set
MutableSequence as the first class to inherit from.
BaseDataContainer is what makes this object a dantro data container.
It implements some of the required methods to concur with the
MutableSequence interface but leaves others abstract.
Now, we need to supply implementations of these abstract methods.
That is the job of the following two (reading from right to left) mixin classes.
In this case, the
Sequence interface has to be fulfilled.
Sequence is nothing more than a
Collection with item access, we can fulfill this by inheriting from the
CollectionMixin and the
CheckDataMixin is an example of how functionality can be added to the container while still adhering to the interface.
This mixin checks the provided data before storing it and allows specifying whether unexpected data should lead to warnings or exceptions; for an example, see below
def insert(self, idx: int, val) -> None: """Insert an item at a given position. Args: idx (int): The index before which to insert val: The value to insert """ self.data.insert(idx, val)
Once defined, instantiation of a custom container works the same way as for other data containers:
dc = MutableSequenceContainer(name="my_mutable_sequence", data=[4, 8, 16]) # Insert values dc.insert(0, 2) dc.insert(0, 1) # Item access and collection interface assert 16 in dc assert 32 not in dc assert dc == 1 for num in dc: print(num, end=", ") # prints: 1, 2, 4, 8, 16,
Many mixins allow some form of configuration. This typically happens via class variables.
Let’s define a new container that strictly requires its stored data to be a
list, i.e. an often-used mutable sequence type.
We can use the already-included
CheckDataMixin such that it checks a type.
To do so, we set the
DATA_EXPECTED_TYPES to only allow
list and we set
DATA_UNEXPECTED_ACTION to raise an exception if this is not the case.
class StrictlyListContainer(MutableSequenceContainer): """A MutableSequenceContainer that allows only a list as data""" DATA_EXPECTED_TYPES = (list,) # as tuple or None (allow all) DATA_UNEXPECTED_ACTION = 'raise' # can be: raise, warn, ignore # This will work some_list = StrictlyListContainer(name="some_list", data=["foo", "bar"]) # The following will fail with pytest.raises(TypeError): StrictlyListContainer(name="some_tuple", data=("foo", "bar")) with pytest.raises(TypeError): StrictlyListContainer(name="some_tuple", data="just some string")
Other mixins provide other class variables for specializing behavior. Consult the documentation or the source code to find out which ones.
The class variables typically define the default behavior for a certain specialized type.
However, depending on the mixin, its behavior might also depend on runtime information, e.g. specified in
We advise against overwriting class variables during the lifetime of an object.
import dantro from dantro.data_loaders import PickleLoaderMixin, YamlLoaderMixin class MyDataManager(PickleLoaderMixin, YamlLoaderMixin, dantro.DataManager): """A DataManager specialization that can load pickle and yaml data"""
For more information, see The DataManager.
When using specialized container classes such a custom
DataManager is also the place to configure data loaders to use those classes.
For example, when using the
_HDF5-prefixed class variables can be set to use the specialized container classes rather than the defaults.
For an integration example, you can have a look at the data manager used in utopya.
Adding a custom data loader is simple.
As an example, let’s look at how a data loader mixin for plain text files (
TextLoaderMixin) is implemented in dantro:
"""Defines a loader mixin to load plain text files""" from ..containers import StringContainer from ._tools import add_loader class TextLoaderMixin: """A mixin for :py:class:`~dantro.data_mngr.DataManager` that supports loading of plain text files.""" @add_loader(TargetCls=StringContainer) def _load_plain_text( filepath: str, *, TargetCls: type, **load_kwargs ) -> StringContainer: """Loads the content of a plain text file into a :py:class:`~dantro.containers.general.StringContainer`. Args: filepath (str): Where the plain text file is located TargetCls (type): The class constructor **load_kwargs: Passed on to :py:func:`open` Returns: StringContainer: The reconstructed StringContainer """ with open(filepath, **load_kwargs) as f: data = f.read() return TargetCls(data=data, attrs=dict(filepath=filepath)) # Also make the loader available under the ``text`` label _load_text = _load_plain_text
Define your mixin class
Add a method named
_load_<name>and decorate it with
Fill in the method’s body to implement the loading of your data.
Initialize and return the
TargetClsobject, passing the loaded
The plot manager can be specialized to support further functionality simply by overloading methods that may or may not invoke the parent methods. However, given the complexity of the plot manager, there is no guide on how to do this exactly: It depends a lot on what you want to achieve.
In a simple situation, a specialized
PlotManager may simply overwrite some default values via the class variables.
This could, for instance, be the plot function resolver, which defaults to
import dantro class MyPlotFuncResolver(dantro.plot.utils.PlotFuncResolver): """A custom plot function resolver class""" BASE_PKG = "my_custom_package.plot_functions" """For relative module imports, regard this as the base package. A plot configuration ``module`` argument starting with a ``.`` is looked up in that module. Note that this needs to be an importable module. """ class MyPyPlotManager(dantro.PlotManager): """My custom plot manager""" PLOT_FUNC_RESOLVER = MyPlotFuncResolver """Use a custom plot function resolver"""
As described in Plot Creators, dantro already supplies a range of plot creators.
Furthermore, dantro provides the
BasePlotCreator, which provides an interface and a lot of the commonly used functionality.
Specialization thus can be of two kinds:
Using an existing plot creator and configuring it to your needs.
Implementing a whole new plot creator, e.g. because you desire to use a different plotting backend.
In general, we recommend to refer to the implementation of existing
dantro.plot.creators as examples for how this can be achieved.
We are happy to support the implementation of new plot creators, so feel free to post an issue to the project page.