Skip to content

API Reference

loman

Loman: A Python library for building computation graphs.

Loman provides tools for creating and managing dependency-aware computation graphs where nodes represent data or calculations, and edges represent dependencies.

Computation

A computation graph that manages dependencies and calculations.

The Computation class provides a framework for building and executing computation graphs where nodes represent data or calculations, and edges represent dependencies between them.

revision: int property

Return the revision number of the most recently published change.

__init__(*, default_executor: Executor | None = None, executor_map: dict[str, Executor] | None = None, metadata: dict[str, Any] | None = None) -> None

Initialize a new Computation.

:param default_executor: An executor :type default_executor: concurrent.futures.Executor, default ThreadPoolExecutor(max_workers=1)

subscribe(callback: ComputationSubscriber) -> Callable[[], None]

Subscribe to batched computation changes.

Subscribers are notified in registration order, synchronously, on the thread that completes the outermost public mutation. A subscriber that raises is logged and skipped; it never interrupts the mutation or the other subscribers. A subscriber that itself mutates the computation causes a further event to be published once the current round finishes, up to a bounded number of cascades.

A callback with an object behind it --- anything carrying a __self__, whether written in Python or in C --- is held weakly, so subscribing obj.handler or events.append does not keep the owner alive; callers must retain it themselves. Everything else --- plain functions, lambdas, callable objects, :func:functools.partial --- is held strongly until unsubscribed, because callers commonly pass a throwaway closure that nothing else references.

The exception is an owner that supports no weak references at all, such as :class:list, :class:dict and :class:bytearray. There some_list.append falls back to a strong reference, which is a limitation of the type rather than a choice, and errs towards a subscription that keeps delivering over one that silently stops.

Subscriptions are not copied by :meth:copy and are not serialized.

:param callback: Function accepting a :class:ComputationEvent. :return: An idempotent, no-argument unsubscribe function.

get_attribute_view_for_path(nodekey: NodeKey, get_one_func: Callable[[Name], Any], get_many_func: Callable[[Name | Names], Any]) -> AttributeView

Create an attribute view for a specific node path.

add_node(name: Name, func: Callable[..., Any] | None = None, *, args: list[Any] | None = None, kwds: dict[str, Any] | None = None, value: Any = _MISSING_VALUE_SENTINEL, converter: Callable[[Any], Any] | None = None, serialize: bool = True, inspect: bool = True, group: str | None = None, tags: Iterable[str] | None = None, style: str | None = None, executor: str | None = None, store: str | None = None, metadata: dict[str, Any] | None = None) -> None

Adds or updates a node in a computation.

:param name: Name of the node to add. This may be any hashable object. :param func: Function to use to calculate the node if the node is a calculation node. By default, the input nodes to the function will be implied from the names of the function parameters. For example, a parameter called a would be taken from the node called a. This can be modified with the kwds parameter. :type func: Function, default None :param args: Specifies a list of nodes that will be used to populate arguments of the function positionally for a calculation node. e.g. If args is ['a', 'b', 'c'] then the function would be called with three parameters, taken from the nodes 'a', 'b' and 'c' respectively. :type args: List, default None :param kwds: Specifies a mapping from parameter name to the node that should be used to populate that parameter when calling the function for a calculation node. e.g. If args is {'x': 'a', 'y': 'b'} then the function would be called with parameters named 'x' and 'y', and their values would be taken from nodes 'a' and 'b' respectively. Each entry in the dictionary can be read as "take parameter [key] from node [value]". :type kwds: Dictionary, default None :param value: If given, the value is inserted into the node, and the node state set to UPTODATE. :type value: default None :param converter: Callable applied to any value on its way into the node. The node stores what the converter returns, both for values supplied by value, insert and insert_many, and for values the node calculates with func. A converter that raises leaves the node in state ERROR without storing the value, which is how a validator is written: check the value and return it unchanged when it is acceptable. The exception propagates to the caller when the value was supplied, but not when it was calculated, where the failure is reported as node state instead. A converter is saved by reference, like a node's function, so it must be importable: a module-level function or builtin round-trips, while a lambda raises SerializationError. :type converter: Callable, default None :param serialize: Whether the node should be serialized. Some objects cannot be serialized, in which case, set serialize to False :type serialize: boolean, default True :param inspect: Whether to use introspection to determine the arguments of the function, which can be slow. If this is not set, kwds and args must be set for the function to obtain parameters. :type inspect: boolean, default True :param group: Subgraph to render node in :type group: default None :param tags: Set of tags to apply to node :type tags: Iterable :param styles: Style to apply to node :type styles: String, default None :param executor: Name of executor to run node on :type executor: string :param store: Name of the blob store this node's value should be saved to, for values that belong somewhere other than the saved file --- a bucket, a database. The store itself is supplied at save and load time as stores={name: ...}, so the graph names a destination without holding a bucket or a credential. A profile override for the same node wins over this, which is what lets one computation be saved to that store in production and to a plain container in a test. :type store: string, default None

set_tag(name: Name | Names, tag: str | Iterable[str]) -> None

Set tags on a node or nodes. Ignored if tags are already set.

:param name: Node or nodes to set tag for :param tag: Tag to set

clear_tag(name: Name | Names, tag: str | Iterable[str]) -> None

Clear tag on a node or nodes. Ignored if tags are not set.

:param name: Node or nodes to clear tags for :param tag: Tag to clear

set_style(name: Name | Names, style: str) -> None

Set styles on a node or nodes.

:param name: Node or nodes to set style for :param style: Style to set

clear_style(name: Name | Names) -> None

Clear style on a node or nodes.

:param name: Node or nodes to clear styles for

metadata(name: Name) -> dict[str, Any]

Get metadata for a node.

delete_node(name: Name) -> None

Delete a node from a computation.

When nodes are explicitly deleted with delete_node, but are still depended on by other nodes, then they will be set to PLACEHOLDER status. In this case, if the nodes that depend on a PLACEHOLDER node are deleted, then the PLACEHOLDER node will also be deleted.

:param name: Name of the node to delete. If the node does not exist, a NonExistentNodeException will be raised.

rename_node(old_name: Name | Mapping[Name, Name], new_name: Name | None = None) -> None

Rename a node in a computation.

:param old_name: Node to rename, or a dictionary of nodes to rename, with existing names as keys, and new names as values :param new_name: New name for node.

repoint(old_name: Name, new_name: Name) -> None

Changes all nodes that use old_name as an input to use new_name instead.

Note that if old_name is an input to new_name, then that will not be changed, to try to avoid introducing circular dependencies, but other circular dependencies will not be checked.

If new_name does not exist, then it will be created as a PLACEHOLDER node.

:param old_name: :param new_name: :return:

insert(name: Name, value: Any, force: bool = False) -> None

Insert a value into a node of a computation.

Following insertation, the node will have state UPTODATE, and all its descendents will be COMPUTABLE or STALE.

If an attempt is made to insert a value into a node that does not exist, a NonExistentNodeException will be raised.

:param name: Name of the node to add. :param value: The value to be inserted into the node. :param force: Whether to force recalculation of descendents if node value and state would not be changed

insert_many(name_value_pairs: Iterable[tuple[Name, object]]) -> None

Insert values into many nodes of a computation simultaneously.

Following insertation, the nodes will have state UPTODATE, and all their descendents will be COMPUTABLE or STALE. In the case of inserting many nodes, some of which are descendents of others, this ensures that the inserted nodes have correct status, rather than being set as STALE when their ancestors are inserted.

If an attempt is made to insert a value into a node that does not exist, a NonExistentNodeException will be raised, and none of the nodes will be inserted.

:param name_value_pairs: Each tuple should be a pair (name, value), where name is the name of the node to insert the value into. :type name_value_pairs: List of tuples

insert_from(other: Computation, nodes: Iterable[Name] | None = None) -> None

Insert values into another Computation object into this Computation object.

:param other: The computation object to take values from :type Computation: :param nodes: Only populate the nodes with the names provided in this list. By default, all nodes from the other Computation object that have corresponding nodes in this Computation object will be inserted :type nodes: List, default None

set_stale(name: Name) -> None

Set the state of a node and all its dependencies to STALE.

:param name: Name of the node to set as STALE.

pin(name: Name, value: Any = None) -> None

Set the state of a node to PINNED.

:param name: Name of the node to set as PINNED. :param value: Value to pin to the node, if provided. :type value: default None

unpin(name: Name) -> None

Unpin a node (state of node and all descendents will be set to STALE).

:param name: Name of the node to set as PINNED.

validate() -> ValidationReport

Inspect the entire graph for structural and readiness problems.

Validation does not execute functions or mutate the computation.

plan(targets: Name | Names | None = None) -> ExecutionPlan

Describe the work needed to compute one or more targets.

Passing None plans the whole graph. Planning does not execute functions or mutate the computation.

:param targets: Target node, list of target nodes, or None for all nodes.

get_definition_args_kwds(name: Name) -> tuple[list[Any], dict[str, Any]]

Get the arguments and keyword arguments for a node's function definition.

compute(name: Name | Iterable[Name], raise_exceptions: bool = False) -> None

Compute a node or block and all necessary predecessors.

Following the computation, if successful, the target node, and all necessary ancestors that were not already UPTODATE will have been calculated and set to UPTODATE. Any node that did not need to be calculated will not have been recalculated.

If any nodes raises an exception, then the state of that node will be set to ERROR, and its value set to an object containing the exception object, as well as a traceback. This will not halt the computation, which will proceed as far as it can, until no more nodes that would be required to calculate the target are COMPUTABLE.

A block name computes every node below that path. Multiple node and block names may be supplied in a list or generator.

:param name: Name of the node or block to compute :param raise_exceptions: Whether to pass exceptions raised by node computations back to the caller :type raise_exceptions: Boolean, default False

compute_all(raise_exceptions: bool = False) -> None

Compute all nodes of a computation that can be computed.

Nodes that are already UPTODATE will not be recalculated. Following the computation, if successful, all nodes will have state UPTODATE, except UNINITIALIZED input nodes and PLACEHOLDER nodes.

If any nodes raises an exception, then the state of that node will be set to ERROR, and its value set to an object containing the exception object, as well as a traceback. This will not halt the computation, which will proceed as far as it can, until no more nodes are COMPUTABLE.

:param raise_exceptions: Whether to pass exceptions raised by node computations back to the caller :type raise_exceptions: Boolean, default False

nodes() -> list[Name]

Get a list of nodes in this computation.

:return: List of nodes.

get_tree_list_children(name: Name) -> set[Name]

Get a list of nodes in this computation.

:return: List of nodes.

has_node(name: Name) -> bool

Check if a node with the given name exists in the computation.

tree_has_path(name: Name) -> bool

Check if a hierarchical path exists in the computation tree.

get_tree_descendents(name: Name | None = None, *, include_stem: bool = True, graph_nodes_only: bool = False) -> set[Name]

Get a list of descendent blocks and nodes.

Returns blocks and nodes that are descendents of the input node, e.g. for node 'foo', might return ['foo/bar', 'foo/baz'].

:param name: Name of node to get descendents for :return: List of descendent node names

state(name: Name | Names) -> States | list[States]

state(name: Name) -> States
state(name: Names) -> list[States]

Get the state of a node.

This can also be accessed using the attribute-style accessor s if name is a valid Python attribute name::

>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.state('foo')
<States.UPTODATE: 4>
>>> comp.s.foo
<States.UPTODATE: 4>

:param name: Name or names of the node to get state for :type name: Name or Names

value(name: Name | Names) -> Any | list[Any]

value(name: Name) -> Any
value(name: Names) -> list[Any]

Get the current value of a node.

This can also be accessed using the attribute-style accessor v if name is a valid Python attribute name::

>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.value('foo')
1
>>> comp.v.foo
1

:param name: Name or names of the node to get the value of :type name: Name or Names

compute_and_get_value(name: Name) -> Any

Get the current value of a node.

This can also be accessed using the attribute-style accessor v if name is a valid Python attribute name::

>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', lambda foo: foo + 1)
>>> comp.compute_and_get_value('bar')
2
>>> comp.x.bar
2

:param name: Name or names of the node to get the value of :type name: Name

tags(name: Name | Names) -> set[str] | list[set[str]]

tags(name: Name) -> set[str]
tags(name: Names) -> list[set[str]]

Get the tags associated with a node.

>>> comp = Computation()
>>> comp.add_node('a', tags=['foo', 'bar'])
>>> sorted(comp.t.a)
['__serialize__', 'bar', 'foo']

:param name: Name or names of the node to get the tags of :return:

nodes_by_tag(tag: str | Iterable[str]) -> set[Name]

Get the names of nodes with a particular tag or tags.

:param tag: Tag or tags for which to retrieve nodes :return: Names of the nodes with those tags

styles(name: Name | Names) -> str | list[str | None] | None

styles(name: Name) -> str | None
styles(name: Names) -> list[str | None]

Get the tags associated with a node.

>>> comp = Computation()
>>> comp.add_node('a', style='dot')
>>> comp.style.a
'dot'

:param name: Name or names of the node to get the tags of :return:

__getitem__(name: Name | Names) -> NodeData | list[NodeData]

__getitem__(name: Name) -> NodeData
__getitem__(name: Names) -> list[NodeData]

Get the state and current value of a node.

:param name: Name of the node to get the state and value of

get_timing(name: Name | Names) -> TimingData | list[TimingData | None] | None

get_timing(name: Name) -> TimingData | None
get_timing(name: Names) -> list[TimingData | None]

Get the timing information for a node.

:param name: Name or names of the node to get the timing information of :return:

to_df() -> pd.DataFrame

Get a dataframe containing the states and value of all nodes of computation.

::

>>> import loman
>>> comp = loman.Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', value=2)
>>> comp.to_df()  # doctest: +NORMALIZE_WHITESPACE
               state  value
foo  States.UPTODATE      1
bar  States.UPTODATE      2

to_dict() -> dict[NodeKey, Any]

Get a dictionary containing the values of all nodes of a computation.

::

>>> import loman
>>> comp = loman.Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', value=2)
>>> comp.to_dict()  # doctest: +ELLIPSIS
{NodeKey('foo'): 1, NodeKey('bar'): 2}

get_inputs(name: Name | Names) -> Names | list[Names]

get_inputs(name: Name) -> Names
get_inputs(name: Names) -> list[Names]

Get a list of the inputs for a node or set of nodes.

:param name: Name or names of nodes to get inputs for :return: If name is scalar, return a list of upstream nodes used as input. If name is a list, return a list of list of inputs.

get_ancestors(names: Name | Names, include_self: bool = True) -> Names

Get all ancestor nodes of the specified nodes.

get_original_inputs(names: Name | Names | None = None) -> Names

Get a list of the original non-computed inputs for a node or set of nodes.

:param names: Name or names of nodes to get inputs for :return: Return a list of original non-computed inputs that are ancestors of the input nodes

get_outputs(name: Name | Names) -> Names | list[Names]

get_outputs(name: Name) -> Names
get_outputs(name: Names) -> list[Names]

Get a list of the outputs for a node or set of nodes.

:param name: Name or names of nodes to get outputs for :return: If name is scalar, return a list of downstream nodes used as output. If name is a list, return a list of list of outputs.

get_descendents(names: Name | Names, include_self: bool = True) -> Names

Get all descendent nodes of the specified nodes.

get_final_outputs(names: Name | Names | None = None) -> Names

Get final output nodes (nodes with no descendants) from the specified nodes.

get_source(name: Name) -> str

Get the source code for a node.

print_source(name: Name) -> None

Print the source code for a computation node.

restrict(output_names: Name | Names, input_names: Name | Names | None = None) -> None

Restrict a computation to the ancestors of a set of output nodes.

Excludes ancestors of a set of input nodes.

If the set of input_nodes that is specified is not sufficient for the set of output_nodes then additional nodes that are ancestors of the output_nodes will be included, but the input nodes specified will be input nodes of the modified Computation.

:param output_nodes: :param input_nodes: :return: None - modifies existing computation in place

__getstate__() -> dict[str, Any]

Prepare computation for serialization by removing non-serializable nodes.

__setstate__(state: dict[str, Any]) -> None

Restore computation from serialized state.

write_dill_old(file_: str | BinaryIO) -> None

Serialize a computation to a file or file-like object.

.. deprecated:: Superseded by :meth:write_dill, and by :meth:save in turn. Kept because removing it would break callers without warning; it will go in a release that says so.

.. warning:: Not safe to call concurrently. It removes __getstate__ and __setstate__ from the class for the duration of the write, which is process-wide, so another thread pickling a Computation at the same moment gets the wrong representation. :meth:write_dill has no such problem, and :meth:save supersedes both.

:param file_: If string, writes to a file :type file_: File-like object, or string

write_dill(file_: str | BinaryIO) -> None

Serialize a computation to a file or file-like object.

.. deprecated:: Use :meth:write_json instead. dill-based serialization will be removed in a future release.

:param file_: If string, writes to a file :type file_: File-like object, or string

read_dill(file_: str | BinaryIO) -> Computation staticmethod

Deserialize a computation from a file or file-like object.

.. deprecated:: Use :meth:read_json instead. dill-based serialization will be removed in a future release.

.. warning:: This method uses dill.load() which can execute arbitrary code. Only load files from trusted sources. Never load data from untrusted or unauthenticated sources as it may lead to arbitrary code execution.

:param file_: If string, writes to a file :type file_: File-like object, or string

save(path: str, *, profile: str | SerializationProfile | None = None, container: str | None = None, stores: dict[str, BlobStore] | None = None, serializer: ComputationSerializer | None = None) -> None

Save this computation to path.

The default is a .loman file: one zip holding a manifest.json describing the graph, plus a blobs/ directory holding large values as binary. The manifest still records every value's shape, dtype and index type, so the file can be inspected without decoding any of the data.

::

comp.save('run.loman')                       # efficient, zipped
comp.save('run.loman', profile='readable')   # inline JSON, zipped
comp.save('run.json')                        # single JSON document
comp.save('run_dir', container='dir')        # same layout, unzipped

profile and container are independent. The profile decides whether a value's bytes are written inline or out of line; the container decides where they land. The one combination that cannot work is the efficient profile in the json container, which raises.

Prefer container='dir' when saving repeatedly --- updating one value in a zip rewrites the whole archive, at a cost that grows with its size, while a directory rewrites only the file that changed.

:param path: Destination path. A .json suffix selects the single document container; anything else defaults to a .loman zip. :param profile: "readable", "efficient" (the default), or a :class:~loman.serialization.profile.SerializationProfile. :param container: "zip", "dir" or "json". Inferred from path when omitted. :param stores: Named :class:~loman.serialization.blobs.BlobStore instances for values that belong somewhere other than the saved file --- a bucket, a database. A node is routed to one by add_node(store=...) or a profile override. The same names must be supplied to :meth:load. :param serializer: Optional custom serializer, for user-defined types.

load(path: str, *, serializer: ComputationSerializer | None = None, allow_code: bool = True, stores: dict[str, BlobStore] | None = None) -> Computation staticmethod

Load a computation saved by :meth:save, in any container.

The container is detected from the file itself, so a .loman archive, a directory and a plain JSON document all load through this one call --- including documents written by earlier format versions.

.. warning:: Loading restores node functions, which means importing the modules the file names, or unpickling a dill blob out of it. Both run code chosen by the file. Only load files you trust, or pass allow_code=False.

:param path: Path to a .loman file, a container directory, or a JSON document. :param serializer: Optional custom serializer, matching the one used to save. :param allow_code: When false, callables are not resolved; values, structure, states and tags still load. :param stores: Named stores for values held outside the file. A saved file records a store's name but never its configuration, so it never contains a bucket or a credential --- and cannot resolve external values without the matching store being supplied here. :rtype: Computation

write_json(file_: str | TextIO, *, serializer: ComputationSerializer | None = None) -> None

Serialize a computation to a JSON file or file-like object.

Custom types can be supported by passing a custom serializer — either a :class:~loman.serialization.computation.ComputationSerializer instance with extra transformers registered, or a subclass that overrides the transformer factory.

:param file_: Destination file path (str) or text-mode file-like object. :param serializer: Optional custom serializer. If None the default :class:~loman.serialization.computation.ComputationSerializer is used.

read_json(file_: str | TextIO, *, serializer: ComputationSerializer | None = None, allow_code: bool = True) -> Computation staticmethod

Deserialize a computation from a JSON file or file-like object.

.. warning:: Loading a computation restores its node functions, which means importing the modules the file names, or unpickling a dill blob out of it. Both run code chosen by the file. Only load files from sources you trust, or pass allow_code=False.

:param file_: Source file path (str) or text-mode file-like object. :param serializer: Optional custom serializer. If None the default :class:~loman.serialization.computation.ComputationSerializer is used. :param allow_code: When false, encoded callables are skipped rather than resolved, so no module named by the file is imported and no dill blob is unpickled. Values, structure, states and tags still load; every node's function and converter comes back as None, so the graph can be inspected but not recalculated. :rtype: Computation

copy() -> Computation

Create a copy of a computation.

The copy is shallow. Any values in the new Computation's DAG will be the same object as this Computation's DAG. As new objects will be created by any further computations, this should not be an issue.

:rtype: Computation

add_named_tuple_expansion(name: Name, namedtuple_type: type, group: str | None = None) -> None

Automatically add nodes to extract each element of a named tuple type.

It is often convenient for a calculation to return multiple values, and it is polite to do this a namedtuple rather than a regular tuple, so that later users have same name to identify elements of the tuple. It can also help make a computation clearer if a downstream computation depends on one element of such a tuple, rather than the entire tuple. This does not affect the computation per se, but it does make the intention clearer.

To avoid having to create many boiler-plate node definitions to expand namedtuples, the add_named_tuple_expansion method automatically creates new nodes for each element of a tuple. The convention is that an element called 'element', in a node called 'node' will be expanded into a new node called 'node.element', and that this will be applied for each element.

Example::

>>> from collections import namedtuple
>>> Coordinate = namedtuple('Coordinate', ['x', 'y'])
>>> comp = Computation()
>>> comp.add_node('c', value=Coordinate(1, 2))
>>> comp.add_named_tuple_expansion('c', Coordinate)
>>> comp.compute_all()
>>> comp.value('c.x')
1
>>> comp.value('c.y')
2

:param name: Node to cera :param namedtuple_type: Expected type of the node :type namedtuple_type: namedtuple class

add_map_node(result_node: Name, input_node: Name, subgraph: Computation, subgraph_input_node: Name, subgraph_output_node: Name) -> None

Apply a graph to each element of iterable.

In turn, each element in the input_node of this graph will be inserted in turn into the subgraph's subgraph_input_node, then the subgraph's subgraph_output_node calculated. The resultant list, with an element or each element in input_node, will be inserted into result_node of this graph. In this way add_map_node is similar to map in functional programming.

:param result_node: The node to place a list of results in this graph :param input_node: The node to get a list input values from this graph :param subgraph: The graph to use to perform calculation for each element :param subgraph_input_node: The node in subgraph to insert each element in turn :param subgraph_output_node: The node in subgraph to read the result for each element

prepend_path(path: Name | ConstantValue, prefix_path: NodeKey) -> NodeKey | ConstantValue

Prepend a prefix path to a node path.

add_block(base_path: Name, block: Computation, *, keep_values: bool = True, links: dict[str, Name] | None = None, metadata: dict[str, Any] | None = None) -> None

Add a computation block as a subgraph to this computation.

keep_values defaults to True, so values already held by block are copied along with its structure: a block added this way is often a sub-model that has already been populated or calibrated, and would not be computable without them. The repeated-block utilities in :mod:loman.util default to False instead, because they stamp out many copies of one template. See :class:loman.util.RepeatedBlocks.

:param base_path: Parent path to add the block's nodes below :param block: Computation to copy into this computation :param keep_values: Whether to copy the block's current values :param links: Mapping from the block's relative input names to outer nodes :param metadata: Metadata to attach to the block path

Create a link between two nodes in the computation graph.

draw(root: NodeKey | None = None, *, node_transformations: dict[Name, str] | None = None, cmap: Any = None, colors: str = 'state', shapes: str | None = None, graph_attr: dict[str, Any] | None = None, node_attr: dict[str, Any] | None = None, edge_attr: dict[str, Any] | None = None, show_expansion: bool = False, collapse_all: bool = True) -> GraphView

Draw a computation's current state using the GraphViz utility.

:param root: Optional PathType. Sub-block to draw :param cmap: Default: None :param colors: 'state' - colors indicate state. 'timing' - colors indicate execution time. Default: 'state'. :param shapes: None - ovals. 'type' - shapes indicate type. Default: None. :param graph_attr: Mapping of (attribute, value) pairs for the graph. For example graph_attr={'size': '"10,8"'} can control the size of the output graph :param node_attr: Mapping of (attribute, value) pairs set for all nodes. :param edge_attr: Mapping of (attribute, value) pairs set for all edges. :param collapse_all: Whether to collapse all blocks that aren't explicitly expanded.

widget(root: NodeKey | None = None, *, node_transformations: dict[Name, str] | None = None, cmap: Any = None, colors: str = 'state', shapes: str | None = None, graph_attr: dict[str, Any] | None = None, node_attr: dict[str, Any] | None = None, edge_attr: dict[str, Any] | None = None, show_expansion: bool = False, collapse_all: bool = True, editable: bool = True, buildable: bool = False, namespace: dict[str, Any] | None = None, fit_on_render: bool = False, max_rendered_nodes: int = 500, rankdir: str = 'LR') -> ComputationWidget

Create an interactive notebook widget for this computation.

Requires the ui extra: pip install 'loman[ui]'. It is imported lazily, so ordinary use of Loman does not load AnyWidget or its notebook dependencies.

Arguments other than editable mirror :meth:draw, so the two are learnable together. The widget subscribes to this computation and follows it; comp.draw() and _repr_svg_ stay static pictures.

Note that colors="state" repaints in place, while any other colouring re-runs Graphviz on every change. See :class:~loman.ui.ComputationWidget for the details.

:param editable: Allow scalar input editing and computation controls. Expanding and collapsing blocks stays available either way. :param buildable: Allow the graph itself to be built in the widget: adding, redefining, renaming and deleting nodes. Off by default, and needs editable too, because defining a calculation node compiles and runs an expression typed in the browser inside this kernel. :param namespace: Globals a node built in the widget is compiled against, so its expression can use the notebook's own imports. Pass globals(). The default is an empty namespace, in which only builtins are in scope. :param fit_on_render: Scale the graph to fit the pane on every render, rather than opening at natural size. Useful when the shape of a large graph matters more than its labels. :param max_rendered_nodes: Refuse to open a block that would put more than this many nodes on screen. Does not cap the initial view. :param rankdir: Initial Graphviz layout direction, LR (default) or TB, toggled live from the toolbar. :return: A live widget subscribed to this computation.

view(cmap: Any = None, colors: str = 'state', shapes: str | None = None) -> None

Create and display a visualization of the computation graph.

print_errors() -> None

Print tracebacks for every node with state "ERROR" in a Computation.

from_class(definition_class: type, ignore_self: bool = True) -> Computation classmethod

Create a computation from a class with decorated methods.

inject_dependencies(dependencies: dict[Name, Any], *, force: bool = False) -> None

Injects dependencies into the nodes of the current computation where nodes are in a placeholder state.

(or all possible nodes when the 'force' parameter is set to True), using values provided in the 'dependencies' dictionary.

Each key in the 'dependencies' dictionary corresponds to a node identifier, and the associated value is the dependency object to inject. If the value is a callable, it will be added as a calc node.

:param dependencies: A dictionary where each key-value pair consists of a node identifier and its corresponding dependency object or a callable that returns the dependency object. :param force: A boolean flag that, when set to True, forces the replacement of existing node values with the ones provided in 'dependencies', regardless of their current state. Defaults to False. :return: None

ComputationEvent dataclass

A batched notification describing a mutation to a computation.

Subscribers receive one event after each outermost public mutation, even when that operation performs many internal state transitions. Values are deliberately excluded: consumers can fetch a changed value lazily from :attr:computation without copying large objects into every event.

:ivar computation: The live computation that produced the event. It is not a snapshot, and continues to change after the event is delivered. :ivar revision: Monotonic counter, matching :attr:Computation.revision at the moment the event was published. :ivar changed_nodes: Nodes whose state changed during the operation. When :attr:graph_changed is true this is not a complete description of the change, because adding, deleting or renaming nodes and altering tags or styles need not change any node's state. Consumers reacting to a structural event should re-read the graph rather than trusting this set. :ivar states: The state of each entry in :attr:changed_nodes that still exists, as of publication. Deleted nodes are absent. :ivar graph_changed: True when the structure or presentation of the graph changed, so any cached rendering of it is stale.

NodeTransformations

Node transformation types for visualization.

States

Bases: Enum

Possible states for a computation node.

CannotInsertToPlaceholderNodeError

Bases: ComputationError

Exception raised when trying to insert into a placeholder node.

DeserializedError

Bases: ComputationError

Stand-in for an exception whose original class could not be rebuilt.

A saved ERROR node records the name of the exception that produced it. Rebuilding an arbitrary one would mean importing whatever module the file names, which is executing code chosen by the file, so only builtin exception types are reconstructed. Everything else becomes one of these, carrying the original identity as data for post-mortem reading.

:ivar exception_type: Name of the exception class that was originally raised. :ivar exception_module: Module that class came from, when it was recorded.

__init__(message: str, exception_type: str, exception_module: str | None = None) -> None

Record the original exception's identity alongside its message.

__repr__() -> str

Show the original exception's type so it is not mistaken for this one.

FittingError

Bases: ComputationError

Exception raised when curve fitting exceeds error tolerance.

InvalidBlockTypeError

Bases: TypeError, ComputationError

Exception raised when a block is not callable or a Computation.

LoopDetectedError

Bases: ComputationError

Exception raised when a dependency loop is detected.

MapError

Bases: ComputationError

Exception raised during map operations with partial results.

__init__(message: str, results: list[object]) -> None

Initialize MapError with message and partial results.

NonExistentNodeError

Bases: ComputationError

Exception raised when trying to access a non-existent node.

SerializationError

Bases: ComputationError

Exception raised during serialization/deserialization.

ValidationError

Bases: ComputationError

Exception raised during computation validation.

NodeKey dataclass

Immutable key for identifying nodes in the computation graph hierarchy.

name: Name property

Get the name of this node (last part of the path).

label: str property

Get the label for this node (for display purposes).

parent: NodeKey property

Get the parent node key.

is_root: bool property

Check if this is the root node key.

__str__() -> str

Return string representation using path notation.

drop_root(root: Optional[Name]) -> Optional[NodeKey]

Remove a root prefix from this node key if it matches.

join(*others: Name) -> NodeKey

Join this node key with other names to create a new node key.

join_parts(*parts: Hashable) -> NodeKey

Join this node key with raw parts to create a new node key.

__truediv__(other: Name) -> NodeKey

Join this node key with other to create a new node key.

is_descendent_of(other: NodeKey) -> bool

Check if this node key is a descendant of another node key.

prepend(nk: NodeKey) -> NodeKey

Prepend another node key to this one.

__repr__() -> str

Return string representation for debugging.

__eq__(other: object) -> bool

Check equality with another NodeKey.

root() -> NodeKey classmethod

Get the root node key.

common_parent(nodekey1: Name, nodekey2: Name) -> NodeKey staticmethod

Find the common parent of two node keys.

ancestors() -> list[NodeKey]

Get all ancestor node keys from root to parent.

ExecutionPlan dataclass

A non-mutating description of work required by a computation.

is_feasible: bool property

Return whether all requested targets can be brought up to date.

to_df() -> pd.DataFrame

Return current, runnable, and blocked nodes as an execution table.

ValidationReport dataclass

Structural and readiness findings for a computation graph.

is_valid: bool property

Return whether the graph is structurally valid.

is_ready: bool property

Return whether the graph is valid and has all required inputs.

to_df() -> pd.DataFrame

Return validation findings as one row per issue.

ComputationSerializer

Serialize and deserialize a :class:~loman.computeengine.Computation graph to JSON.

The serialized format is a JSON object with the following top-level keys:

  • version: integer format version
  • nodes: list of node objects
  • edges: list of edge objects

Each node object has:

  • key: string representation of the NodeKey
  • state: name of the :class:~loman.consts.States enum member (or null)
  • value: transformer-encoded value (or null when absent / not serialized)
  • has_value: bool — false when the node has no meaningful value to restore
  • func: transformer-encoded callable (or null)
  • args: transformer-encoded constant positional arguments, keyed by stringified positional index
  • kwds: transformer-encoded constant keyword arguments, keyed by parameter name
  • serialize: bool — whether the node has the __serialize__ tag
  • tags: list of non-system tags

Each edge object has:

  • src: string key of the source node
  • dst: string key of the destination node
  • param_type: "arg" or "kwd"
  • param: positional index (int) for args, parameter name (str) for kwds

Arguments taken from other nodes are recorded on edges, while arguments given as :class:~loman.computeengine.ConstantValue are held on the node itself and recorded in args and kwds. Both are needed to call a node's function, so a graph that dropped its constants would raise a :class:TypeError the next time the node was calculated.

Parameters

transformer: Custom :class:~loman.serialization.transformer.Transformer instance. If None, a default transformer is built based on use_dill_for_functions. use_dill_for_functions: When True, lambdas and closures are serialized as base64-encoded dill blobs rather than raising :class:~loman.exception.SerializationError. Has no effect when a custom transformer is supplied. Defaults to False. on_unserializable_constant: What to do when a constant argument cannot be encoded. "raise", the default, refuses to write a graph that could not be recalculated. "drop" omits the constant and emits :class:UnserializableConstantWarning, restoring the behaviour of releases before constants were recorded — where such a graph saved silently and then raised :class:TypeError from the missing argument on the first recalculation. It exists so an existing codebase can keep writing files while it is fixed, not as a setting to leave in place.

__init__(transformer: Transformer | None = None, *, use_dill_for_functions: bool = False, on_unserializable_constant: str = 'raise') -> None

Initialise with an optional custom transformer.

register(t: Any) -> None

Register a transformer or type with this serializer's transformer.

Accepts anything :meth:~loman.serialization.transformer.Transformer.register accepts: a :class:~loman.serialization.transformer.CustomTransformer instance, a :class:~loman.serialization.transformer.Transformable subclass, an attrs class, or a dataclass.

The same serializer instance must be used for both writing and reading, since the registration lives on the instance::

s = ComputationSerializer()
s.register(my_transformer)
comp.write_json('comp.json', serializer=s)
comp2 = Computation.read_json('comp.json', serializer=s)

save(comp: Any, path: str | Path, *, profile: str | SerializationProfile | None = None, container: str | None = None, stores: dict[str, BlobStore] | None = None) -> None

Write comp to path.

:param path: Destination. A .json suffix implies the single-document container; anything else defaults to a .loman zip. :param profile: "readable", "efficient", or a :class:~loman.serialization.profile.SerializationProfile. Defaults to efficient, except in the json container where only readable is possible. :param container: "zip", "dir" or "json". Inferred from path when omitted. :param stores: Named :class:~loman.serialization.blobs.BlobStore instances that nodes may be routed to. A node names a store through add_node(store=...) or a profile override.

load_path(path: str | Path, *, serializer: ComputationSerializer | None = None, allow_code: bool = True, stores: dict[str, BlobStore] | None = None) -> Any staticmethod

Read a computation from path, whatever container it uses.

:param stores: Named stores for blobs held outside the container. A saved file records a store's name but never its configuration, so a file with external blobs cannot resolve them unaided.

dump(comp: Any, fp: TextIO) -> None

Serialize comp to fp (a text-mode file-like object).

dumps(comp: Any) -> str

Serialize comp and return a JSON string.

Always the readable single-document form: a string has nowhere to put out-of-line bytes.

load(fp: TextIO, *, allow_code: bool = True) -> Any

Deserialize a Computation from fp (a text-mode file-like object).

:param allow_code: When false, node functions and converters are not restored. See :meth:loads.

loads(s: str, *, allow_code: bool = True) -> Any

Deserialize a Computation from a JSON string.

:param allow_code: When false, encoded callables are skipped rather than resolved, and every node's function and converter comes back as None. Restoring a callable means importing the module the file names, or unpickling a dill blob out of it --- both of which run code the file chose. Values, structure, states and tags still load, which is enough to inspect a graph from an untrusted source. Defaults to true, preserving existing behaviour.

SerializationProfile

How values are encoded for one save.

:ivar name: Identifier recorded in the manifest. :ivar inline_max_bytes: Values estimated at or below this many bytes stay inline. None keeps everything inline, whatever the container. :ivar overrides: Selector-to-settings map, letting one save treat some nodes differently. A selector is a node-key glob ("market_data/**") or a tag ("tag:raw").

wants_blob(nbytes: int | None) -> bool

Return whether a value of nbytes should be written out of line.

A transformer that cannot estimate its size passes None and is taken at its word that the value is worth storing out of line.

settings_for(node: str | None, tags: frozenset[str] = frozenset()) -> dict[str, Any]

Return the override settings that apply to node.

Later matches win, so a more specific selector listed after a general one takes precedence --- the order the overrides were written in.

BlockContext dataclass

Bases: Generic[K]

What a feature is given when it plans its nodes.

blocks maps each key to the path of its generated block. block is the template, so a feature can check that a relative name it was given really is a node, or an input node, of the block being repeated.

require_block_node(name: Name, description: str) -> NodeKey

Resolve a relative name that must be a node inside every block.

The name may come from the template, or from a node an earlier feature already planned, so one feature can build on another's output. Features are planned in the order they are declared.

require_block_input(name: Name, description: str, *, create: bool = False) -> NodeKey

Resolve a relative name that must be an input node inside every block.

A node the template declares must be an input: replacing a calculation would silently discard it. A node an earlier feature planned is accepted as-is, since the template has nothing to say about it. A name the template never mentions is rejected, because it is usually a typo that would otherwise add a dead node to every block — pass create=True to allow it deliberately.

bind(func: Any) -> Any

Bind a callback to the class a computation factory was defined from.

Returns anything that is not callable unchanged, so a plain node name passes through.

BlockFeature

Bases: Protocol

One wiring pattern applied to every copy of a repeated block.

A feature never changes the computation itself. It describes the nodes it wants, and the builder validates every feature's plan together before applying any of it, so a definition that fails leaves the graph untouched. Implement this protocol to add a wiring pattern of your own.

plan(ctx: BlockContext[Any]) -> Iterable[PlannedNode]

Describe the nodes to create, without changing anything.

FanIn dataclass

Bases: Generic[K]

Collect one relative output from every repeated block into a result node.

source is a name inside each block. result is not relative to the definition's base_path: it names a node in the outer computation, so the aggregate can live wherever it belongs rather than being forced under the blocks. BuiltRepeatedBlocks.named reports the key that was created.

plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]

Plan one result node gathering the same relative node from every block.

FanOut dataclass

Bases: Generic[K]

Wire one source node to a relative input in every repeated block.

source normally names a single outer node feeding every block. Passing a callable instead resolves a source node per key, as source(key), so each block can read from a different outer node. With a transform, each target is calculated as transform(value, key).

target is a name inside each block. It must be something the template declares or refers to, so a typo does not quietly add a dead node to every block; set create=True to feed a name the template never mentions.

plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]

Plan one target node per key, linked or transformed from its source.

IdNode dataclass

Give every block a node holding its own key.

Block functions can then depend on their key by name, to look data up or to branch on it, without the key being wired in from outside.

create defaults to True, unlike :class:FanOut, because creating the node is this feature's whole job: a template that never mentions the name is the ordinary case, not a mistake. The cost is that a misspelled name adds a node nothing reads and leaves the real one unfilled — validate() reports that as an uninitialized input, but only once the graph is built. Set create=False where the template does declare the node, to have the misspelling rejected at definition time instead.

plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]

Plan one value node per key, holding that key.

InputValue dataclass

Give every block the same constant value for one relative input.

Unlike :class:FanIn's result, the shared node this creates is relative to the definition's base_path, landing at <base_path>/<name> so that two definitions with different base paths do not collide.

create behaves as it does on :class:FanOut, and defaults the same way: seeding a value into a name the template never mentions is usually a typo, so it must be asked for.

plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]

Plan one shared outer node, linked into every block.

PlannedNode dataclass

One node a feature intends to create, described rather than created.

Features return these instead of changing the computation, so the builder can check every node and edge of a definition before any of it is applied. A node with no func is an input node holding value; otherwise func is called with args, where each argument is either a :class:NodeKey to depend on or a :class:ConstantValue to pass through unchanged.

predecessors: tuple[NodeKey, ...] property

Return the nodes this planned node would depend on.

input_node(node_key: NodeKey, value: Any, label: Name | None = None) -> PlannedNode classmethod

Plan an input node holding a fixed value.

Plan a node that takes its value from another node unchanged.

calc(node_key: NodeKey, func: Callable[..., Any], args: Sequence[Any], label: Name | None = None) -> PlannedNode classmethod

Plan a calculation node.

apply_to(comp: Computation) -> NodeKey

Create this node in a computation.

Positional dataclass

Adapt an aggregator that takes positional arguments to a keyed combine.

combine receives an ordered mapping so keys stay attached to values, which is usually what you want. Where an existing function takes the values positionally, wrap it rather than repeating lambda m: fn(*m.values())::

FanIn("value", "total", combine=Positional(df_hconcat))

Keys are discarded. A keyed aggregator can use them instead — for dataframes, lambda m: pd.concat(m, axis=1) turns them into column labels — but that is a different result, not a drop-in replacement: it adds an outer level to the column index, where a flat positional concatenation does not. Choose it because you want the keys in the output, not as a like-for-like swap.

Like any callable that is not an importable module-level function, this needs use_dill_for_functions=True to serialize.

__call__(values: Mapping[Any, Any]) -> Any

Call the wrapped function with the mapping's values, in order.

GraphView dataclass

A view for visualizing computation graphs as graphical diagrams.

__post_init__() -> None

Initialize the graph view after dataclass construction.

get_sub_block(dag: nx.DiGraph, root: Name | None, node_transformations: dict[NodeKey, str]) -> tuple[nx.DiGraph, defaultdict[NodeKey, list[NodeKey]], set[NodeKey]] staticmethod

Extract a subgraph with node transformations for visualization.

refresh() -> None

Refresh the visualization by rebuilding the graph structure.

svg() -> str | None

Generate SVG representation of the visualization.

view() -> None

Open the visualization in a PDF viewer.

calc_node(f: F | None = None, **kwds: Any) -> F | Callable[[F], F]

calc_node(f: F, **kwds: Any) -> F
calc_node(f: None = None, **kwds: Any) -> Callable[[F], F]

Decorator to mark a function as a calculation node.

computation_factory(maybe_cls: type | None = None, *, ignore_self: bool = True) -> Callable[..., Computation] | Callable[[type], Callable[..., Computation]]

Factory function to create computations from class definitions.

node(comp: Computation, name: Name | None = None, *args: Any, **kw: Any) -> Callable[[F], F]

Decorator to add a function as a node to a computation graph.

to_nodekey(name: Name) -> NodeKey

Convert a name to a NodeKey object.