API Reference¶
loman
¶
Loman: A Python library for building computation graphs.
Loman provides tools for creating and managing dependency-aware computation graphs where nodes represent data or calculations, and edges represent dependencies.
Computation
¶
A computation graph that manages dependencies and calculations.
The Computation class provides a framework for building and executing computation graphs where nodes represent data or calculations, and edges represent dependencies between them.
revision: int
property
¶
Return the revision number of the most recently published change.
__init__(*, default_executor: Executor | None = None, executor_map: dict[str, Executor] | None = None, metadata: dict[str, Any] | None = None) -> None
¶
Initialize a new Computation.
:param default_executor: An executor :type default_executor: concurrent.futures.Executor, default ThreadPoolExecutor(max_workers=1)
subscribe(callback: ComputationSubscriber) -> Callable[[], None]
¶
Subscribe to batched computation changes.
Subscribers are notified in registration order, synchronously, on the thread that completes the outermost public mutation. A subscriber that raises is logged and skipped; it never interrupts the mutation or the other subscribers. A subscriber that itself mutates the computation causes a further event to be published once the current round finishes, up to a bounded number of cascades.
A callback with an object behind it --- anything carrying a
__self__, whether written in Python or in C --- is held weakly, so
subscribing obj.handler or events.append does not keep the
owner alive; callers must retain it themselves. Everything else ---
plain functions, lambdas, callable objects, :func:functools.partial
--- is held strongly until unsubscribed, because callers commonly pass
a throwaway closure that nothing else references.
The exception is an owner that supports no weak references at all, such
as :class:list, :class:dict and :class:bytearray. There
some_list.append falls back to a strong reference, which is a
limitation of the type rather than a choice, and errs towards a
subscription that keeps delivering over one that silently stops.
Subscriptions are not copied by :meth:copy and are not serialized.
:param callback: Function accepting a :class:ComputationEvent.
:return: An idempotent, no-argument unsubscribe function.
get_attribute_view_for_path(nodekey: NodeKey, get_one_func: Callable[[Name], Any], get_many_func: Callable[[Name | Names], Any]) -> AttributeView
¶
Create an attribute view for a specific node path.
add_node(name: Name, func: Callable[..., Any] | None = None, *, args: list[Any] | None = None, kwds: dict[str, Any] | None = None, value: Any = _MISSING_VALUE_SENTINEL, converter: Callable[[Any], Any] | None = None, serialize: bool = True, inspect: bool = True, group: str | None = None, tags: Iterable[str] | None = None, style: str | None = None, executor: str | None = None, store: str | None = None, metadata: dict[str, Any] | None = None) -> None
¶
Adds or updates a node in a computation.
:param name: Name of the node to add. This may be any hashable object.
:param func: Function to use to calculate the node if the node is a calculation node. By default, the input
nodes to the function will be implied from the names of the function parameters. For example, a
parameter called a would be taken from the node called a. This can be modified with the
kwds parameter.
:type func: Function, default None
:param args: Specifies a list of nodes that will be used to populate arguments of the function positionally
for a calculation node. e.g. If args is ['a', 'b', 'c'] then the function would be called with
three parameters, taken from the nodes 'a', 'b' and 'c' respectively.
:type args: List, default None
:param kwds: Specifies a mapping from parameter name to the node that should be used to populate that
parameter when calling the function for a calculation node. e.g. If args is {'x': 'a', 'y': 'b'}
then the function would be called with parameters named 'x' and 'y', and their values would be taken
from nodes 'a' and 'b' respectively. Each entry in the dictionary can be read as "take parameter
[key] from node [value]".
:type kwds: Dictionary, default None
:param value: If given, the value is inserted into the node, and the node state set to UPTODATE.
:type value: default None
:param converter: Callable applied to any value on its way into the node. The node stores what the
converter returns, both for values supplied by value, insert and insert_many, and for
values the node calculates with func. A converter that raises leaves the node in state ERROR
without storing the value, which is how a validator is written: check the value and return it
unchanged when it is acceptable. The exception propagates to the caller when the value was
supplied, but not when it was calculated, where the failure is reported as node state instead.
A converter is saved by reference, like a node's function, so it must be importable: a
module-level function or builtin round-trips, while a lambda raises SerializationError.
:type converter: Callable, default None
:param serialize: Whether the node should be serialized. Some objects cannot be serialized, in which
case, set serialize to False
:type serialize: boolean, default True
:param inspect: Whether to use introspection to determine the arguments of the function, which can be
slow. If this is not set, kwds and args must be set for the function to obtain parameters.
:type inspect: boolean, default True
:param group: Subgraph to render node in
:type group: default None
:param tags: Set of tags to apply to node
:type tags: Iterable
:param styles: Style to apply to node
:type styles: String, default None
:param executor: Name of executor to run node on
:type executor: string
:param store: Name of the blob store this node's value should be saved
to, for values that belong somewhere other than the saved file --- a
bucket, a database. The store itself is supplied at save and load
time as stores={name: ...}, so the graph names a destination
without holding a bucket or a credential. A profile override for the
same node wins over this, which is what lets one computation be
saved to that store in production and to a plain container in a test.
:type store: string, default None
set_tag(name: Name | Names, tag: str | Iterable[str]) -> None
¶
Set tags on a node or nodes. Ignored if tags are already set.
:param name: Node or nodes to set tag for :param tag: Tag to set
clear_tag(name: Name | Names, tag: str | Iterable[str]) -> None
¶
Clear tag on a node or nodes. Ignored if tags are not set.
:param name: Node or nodes to clear tags for :param tag: Tag to clear
set_style(name: Name | Names, style: str) -> None
¶
Set styles on a node or nodes.
:param name: Node or nodes to set style for :param style: Style to set
clear_style(name: Name | Names) -> None
¶
Clear style on a node or nodes.
:param name: Node or nodes to clear styles for
metadata(name: Name) -> dict[str, Any]
¶
Get metadata for a node.
delete_node(name: Name) -> None
¶
Delete a node from a computation.
When nodes are explicitly deleted with delete_node, but are still depended on by other nodes, then they
will be set to PLACEHOLDER status. In this case, if the nodes that depend on a PLACEHOLDER node are deleted,
then the PLACEHOLDER node will also be deleted.
:param name: Name of the node to delete. If the node does not exist, a NonExistentNodeException will
be raised.
rename_node(old_name: Name | Mapping[Name, Name], new_name: Name | None = None) -> None
¶
Rename a node in a computation.
:param old_name: Node to rename, or a dictionary of nodes to rename, with existing names as keys, and new names as values :param new_name: New name for node.
repoint(old_name: Name, new_name: Name) -> None
¶
Changes all nodes that use old_name as an input to use new_name instead.
Note that if old_name is an input to new_name, then that will not be changed, to try to avoid introducing circular dependencies, but other circular dependencies will not be checked.
If new_name does not exist, then it will be created as a PLACEHOLDER node.
:param old_name: :param new_name: :return:
insert(name: Name, value: Any, force: bool = False) -> None
¶
Insert a value into a node of a computation.
Following insertation, the node will have state UPTODATE, and all its descendents will be COMPUTABLE or STALE.
If an attempt is made to insert a value into a node that does not exist, a NonExistentNodeException
will be raised.
:param name: Name of the node to add. :param value: The value to be inserted into the node. :param force: Whether to force recalculation of descendents if node value and state would not be changed
insert_many(name_value_pairs: Iterable[tuple[Name, object]]) -> None
¶
Insert values into many nodes of a computation simultaneously.
Following insertation, the nodes will have state UPTODATE, and all their descendents will be COMPUTABLE or STALE. In the case of inserting many nodes, some of which are descendents of others, this ensures that the inserted nodes have correct status, rather than being set as STALE when their ancestors are inserted.
If an attempt is made to insert a value into a node that does not exist, a NonExistentNodeException will be
raised, and none of the nodes will be inserted.
:param name_value_pairs: Each tuple should be a pair (name, value), where name is the name of the node to insert the value into. :type name_value_pairs: List of tuples
insert_from(other: Computation, nodes: Iterable[Name] | None = None) -> None
¶
Insert values into another Computation object into this Computation object.
:param other: The computation object to take values from :type Computation: :param nodes: Only populate the nodes with the names provided in this list. By default, all nodes from the other Computation object that have corresponding nodes in this Computation object will be inserted :type nodes: List, default None
set_stale(name: Name) -> None
¶
Set the state of a node and all its dependencies to STALE.
:param name: Name of the node to set as STALE.
pin(name: Name, value: Any = None) -> None
¶
Set the state of a node to PINNED.
:param name: Name of the node to set as PINNED. :param value: Value to pin to the node, if provided. :type value: default None
unpin(name: Name) -> None
¶
Unpin a node (state of node and all descendents will be set to STALE).
:param name: Name of the node to set as PINNED.
validate() -> ValidationReport
¶
Inspect the entire graph for structural and readiness problems.
Validation does not execute functions or mutate the computation.
plan(targets: Name | Names | None = None) -> ExecutionPlan
¶
Describe the work needed to compute one or more targets.
Passing None plans the whole graph. Planning does not execute
functions or mutate the computation.
:param targets: Target node, list of target nodes, or None for all nodes.
get_definition_args_kwds(name: Name) -> tuple[list[Any], dict[str, Any]]
¶
Get the arguments and keyword arguments for a node's function definition.
compute(name: Name | Iterable[Name], raise_exceptions: bool = False) -> None
¶
Compute a node or block and all necessary predecessors.
Following the computation, if successful, the target node, and all necessary ancestors that were not already UPTODATE will have been calculated and set to UPTODATE. Any node that did not need to be calculated will not have been recalculated.
If any nodes raises an exception, then the state of that node will be set to ERROR, and its value set to an object containing the exception object, as well as a traceback. This will not halt the computation, which will proceed as far as it can, until no more nodes that would be required to calculate the target are COMPUTABLE.
A block name computes every node below that path. Multiple node and block names may be supplied in a list or generator.
:param name: Name of the node or block to compute :param raise_exceptions: Whether to pass exceptions raised by node computations back to the caller :type raise_exceptions: Boolean, default False
compute_all(raise_exceptions: bool = False) -> None
¶
Compute all nodes of a computation that can be computed.
Nodes that are already UPTODATE will not be recalculated. Following the computation, if successful, all nodes will have state UPTODATE, except UNINITIALIZED input nodes and PLACEHOLDER nodes.
If any nodes raises an exception, then the state of that node will be set to ERROR, and its value set to an object containing the exception object, as well as a traceback. This will not halt the computation, which will proceed as far as it can, until no more nodes are COMPUTABLE.
:param raise_exceptions: Whether to pass exceptions raised by node computations back to the caller :type raise_exceptions: Boolean, default False
nodes() -> list[Name]
¶
Get a list of nodes in this computation.
:return: List of nodes.
get_tree_list_children(name: Name) -> set[Name]
¶
Get a list of nodes in this computation.
:return: List of nodes.
has_node(name: Name) -> bool
¶
Check if a node with the given name exists in the computation.
tree_has_path(name: Name) -> bool
¶
Check if a hierarchical path exists in the computation tree.
get_tree_descendents(name: Name | None = None, *, include_stem: bool = True, graph_nodes_only: bool = False) -> set[Name]
¶
Get a list of descendent blocks and nodes.
Returns blocks and nodes that are descendents of the input node, e.g. for node 'foo', might return ['foo/bar', 'foo/baz'].
:param name: Name of node to get descendents for :return: List of descendent node names
state(name: Name | Names) -> States | list[States]
¶
state(name: Name) -> States
state(name: Names) -> list[States]
Get the state of a node.
This can also be accessed using the attribute-style accessor s if name is a valid Python
attribute name::
>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.state('foo')
<States.UPTODATE: 4>
>>> comp.s.foo
<States.UPTODATE: 4>
:param name: Name or names of the node to get state for :type name: Name or Names
value(name: Name | Names) -> Any | list[Any]
¶
value(name: Name) -> Any
value(name: Names) -> list[Any]
Get the current value of a node.
This can also be accessed using the attribute-style accessor v if name is a valid Python
attribute name::
>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.value('foo')
1
>>> comp.v.foo
1
:param name: Name or names of the node to get the value of :type name: Name or Names
compute_and_get_value(name: Name) -> Any
¶
Get the current value of a node.
This can also be accessed using the attribute-style accessor v if name is a valid Python
attribute name::
>>> comp = Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', lambda foo: foo + 1)
>>> comp.compute_and_get_value('bar')
2
>>> comp.x.bar
2
:param name: Name or names of the node to get the value of :type name: Name
tags(name: Name | Names) -> set[str] | list[set[str]]
¶
tags(name: Name) -> set[str]
tags(name: Names) -> list[set[str]]
Get the tags associated with a node.
>>> comp = Computation()
>>> comp.add_node('a', tags=['foo', 'bar'])
>>> sorted(comp.t.a)
['__serialize__', 'bar', 'foo']
:param name: Name or names of the node to get the tags of :return:
nodes_by_tag(tag: str | Iterable[str]) -> set[Name]
¶
Get the names of nodes with a particular tag or tags.
:param tag: Tag or tags for which to retrieve nodes :return: Names of the nodes with those tags
styles(name: Name | Names) -> str | list[str | None] | None
¶
styles(name: Name) -> str | None
styles(name: Names) -> list[str | None]
Get the tags associated with a node.
>>> comp = Computation()
>>> comp.add_node('a', style='dot')
>>> comp.style.a
'dot'
:param name: Name or names of the node to get the tags of :return:
__getitem__(name: Name | Names) -> NodeData | list[NodeData]
¶
__getitem__(name: Name) -> NodeData
__getitem__(name: Names) -> list[NodeData]
Get the state and current value of a node.
:param name: Name of the node to get the state and value of
get_timing(name: Name | Names) -> TimingData | list[TimingData | None] | None
¶
get_timing(name: Name) -> TimingData | None
get_timing(name: Names) -> list[TimingData | None]
Get the timing information for a node.
:param name: Name or names of the node to get the timing information of :return:
to_df() -> pd.DataFrame
¶
Get a dataframe containing the states and value of all nodes of computation.
::
>>> import loman
>>> comp = loman.Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', value=2)
>>> comp.to_df() # doctest: +NORMALIZE_WHITESPACE
state value
foo States.UPTODATE 1
bar States.UPTODATE 2
to_dict() -> dict[NodeKey, Any]
¶
Get a dictionary containing the values of all nodes of a computation.
::
>>> import loman
>>> comp = loman.Computation()
>>> comp.add_node('foo', value=1)
>>> comp.add_node('bar', value=2)
>>> comp.to_dict() # doctest: +ELLIPSIS
{NodeKey('foo'): 1, NodeKey('bar'): 2}
get_inputs(name: Name | Names) -> Names | list[Names]
¶
get_inputs(name: Name) -> Names
get_inputs(name: Names) -> list[Names]
Get a list of the inputs for a node or set of nodes.
:param name: Name or names of nodes to get inputs for :return: If name is scalar, return a list of upstream nodes used as input. If name is a list, return a list of list of inputs.
get_ancestors(names: Name | Names, include_self: bool = True) -> Names
¶
Get all ancestor nodes of the specified nodes.
get_original_inputs(names: Name | Names | None = None) -> Names
¶
Get a list of the original non-computed inputs for a node or set of nodes.
:param names: Name or names of nodes to get inputs for :return: Return a list of original non-computed inputs that are ancestors of the input nodes
get_outputs(name: Name | Names) -> Names | list[Names]
¶
get_outputs(name: Name) -> Names
get_outputs(name: Names) -> list[Names]
Get a list of the outputs for a node or set of nodes.
:param name: Name or names of nodes to get outputs for :return: If name is scalar, return a list of downstream nodes used as output. If name is a list, return a list of list of outputs.
get_descendents(names: Name | Names, include_self: bool = True) -> Names
¶
Get all descendent nodes of the specified nodes.
get_final_outputs(names: Name | Names | None = None) -> Names
¶
Get final output nodes (nodes with no descendants) from the specified nodes.
get_source(name: Name) -> str
¶
Get the source code for a node.
print_source(name: Name) -> None
¶
Print the source code for a computation node.
restrict(output_names: Name | Names, input_names: Name | Names | None = None) -> None
¶
Restrict a computation to the ancestors of a set of output nodes.
Excludes ancestors of a set of input nodes.
If the set of input_nodes that is specified is not sufficient for the set of output_nodes then additional nodes that are ancestors of the output_nodes will be included, but the input nodes specified will be input nodes of the modified Computation.
:param output_nodes: :param input_nodes: :return: None - modifies existing computation in place
__getstate__() -> dict[str, Any]
¶
Prepare computation for serialization by removing non-serializable nodes.
__setstate__(state: dict[str, Any]) -> None
¶
Restore computation from serialized state.
write_dill_old(file_: str | BinaryIO) -> None
¶
Serialize a computation to a file or file-like object.
.. deprecated::
Superseded by :meth:write_dill, and by :meth:save in turn. Kept
because removing it would break callers without warning; it will go
in a release that says so.
.. warning::
Not safe to call concurrently. It removes __getstate__ and
__setstate__ from the class for the duration of the write, which
is process-wide, so another thread pickling a Computation at the same
moment gets the wrong representation. :meth:write_dill has no such
problem, and :meth:save supersedes both.
:param file_: If string, writes to a file :type file_: File-like object, or string
write_dill(file_: str | BinaryIO) -> None
¶
Serialize a computation to a file or file-like object.
.. deprecated::
Use :meth:write_json instead. dill-based serialization will be
removed in a future release.
:param file_: If string, writes to a file :type file_: File-like object, or string
read_dill(file_: str | BinaryIO) -> Computation
staticmethod
¶
Deserialize a computation from a file or file-like object.
.. deprecated::
Use :meth:read_json instead. dill-based serialization will be
removed in a future release.
.. warning:: This method uses dill.load() which can execute arbitrary code. Only load files from trusted sources. Never load data from untrusted or unauthenticated sources as it may lead to arbitrary code execution.
:param file_: If string, writes to a file :type file_: File-like object, or string
save(path: str, *, profile: str | SerializationProfile | None = None, container: str | None = None, stores: dict[str, BlobStore] | None = None, serializer: ComputationSerializer | None = None) -> None
¶
Save this computation to path.
The default is a .loman file: one zip holding a manifest.json
describing the graph, plus a blobs/ directory holding large values as
binary. The manifest still records every value's shape, dtype and index
type, so the file can be inspected without decoding any of the data.
::
comp.save('run.loman') # efficient, zipped
comp.save('run.loman', profile='readable') # inline JSON, zipped
comp.save('run.json') # single JSON document
comp.save('run_dir', container='dir') # same layout, unzipped
profile and container are independent. The profile decides whether a
value's bytes are written inline or out of line; the container decides
where they land. The one combination that cannot work is the efficient
profile in the json container, which raises.
Prefer container='dir' when saving repeatedly --- updating one value
in a zip rewrites the whole archive, at a cost that grows with its size,
while a directory rewrites only the file that changed.
:param path: Destination path. A .json suffix selects the single
document container; anything else defaults to a .loman zip.
:param profile: "readable", "efficient" (the default), or a
:class:~loman.serialization.profile.SerializationProfile.
:param container: "zip", "dir" or "json". Inferred from
path when omitted.
:param stores: Named
:class:~loman.serialization.blobs.BlobStore instances for values
that belong somewhere other than the saved file --- a bucket, a
database. A node is routed to one by add_node(store=...) or a
profile override. The same names must be supplied to :meth:load.
:param serializer: Optional custom serializer, for user-defined types.
load(path: str, *, serializer: ComputationSerializer | None = None, allow_code: bool = True, stores: dict[str, BlobStore] | None = None) -> Computation
staticmethod
¶
Load a computation saved by :meth:save, in any container.
The container is detected from the file itself, so a .loman archive,
a directory and a plain JSON document all load through this one call ---
including documents written by earlier format versions.
.. warning::
Loading restores node functions, which means importing the modules
the file names, or unpickling a dill blob out of it. Both run code
chosen by the file. Only load files you trust, or pass
allow_code=False.
:param path: Path to a .loman file, a container directory, or a
JSON document.
:param serializer: Optional custom serializer, matching the one used to
save.
:param allow_code: When false, callables are not resolved; values,
structure, states and tags still load.
:param stores: Named stores for values held outside the file. A saved
file records a store's name but never its configuration, so it never
contains a bucket or a credential --- and cannot resolve external
values without the matching store being supplied here.
:rtype: Computation
write_json(file_: str | TextIO, *, serializer: ComputationSerializer | None = None) -> None
¶
Serialize a computation to a JSON file or file-like object.
Custom types can be supported by passing a custom serializer —
either a :class:~loman.serialization.computation.ComputationSerializer
instance with extra transformers registered, or a subclass that
overrides the transformer factory.
:param file_: Destination file path (str) or text-mode file-like object.
:param serializer: Optional custom serializer. If None the default
:class:~loman.serialization.computation.ComputationSerializer is used.
read_json(file_: str | TextIO, *, serializer: ComputationSerializer | None = None, allow_code: bool = True) -> Computation
staticmethod
¶
Deserialize a computation from a JSON file or file-like object.
.. warning::
Loading a computation restores its node functions, which means
importing the modules the file names, or unpickling a dill blob out
of it. Both run code chosen by the file. Only load files from
sources you trust, or pass allow_code=False.
:param file_: Source file path (str) or text-mode file-like object.
:param serializer: Optional custom serializer. If None the default
:class:~loman.serialization.computation.ComputationSerializer is used.
:param allow_code: When false, encoded callables are skipped rather than
resolved, so no module named by the file is imported and no dill blob
is unpickled. Values, structure, states and tags still load; every
node's function and converter comes back as None, so the graph
can be inspected but not recalculated.
:rtype: Computation
copy() -> Computation
¶
Create a copy of a computation.
The copy is shallow. Any values in the new Computation's DAG will be the same object as this Computation's DAG. As new objects will be created by any further computations, this should not be an issue.
:rtype: Computation
add_named_tuple_expansion(name: Name, namedtuple_type: type, group: str | None = None) -> None
¶
Automatically add nodes to extract each element of a named tuple type.
It is often convenient for a calculation to return multiple values, and it is polite to do this a namedtuple rather than a regular tuple, so that later users have same name to identify elements of the tuple. It can also help make a computation clearer if a downstream computation depends on one element of such a tuple, rather than the entire tuple. This does not affect the computation per se, but it does make the intention clearer.
To avoid having to create many boiler-plate node definitions to expand namedtuples, the
add_named_tuple_expansion method automatically creates new nodes for each element of a tuple. The
convention is that an element called 'element', in a node called 'node' will be expanded into a new node
called 'node.element', and that this will be applied for each element.
Example::
>>> from collections import namedtuple
>>> Coordinate = namedtuple('Coordinate', ['x', 'y'])
>>> comp = Computation()
>>> comp.add_node('c', value=Coordinate(1, 2))
>>> comp.add_named_tuple_expansion('c', Coordinate)
>>> comp.compute_all()
>>> comp.value('c.x')
1
>>> comp.value('c.y')
2
:param name: Node to cera :param namedtuple_type: Expected type of the node :type namedtuple_type: namedtuple class
add_map_node(result_node: Name, input_node: Name, subgraph: Computation, subgraph_input_node: Name, subgraph_output_node: Name) -> None
¶
Apply a graph to each element of iterable.
In turn, each element in the input_node of this graph will be inserted in turn into the subgraph's
subgraph_input_node, then the subgraph's subgraph_output_node calculated. The resultant list, with
an element or each element in input_node, will be inserted into result_node of this graph. In this
way add_map_node is similar to map in functional programming.
:param result_node: The node to place a list of results in this graph :param input_node: The node to get a list input values from this graph :param subgraph: The graph to use to perform calculation for each element :param subgraph_input_node: The node in subgraph to insert each element in turn :param subgraph_output_node: The node in subgraph to read the result for each element
prepend_path(path: Name | ConstantValue, prefix_path: NodeKey) -> NodeKey | ConstantValue
¶
Prepend a prefix path to a node path.
add_block(base_path: Name, block: Computation, *, keep_values: bool = True, links: dict[str, Name] | None = None, metadata: dict[str, Any] | None = None) -> None
¶
Add a computation block as a subgraph to this computation.
keep_values defaults to True, so values already held by block
are copied along with its structure: a block added this way is often a
sub-model that has already been populated or calibrated, and would not be
computable without them. The repeated-block utilities in
:mod:loman.util default to False instead, because they stamp out
many copies of one template. See :class:loman.util.RepeatedBlocks.
:param base_path: Parent path to add the block's nodes below :param block: Computation to copy into this computation :param keep_values: Whether to copy the block's current values :param links: Mapping from the block's relative input names to outer nodes :param metadata: Metadata to attach to the block path
link(target: Name, source: Name) -> None
¶
Create a link between two nodes in the computation graph.
draw(root: NodeKey | None = None, *, node_transformations: dict[Name, str] | None = None, cmap: Any = None, colors: str = 'state', shapes: str | None = None, graph_attr: dict[str, Any] | None = None, node_attr: dict[str, Any] | None = None, edge_attr: dict[str, Any] | None = None, show_expansion: bool = False, collapse_all: bool = True) -> GraphView
¶
Draw a computation's current state using the GraphViz utility.
:param root: Optional PathType. Sub-block to draw
:param cmap: Default: None
:param colors: 'state' - colors indicate state. 'timing' - colors indicate execution time. Default: 'state'.
:param shapes: None - ovals. 'type' - shapes indicate type. Default: None.
:param graph_attr: Mapping of (attribute, value) pairs for the graph. For example
graph_attr={'size': '"10,8"'} can control the size of the output graph
:param node_attr: Mapping of (attribute, value) pairs set for all nodes.
:param edge_attr: Mapping of (attribute, value) pairs set for all edges.
:param collapse_all: Whether to collapse all blocks that aren't explicitly expanded.
widget(root: NodeKey | None = None, *, node_transformations: dict[Name, str] | None = None, cmap: Any = None, colors: str = 'state', shapes: str | None = None, graph_attr: dict[str, Any] | None = None, node_attr: dict[str, Any] | None = None, edge_attr: dict[str, Any] | None = None, show_expansion: bool = False, collapse_all: bool = True, editable: bool = True, buildable: bool = False, namespace: dict[str, Any] | None = None, fit_on_render: bool = False, max_rendered_nodes: int = 500, rankdir: str = 'LR') -> ComputationWidget
¶
Create an interactive notebook widget for this computation.
Requires the ui extra: pip install 'loman[ui]'. It is imported
lazily, so ordinary use of Loman does not load AnyWidget or its notebook
dependencies.
Arguments other than editable mirror :meth:draw, so the two are
learnable together. The widget subscribes to this computation and
follows it; comp.draw() and _repr_svg_ stay static pictures.
Note that colors="state" repaints in place, while any other
colouring re-runs Graphviz on every change. See
:class:~loman.ui.ComputationWidget for the details.
:param editable: Allow scalar input editing and computation controls.
Expanding and collapsing blocks stays available either way.
:param buildable: Allow the graph itself to be built in the
widget: adding, redefining, renaming and deleting nodes. Off by
default, and needs editable too, because defining a calculation
node compiles and runs an expression typed in the browser inside
this kernel.
:param namespace: Globals a node built in the widget is compiled
against, so its expression can use the notebook's own imports.
Pass globals(). The default is an empty namespace, in which
only builtins are in scope.
:param fit_on_render: Scale the graph to fit the pane on every render,
rather than opening at natural size. Useful when the shape of a
large graph matters more than its labels.
:param max_rendered_nodes: Refuse to open a block that would put more
than this many nodes on screen. Does not cap the initial view.
:param rankdir: Initial Graphviz layout direction, LR (default) or
TB, toggled live from the toolbar.
:return: A live widget subscribed to this computation.
view(cmap: Any = None, colors: str = 'state', shapes: str | None = None) -> None
¶
Create and display a visualization of the computation graph.
print_errors() -> None
¶
Print tracebacks for every node with state "ERROR" in a Computation.
from_class(definition_class: type, ignore_self: bool = True) -> Computation
classmethod
¶
Create a computation from a class with decorated methods.
inject_dependencies(dependencies: dict[Name, Any], *, force: bool = False) -> None
¶
Injects dependencies into the nodes of the current computation where nodes are in a placeholder state.
(or all possible nodes when the 'force' parameter is set to True), using values provided in the 'dependencies' dictionary.
Each key in the 'dependencies' dictionary corresponds to a node identifier, and the associated value is the dependency object to inject. If the value is a callable, it will be added as a calc node.
:param dependencies: A dictionary where each key-value pair consists of a node identifier and its corresponding dependency object or a callable that returns the dependency object. :param force: A boolean flag that, when set to True, forces the replacement of existing node values with the ones provided in 'dependencies', regardless of their current state. Defaults to False. :return: None
ComputationEvent
dataclass
¶
A batched notification describing a mutation to a computation.
Subscribers receive one event after each outermost public mutation, even
when that operation performs many internal state transitions. Values are
deliberately excluded: consumers can fetch a changed value lazily from
:attr:computation without copying large objects into every event.
:ivar computation: The live computation that produced the event. It is not a
snapshot, and continues to change after the event is delivered.
:ivar revision: Monotonic counter, matching :attr:Computation.revision at
the moment the event was published.
:ivar changed_nodes: Nodes whose state changed during the operation. When
:attr:graph_changed is true this is not a complete description of the
change, because adding, deleting or renaming nodes and altering tags or
styles need not change any node's state. Consumers reacting to a
structural event should re-read the graph rather than trusting this set.
:ivar states: The state of each entry in :attr:changed_nodes that still
exists, as of publication. Deleted nodes are absent.
:ivar graph_changed: True when the structure or presentation of the graph
changed, so any cached rendering of it is stale.
NodeTransformations
¶
Node transformation types for visualization.
States
¶
Bases: Enum
Possible states for a computation node.
CannotInsertToPlaceholderNodeError
¶
Bases: ComputationError
Exception raised when trying to insert into a placeholder node.
DeserializedError
¶
Bases: ComputationError
Stand-in for an exception whose original class could not be rebuilt.
A saved ERROR node records the name of the exception that produced it. Rebuilding an arbitrary one would mean importing whatever module the file names, which is executing code chosen by the file, so only builtin exception types are reconstructed. Everything else becomes one of these, carrying the original identity as data for post-mortem reading.
:ivar exception_type: Name of the exception class that was originally raised. :ivar exception_module: Module that class came from, when it was recorded.
FittingError
¶
Bases: ComputationError
Exception raised when curve fitting exceeds error tolerance.
InvalidBlockTypeError
¶
Bases: TypeError, ComputationError
Exception raised when a block is not callable or a Computation.
LoopDetectedError
¶
Bases: ComputationError
Exception raised when a dependency loop is detected.
MapError
¶
Bases: ComputationError
Exception raised during map operations with partial results.
__init__(message: str, results: list[object]) -> None
¶
Initialize MapError with message and partial results.
NonExistentNodeError
¶
Bases: ComputationError
Exception raised when trying to access a non-existent node.
SerializationError
¶
Bases: ComputationError
Exception raised during serialization/deserialization.
ValidationError
¶
Bases: ComputationError
Exception raised during computation validation.
NodeKey
dataclass
¶
Immutable key for identifying nodes in the computation graph hierarchy.
name: Name
property
¶
Get the name of this node (last part of the path).
label: str
property
¶
Get the label for this node (for display purposes).
parent: NodeKey
property
¶
Get the parent node key.
is_root: bool
property
¶
Check if this is the root node key.
__str__() -> str
¶
Return string representation using path notation.
drop_root(root: Optional[Name]) -> Optional[NodeKey]
¶
Remove a root prefix from this node key if it matches.
join(*others: Name) -> NodeKey
¶
Join this node key with other names to create a new node key.
join_parts(*parts: Hashable) -> NodeKey
¶
Join this node key with raw parts to create a new node key.
__truediv__(other: Name) -> NodeKey
¶
Join this node key with other to create a new node key.
is_descendent_of(other: NodeKey) -> bool
¶
Check if this node key is a descendant of another node key.
prepend(nk: NodeKey) -> NodeKey
¶
Prepend another node key to this one.
__repr__() -> str
¶
Return string representation for debugging.
__eq__(other: object) -> bool
¶
Check equality with another NodeKey.
root() -> NodeKey
classmethod
¶
Get the root node key.
common_parent(nodekey1: Name, nodekey2: Name) -> NodeKey
staticmethod
¶
Find the common parent of two node keys.
ancestors() -> list[NodeKey]
¶
Get all ancestor node keys from root to parent.
ExecutionPlan
dataclass
¶
ValidationReport
dataclass
¶
ComputationSerializer
¶
Serialize and deserialize a :class:~loman.computeengine.Computation graph to JSON.
The serialized format is a JSON object with the following top-level keys:
version: integer format versionnodes: list of node objectsedges: list of edge objects
Each node object has:
key: string representation of the NodeKeystate: name of the :class:~loman.consts.Statesenum member (ornull)value: transformer-encoded value (ornullwhen absent / not serialized)has_value: bool — false when the node has no meaningful value to restorefunc: transformer-encoded callable (ornull)args: transformer-encoded constant positional arguments, keyed by stringified positional indexkwds: transformer-encoded constant keyword arguments, keyed by parameter nameserialize: bool — whether the node has the__serialize__tagtags: list of non-system tags
Each edge object has:
src: string key of the source nodedst: string key of the destination nodeparam_type:"arg"or"kwd"param: positional index (int) for args, parameter name (str) for kwds
Arguments taken from other nodes are recorded on edges, while arguments
given as :class:~loman.computeengine.ConstantValue are held on the node
itself and recorded in args and kwds. Both are needed to call a node's
function, so a graph that dropped its constants would raise a
:class:TypeError the next time the node was calculated.
Parameters¶
transformer:
Custom :class:~loman.serialization.transformer.Transformer instance.
If None, a default transformer is built based on use_dill_for_functions.
use_dill_for_functions:
When True, lambdas and closures are serialized as base64-encoded dill
blobs rather than raising :class:~loman.exception.SerializationError.
Has no effect when a custom transformer is supplied. Defaults to False.
on_unserializable_constant:
What to do when a constant argument cannot be encoded. "raise", the
default, refuses to write a graph that could not be recalculated.
"drop" omits the constant and emits
:class:UnserializableConstantWarning, restoring the behaviour of
releases before constants were recorded — where such a graph saved
silently and then raised :class:TypeError from the missing argument on
the first recalculation. It exists so an existing codebase can keep
writing files while it is fixed, not as a setting to leave in place.
__init__(transformer: Transformer | None = None, *, use_dill_for_functions: bool = False, on_unserializable_constant: str = 'raise') -> None
¶
Initialise with an optional custom transformer.
register(t: Any) -> None
¶
Register a transformer or type with this serializer's transformer.
Accepts anything :meth:~loman.serialization.transformer.Transformer.register
accepts: a :class:~loman.serialization.transformer.CustomTransformer
instance, a :class:~loman.serialization.transformer.Transformable
subclass, an attrs class, or a dataclass.
The same serializer instance must be used for both writing and reading, since the registration lives on the instance::
s = ComputationSerializer()
s.register(my_transformer)
comp.write_json('comp.json', serializer=s)
comp2 = Computation.read_json('comp.json', serializer=s)
save(comp: Any, path: str | Path, *, profile: str | SerializationProfile | None = None, container: str | None = None, stores: dict[str, BlobStore] | None = None) -> None
¶
Write comp to path.
:param path: Destination. A .json suffix implies the single-document
container; anything else defaults to a .loman zip.
:param profile: "readable", "efficient", or a
:class:~loman.serialization.profile.SerializationProfile. Defaults
to efficient, except in the json container where only readable is
possible.
:param container: "zip", "dir" or "json". Inferred from
path when omitted.
:param stores: Named :class:~loman.serialization.blobs.BlobStore
instances that nodes may be routed to. A node names a store through
add_node(store=...) or a profile override.
load_path(path: str | Path, *, serializer: ComputationSerializer | None = None, allow_code: bool = True, stores: dict[str, BlobStore] | None = None) -> Any
staticmethod
¶
Read a computation from path, whatever container it uses.
:param stores: Named stores for blobs held outside the container. A saved file records a store's name but never its configuration, so a file with external blobs cannot resolve them unaided.
dump(comp: Any, fp: TextIO) -> None
¶
Serialize comp to fp (a text-mode file-like object).
dumps(comp: Any) -> str
¶
Serialize comp and return a JSON string.
Always the readable single-document form: a string has nowhere to put out-of-line bytes.
load(fp: TextIO, *, allow_code: bool = True) -> Any
¶
Deserialize a Computation from fp (a text-mode file-like object).
:param allow_code: When false, node functions and converters are not
restored. See :meth:loads.
loads(s: str, *, allow_code: bool = True) -> Any
¶
Deserialize a Computation from a JSON string.
:param allow_code: When false, encoded callables are skipped rather than
resolved, and every node's function and converter comes back as
None. Restoring a callable means importing the module the file
names, or unpickling a dill blob out of it --- both of which run code
the file chose. Values, structure, states and tags still load, which
is enough to inspect a graph from an untrusted source. Defaults to
true, preserving existing behaviour.
SerializationProfile
¶
How values are encoded for one save.
:ivar name: Identifier recorded in the manifest.
:ivar inline_max_bytes: Values estimated at or below this many bytes stay
inline. None keeps everything inline, whatever the container.
:ivar overrides: Selector-to-settings map, letting one save treat some nodes
differently. A selector is a node-key glob ("market_data/**") or a
tag ("tag:raw").
wants_blob(nbytes: int | None) -> bool
¶
Return whether a value of nbytes should be written out of line.
A transformer that cannot estimate its size passes None and is taken
at its word that the value is worth storing out of line.
settings_for(node: str | None, tags: frozenset[str] = frozenset()) -> dict[str, Any]
¶
Return the override settings that apply to node.
Later matches win, so a more specific selector listed after a general one takes precedence --- the order the overrides were written in.
BlockContext
dataclass
¶
Bases: Generic[K]
What a feature is given when it plans its nodes.
blocks maps each key to the path of its generated block. block is the
template, so a feature can check that a relative name it was given really is a
node, or an input node, of the block being repeated.
require_block_node(name: Name, description: str) -> NodeKey
¶
Resolve a relative name that must be a node inside every block.
The name may come from the template, or from a node an earlier feature already planned, so one feature can build on another's output. Features are planned in the order they are declared.
require_block_input(name: Name, description: str, *, create: bool = False) -> NodeKey
¶
Resolve a relative name that must be an input node inside every block.
A node the template declares must be an input: replacing a calculation
would silently discard it. A node an earlier feature planned is accepted
as-is, since the template has nothing to say about it. A name the template
never mentions is rejected, because it is usually a typo that would
otherwise add a dead node to every block — pass create=True to allow
it deliberately.
bind(func: Any) -> Any
¶
Bind a callback to the class a computation factory was defined from.
Returns anything that is not callable unchanged, so a plain node name passes through.
BlockFeature
¶
Bases: Protocol
One wiring pattern applied to every copy of a repeated block.
A feature never changes the computation itself. It describes the nodes it wants, and the builder validates every feature's plan together before applying any of it, so a definition that fails leaves the graph untouched. Implement this protocol to add a wiring pattern of your own.
plan(ctx: BlockContext[Any]) -> Iterable[PlannedNode]
¶
Describe the nodes to create, without changing anything.
FanIn
dataclass
¶
Bases: Generic[K]
Collect one relative output from every repeated block into a result node.
source is a name inside each block. result is not relative to the
definition's base_path: it names a node in the outer computation, so the
aggregate can live wherever it belongs rather than being forced under the
blocks. BuiltRepeatedBlocks.named reports the key that was created.
plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]
¶
Plan one result node gathering the same relative node from every block.
FanOut
dataclass
¶
Bases: Generic[K]
Wire one source node to a relative input in every repeated block.
source normally names a single outer node feeding every block. Passing a
callable instead resolves a source node per key, as source(key), so each
block can read from a different outer node. With a transform, each target
is calculated as transform(value, key).
target is a name inside each block. It must be something the template
declares or refers to, so a typo does not quietly add a dead node to every
block; set create=True to feed a name the template never mentions.
plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]
¶
Plan one target node per key, linked or transformed from its source.
IdNode
dataclass
¶
Give every block a node holding its own key.
Block functions can then depend on their key by name, to look data up or to branch on it, without the key being wired in from outside.
create defaults to True, unlike :class:FanOut, because creating the
node is this feature's whole job: a template that never mentions the name is
the ordinary case, not a mistake. The cost is that a misspelled name adds a
node nothing reads and leaves the real one unfilled — validate() reports
that as an uninitialized input, but only once the graph is built. Set
create=False where the template does declare the node, to have the
misspelling rejected at definition time instead.
plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]
¶
Plan one value node per key, holding that key.
InputValue
dataclass
¶
Give every block the same constant value for one relative input.
Unlike :class:FanIn's result, the shared node this creates is
relative to the definition's base_path, landing at <base_path>/<name>
so that two definitions with different base paths do not collide.
create behaves as it does on :class:FanOut, and defaults the same way:
seeding a value into a name the template never mentions is usually a typo,
so it must be asked for.
plan(ctx: BlockContext[K]) -> Iterable[PlannedNode]
¶
Plan one shared outer node, linked into every block.
PlannedNode
dataclass
¶
One node a feature intends to create, described rather than created.
Features return these instead of changing the computation, so the builder can
check every node and edge of a definition before any of it is applied. A node
with no func is an input node holding value; otherwise func is
called with args, where each argument is either a :class:NodeKey to
depend on or a :class:ConstantValue to pass through unchanged.
predecessors: tuple[NodeKey, ...]
property
¶
Return the nodes this planned node would depend on.
input_node(node_key: NodeKey, value: Any, label: Name | None = None) -> PlannedNode
classmethod
¶
Plan an input node holding a fixed value.
link(node_key: NodeKey, source: NodeKey, label: Name | None = None) -> PlannedNode
classmethod
¶
Plan a node that takes its value from another node unchanged.
calc(node_key: NodeKey, func: Callable[..., Any], args: Sequence[Any], label: Name | None = None) -> PlannedNode
classmethod
¶
Plan a calculation node.
apply_to(comp: Computation) -> NodeKey
¶
Create this node in a computation.
Positional
dataclass
¶
Adapt an aggregator that takes positional arguments to a keyed combine.
combine receives an ordered mapping so keys stay attached to values, which
is usually what you want. Where an existing function takes the values
positionally, wrap it rather than repeating lambda m: fn(*m.values())::
FanIn("value", "total", combine=Positional(df_hconcat))
Keys are discarded. A keyed aggregator can use them instead — for dataframes,
lambda m: pd.concat(m, axis=1) turns them into column labels — but that is
a different result, not a drop-in replacement: it adds an outer level to
the column index, where a flat positional concatenation does not. Choose it
because you want the keys in the output, not as a like-for-like swap.
Like any callable that is not an importable module-level function, this needs
use_dill_for_functions=True to serialize.
__call__(values: Mapping[Any, Any]) -> Any
¶
Call the wrapped function with the mapping's values, in order.
GraphView
dataclass
¶
A view for visualizing computation graphs as graphical diagrams.
__post_init__() -> None
¶
Initialize the graph view after dataclass construction.
get_sub_block(dag: nx.DiGraph, root: Name | None, node_transformations: dict[NodeKey, str]) -> tuple[nx.DiGraph, defaultdict[NodeKey, list[NodeKey]], set[NodeKey]]
staticmethod
¶
Extract a subgraph with node transformations for visualization.
refresh() -> None
¶
Refresh the visualization by rebuilding the graph structure.
svg() -> str | None
¶
Generate SVG representation of the visualization.
view() -> None
¶
Open the visualization in a PDF viewer.
calc_node(f: F | None = None, **kwds: Any) -> F | Callable[[F], F]
¶
calc_node(f: F, **kwds: Any) -> F
calc_node(f: None = None, **kwds: Any) -> Callable[[F], F]
Decorator to mark a function as a calculation node.
computation_factory(maybe_cls: type | None = None, *, ignore_self: bool = True) -> Callable[..., Computation] | Callable[[type], Callable[..., Computation]]
¶
Factory function to create computations from class definitions.
node(comp: Computation, name: Name | None = None, *args: Any, **kw: Any) -> Callable[[F], F]
¶
Decorator to add a function as a node to a computation graph.
to_nodekey(name: Name) -> NodeKey
¶
Convert a name to a NodeKey object.