Staging

Testing the decorators to help orchestrate caching and input/output passing through record state between stages.

Exceptions:

`CachersMismatchError`
`EmptyCachersError`
`InputSignatureError`
`OutputSignatureError`

Functions:

`aggregate`([outputs, cachers])	Decorator to wrap around a function that represents some step that must operate across multiple different argsets or “experiment lines” within an experiment.
`stage`([inputs, outputs, cachers, …])	Decorator to wrap around a function that represents a single step in an experiment, a block with inputs and outputs pertaining to the remainder of that experiment.

exception curifactory.staging.CachersMismatchError

exception curifactory.staging.EmptyCachersError

exception curifactory.staging.InputSignatureError

exception curifactory.staging.OutputSignatureError

curifactory.staging.aggregate(outputs: Optional[list] = None, cachers: Optional[list] = None)

Decorator to wrap around a function that represents some step that must operate across multiple different argsets or “experiment lines” within an experiment. This is normally used to run final analyses and comparisons of results across all passed argument sets.

Important

Any function wrapped with the aggregate decorator must take a Record instance as the first parameter and a list of Record instances as the second. The former is the record that applies to this function, and the latter is the set of other records from elsewhere in the experiment that this function needs to aggregate across.

Parameters

outputs (List[str]) – A list of variable names that this stage will return and store in the record state. These represent, in order, the tuple of returned values from the function being wrapped.
cachers (List[Cacheable]) – An optional list of Cacheable instances (“strategies”) to apply to each of the return outputs. If specified, for each output, an instance of the corresponding cacher is initialized, and the save() function is called. Before the wrapped function is called, the output path is first checked, and if it exists and the current record args are not set to overwrite, the load() function is called and the wrapped function does not execute. Note that caching is all or nothing for a single function, you cannot cache only one returned value out of several.

Example

@aggregate(["final_results"], [JsonCacher])
def compile_results(record: Record, records: List[Record]):
    results = {}
    for prev_record in records:
        results[prev_record.args.name] = prev_record.state["results"]
    return results

curifactory.staging.stage(inputs: Optional[list] = None, outputs: Optional[list] = None, cachers: Optional[list] = None, suppress_missing_inputs: bool = False)

Decorator to wrap around a function that represents a single step in an experiment, a block with inputs and outputs pertaining to the remainder of that experiment.

Important

Any function wrapped with the stage decorator must take a Record instance as the first parameter, followed by the input parameters corresponding to the inputs list.

Parameters

inputs (List[str]) – A list of variable names that this stage will need from the record state. Note that all inputs listed here must have a corresponding input parameter in the function definition line, each with the exact same name as in this list.
outputs (List[Union[str, Lazy]]) – A list of variable names that this stage will return and store in the record state. These represent, in order, the tuple of returned values from the function being wrapped.
cachers (List[Cacheable]) – An optional list of Cacheable instances (“strategies”) to apply to each of the return outputs. If specified, for each output, an instance of the corresponding cacher is initialized, and the save() function is called. Before the wrapped function is called, the output path is first checked, and if it exists and the current record args are not set to overwrite, the load() function is called and the wrapped function does not execute. Note that caching is all or nothing for a single function, you cannot cache only one returned value out of several.
suppress_missing_inputs (bool) – If true, any stage inputs that are not found in the record’s state will be passed in as None rather than raising an exception. This can be used to make all inputs optional, such as if a stage will be used after different sets of previous stages and not all values are necessarily required.

Example

@stage(inputs=["data", "model"], outputs=["results"], cachers=[JsonCacher])
def test_model(record: Record, data: pd.DataFrame, model):
    # ...
    return results_dictionary

Note that from this example, this stage assumes some other stages have output "data" and "model" at some point.