Manager

Contains the relevant classes for experiment state, namely the artifact manager and record classes, as well as experiment store/parameter store management.

Classes:

ArtifactManager(experiment_name, store_full, …)

This class manages the records, metadata, and options for an experiment run.

class curifactory.manager.ArtifactManager(experiment_name: Optional[str] = None, store_full: bool = False, dry: bool = False, dry_cache: bool = False, prefix: Optional[str] = None, run_line: str = '', parallel_lock: Optional[multiprocessing.context.BaseContext.Lock] = None, parallel_mode: bool = False, lazy: bool = False, ignore_lazy: bool = False, notes: Optional[str] = None, manager_cache_path: Optional[str] = None, cache_path: Optional[str] = None, runs_path: Optional[str] = None, logs_path: Optional[str] = None, notebooks_path: Optional[str] = None, reports_path: Optional[str] = None, report_css_path: Optional[str] = None, status_override: str = 'LIVE', suppress_live_log: bool = False, live_log_debug: bool = False, disable_non_cf_loggers: bool = True)

This class manages the records, metadata, and options for an experiment run.

Every experiment run needs an instance of this.

Parameters

experiment_name (str) – The name of the experiment. This is used as a prefix for most relevant information and affects caching paths. If a manager is being used primarily from jupyter notebooks, you could for instance set this to be the name of the notebook.
store_full (bool) – Store/copy environment info, log, output report, and all cached files in a run-specific folder (this is a –store-full run.)
dry (bool) – Setting dry to true will suppress saving any files (including logs), and will not update parameter stores. (It should have no effect on any files.)
dry_cache (bool) – Setting this to true only suppresses saving cache files. This is recommended if you’re running with a cache_dir_override for some previous –store-full run, so you don’t accidentally overwrite or add new data to the –store-full directory.
prefix (str) – Instead of using the experiment name to group cached data, use this name instead.
run_line (str) – The CLI command used to run the current experiment.
parallel_lock (multiprocessing.Lock) – If this function is called from a multiprocessing context, use this lock to help prevent files being written to and read simultaneously.
parallel_mode (bool) – This is handled by the parallel parameter, informing a particular subproc run that it is being executed from a parallel run.
lazy (bool) – If true, attempts to set all stage outputs as Lazy objects. Outputs that do not have a cacher specified will be given a PickleCacher. Note that objects without a cacher tht do not handle pickle serialization correctly may cause errors.
ignore_lazy (bool) – Run the experiment disabling any lazy object caching/keeping everything in memory. This can save time when memory is less of an issue.
notes (str) – A git-log-like message to store in the run info for the current run. If this is an empty string, query the user for an input string.
manager_cache_path (str) – The path where the parameter registry and experiment store data are kept.
cache_path (str) – The path where all intermediate data is cached.
runs_path (str) – The path where full experiment stores (run with --store-full) are kept.
logs_path (str) – The path where run logs get stored.
notebooks_path (str) – The path where run notebooks are placed.
reports_path (str) – The path where experiment run reports are saved.
report_css_path (str) – The path to a CSS file to copy into each report directory.
status_override (str) – This variable is ‘LIVE’ by default, but is overriden by the experiment script. ‘LIVE’ indicates that this manager is being run from a non-experiment script or interactive terminal.
suppress_live_log (bool) – If true, on live/interactive runs don’t automatically spawn a logger.
live_log_debug (bool) – If true, spawn live logger with the DEBUG level.
disable_non_cf_loggers (bool) – Hide loggers from other libraries. This is true by default because some libraries can generate a lot of noise.

Attributes:

`artifacts`	The list of `ArtifactRepresentation` instances for all artifacts stored in all record states.
`cache_path`	The path where all intermediate data is cached.
`conda_env`	The output from a conda env command (`conda env export --from-history`).
`config`	The manager configuration loaded from the curifactory config file if present.
`current_stage_name`	The name of the stage currently executing.
`dry`	Flag for whether to suppress all file outputs (cached files, logs, registries etc.)
`dry_cache`	Flag for whether to suppress only writing cached files.
`error`	The exception class and error string, if one was thrown.
`error_thrown`	A flag indicating whether an error was thrown by the experiment.
`experiment_name`	The name of the experiment and/or the prefix used for caching.
`experiment_run_number`	The run counter for experiments with the given name.
`git_commit_hash`	The current commit hash if a git repo is in use.
`git_workdir_dirty`	Whether there are uncommited changes in the git repo or not.
`hostname`	The hostname of the machine this experiment ran on.
`ignore_lazy`	Run the experiment disabling any lazy object caching/keeping everything in memory.
`interactive`	Indicates if this manager was spawned with LIVE status in an interactive environment (e.g.
`lazy`	If true, attempts to set all stage outputs as Lazy objects.
`live_report_path_generated`	A flag indicting if the default report’s graphs/reportables folders exist, this helps prevent live displays from breaking if multiple display functions called.
`logs_path`	The path where run logs get stored.
`manager_cache_path`	The path where the parameter registry and experiment store data are kept.
`map_mode`	If we’re in map mode, don’t actually execute any stages, we’re only recording the ‘DAG’ (really just the set of stages associated with each record)
`mapped_artifacts`	The list of `MapArtifactRepresentation` instances found during the mapping process.
`notebooks_path`	The path where run notebooks are placed.
`notes`	User-entered notes associated with a session/run to output into the report etc.
`os`	The name of the current OS running curifactory.
`overwrite`	For live session managers where you don’t wish to set overwrite on individual parameter sets, you can universely set the manager to overwrite by changing this flag to True.
`overwrite_stages`	The list of individual stages for which to ignore the cache.
`parallel_lock`	If this function is called from a multiprocessing context, use this lock to help prevent files being written to and read simultaneously.
`parallel_mode`	A flag indicating whether this manager is in a “subproc” run or not.
`param_file_param_sets`	A dictionary of parameter file names for keys, where each value is an array of arrays, each inner array containing the parameter set name and the parameter set hash, for the parameter sets that come from that parameter file.
`parameter_files`	The list of parameter file names to be used in the experiment.
`pip_freeze`	The output from a `pip freeze` command.
`prefix`	If specified, the name to use for grouping cached data instead of the experiment name.
`records`	The list of records currently managed by this manager.
`report_css_path`	The path to a CSS file to copy into each report directory.
`reportables`	The list of all reportables reported from all records.
`reports_path`	The path where experiment run reports are saved.
`reproduction_line`	The CLI command to replicate the experiment this manager is for.
`run_info`	The metadata block associated with this manager from the `ManagerStore`.
`run_line`	The CLI command used to run the current experiment.
`run_timestamp`	The datetime timestamp for when the manager is initialized (and usually also when the experiment starts running.)
`runs_path`	The path where full experiment stores (run with `--store-full`) are kept.
`stage_active`	Flag indicating whether a stage is actively running or not, used to detect if a stage has directly called another stage.
`status`	‘incomplete’, ‘complete’, ‘error’, or ‘LIVE’ if run from a notebook, interactive terminal, or non-experiment script.
`store_full`	Flag for whether to store/copy environment info, log, output report, and all cached files in a run-specific folder.
`stored`	A flag keeping track of if the experiment `store.json` has been updated with this run or not.

Methods:

`display_all_reportables`()	Returns an HTML IPython display all produced reportables.
`display_group_reportables`(group_name)	Returns an HTML IPython display of all reportables in the passed group.
`display_info`()	Returns an IPython HTML display object with the report info block.
`display_record_reportables`(record)	Returns an HTML IPython display of all reportables produced within the passed record.
`display_reportable`(reportable)	Returns an HTML IPython display of the rendered passed reportable.
`display_stage_graph`()	Returns an HTML IPython display of the graphviz SVG map of the records and the stages they were run through.
`display_stage_reportables`(stage_name)	Returns an HTML IPython display of all reportables produced by the given stage.
`generate_report`()	Output report files to a run-specific report folder, the reports/_latest, and the store-full folder if applicable.
`get_all_param_sets`()	This is important to get parameter sets that aren’t obtained through parameter files , e.g.
`get_artifact_filename`(obj_name, record[, …])	Get the non-directory-specific filename of an artifact, following this convention:
`get_artifact_path`(obj_name, record[, …])	Get a record-appropriate full path/filename for a given object name and record.
`get_grouped_reportables`()	Returns a dictionary of reportable groups, each group containing the list of reportables.
`get_reference_name`()	Get the reference name of this run in the experiment registry.
`get_run_output_path`([obj_name])	Get the path for a `--store-full` run folder for this manager.
`get_str_timestamp`()	Convert the manager’s run timestamp into a string representation.
`get_ungrouped_reportables`()	Returns the list of reportables that have no group.
`lock`()	Necessary to avoid file collisions when run in parallel mode.
`map_records`()	Run through every record currently stored and grab the stage and stage i/o list and store it.
`store`()	Update the `ManagerStore` with this manager’s run metadata.
`unlock`()	Necessary to avoid file collisions when run in parallel mode.
`update_map_progress`(record[, update_type])	Update the rich progress bar for the specified record.
`write_run_env_output`()	Write all environment metadata to a run folder for a `--store-full` run.

artifacts: The list of ArtifactRepresentation instances for all artifacts stored in all record states.

cache_path: The path where all intermediate data is cached.

conda_env: The output from a conda env command (conda env export --from-history).

config: The manager configuration loaded from the curifactory config file if present.

current_stage_name: The name of the stage currently executing.

display_all_reportables(): Returns an HTML IPython display all produced reportables.

Note

This only works within an IPython context.

display_group_reportables(group_name: str): Returns an HTML IPython display of all reportables in the passed group.

Note

This only works within an IPython context.

display_info(): Returns an IPython HTML display object with the report info block. This is mostly only useful for making displays in a Juptyer notebook.

display_record_reportables(record: curifactory.record.Record): Returns an HTML IPython display of all reportables produced within the passed record.

Note

This only works within an IPython context.

display_reportable(reportable: curifactory.reporting.Reportable): Returns an HTML IPython display of the rendered passed reportable.

Note

This only works within an IPython context.

display_stage_graph(): Returns an HTML IPython display of the graphviz SVG map of the records and the stages they were run through.

Note

This only works within an IPython context.

display_stage_reportables(stage_name: str): Returns an HTML IPython display of all reportables produced by the given stage.

Note

This only works within an IPython context.

dry: bool: Flag for whether to suppress all file outputs (cached files, logs, registries etc.)

dry_cache: bool: Flag for whether to suppress only writing cached files.

error: The exception class and error string, if one was thrown.

error_thrown: A flag indicating whether an error was thrown by the experiment.

experiment_name: The name of the experiment and/or the prefix used for caching.

experiment_run_number: int: The run counter for experiments with the given name.

generate_report(): Output report files to a run-specific report folder, the reports/_latest, and the store-full folder if applicable.

get_all_param_sets() → list: This is important to get parameter sets that aren’t obtained through parameter files , e.g. in an interactive session.

get_artifact_filename(obj_name: str, record: curifactory.record.Record, prefix: Optional[str] = None, stage_name: Optional[str] = None)

Get the non-directory-specific filename of an artifact, following this convention:

` [prefix]_[parameterset_hash]_[stage name]_[artifact name `

Note

This is used by get_artifact_path, which primarily handles the directory for an artifact. This function was also created to assist with templated cacher paths.

Parameters

obj_name (str) – The name to associate with the object as the last part of the filename.
record (Record) – The record that this object is associated with. (Used to get experiment name, args hash and so on.)
prefix (str) – An optional alternative prefix to the experiment-wide prefix (either the experiment name or custom-specified experiment prefix). This can be used if you want a cached object to work easier across multiple experiments, rather than being experiment specific. WARNING: use with caution, cross-experiment caching can mess with provenance.
stage_name (str) – The stage that produced an artifact. If not supplied, uses the currently executing stage name.

get_artifact_path(obj_name: str, record: curifactory.record.Record, subdir: Optional[str] = None, prefix: Optional[str] = None, stage_name: Optional[str] = None, store: bool = False) → str

Get a record-appropriate full path/filename for a given object name and record.

This is used by the cachers, it automatically handles generating a filename using appropriate experiment name prefixing etc. NOTE: This function sets the record’s args hash if it is None, or if an aggregate stage is involved.

The output path will follow this convention: [base path]/[prefix]_[parameterset hash]_[stage name]_[artifact name], where base path is determined based on the value of store and subdir.

Parameters

obj_name (str) – The name to associate with the object as the last part of the filename.
record (Record) – The record that this object is associated with. (Used to get experiment name, args hash and so on.)
subdir (str) – An optional string of one or more nested subdirectories to prepend to the artifact filepath. This can be used if you want to subdivide cache and run artifacts into logical subsets, e.g. similar to https://towardsdatascience.com/the-importance-of-layered-thinking-in-data-engineering-a09f685edc71.
prefix (str) – An optional alternative prefix to the experiment-wide prefix (either the experiment name or custom-specified experiment prefix). This can be used if you want a cached object to work easier across multiple experiments, rather than being experiment specific. WARNING: use with caution, cross-experiment caching can mess with provenance.
stage_name (str) – The stage that produced an artifact. If not supplied, uses the currently executing stage name.
store (bool) – Set this to true if the path needs to go into a –store-full run folder.

Returns

A string filepath that an object can be written to.

get_grouped_reportables() → dict: Returns a dictionary of reportable groups, each group containing the list of reportables.

get_reference_name() → str

Get the reference name of this run in the experiment registry.

The format for this name is [experiment_name]_[run_number]_[timestamp].

get_run_output_path(obj_name: Optional[str] = None) → str

Get the path for a --store-full run folder for this manager. Similar to get_path, but with an always assumed output=True.

Returns: A string filepath pointing to the runs folder with an optional object name attached at end. (Returns only the folder path if None is passed.)

get_str_timestamp() → str: Convert the manager’s run timestamp into a string representation.

get_ungrouped_reportables() → list: Returns the list of reportables that have no group.

git_commit_hash: The current commit hash if a git repo is in use.

git_workdir_dirty: Whether there are uncommited changes in the git repo or not.

hostname: The hostname of the machine this experiment ran on.

ignore_lazy: Run the experiment disabling any lazy object caching/keeping everything in memory. This can save time when memory is less of an issue.

interactive: Indicates if this manager was spawned with LIVE status in an interactive environment (e.g. a python terminal or Jupyter notebook.)

lazy: If true, attempts to set all stage outputs as Lazy objects. Outputs that do not have. a cacher specified will be given a PickleCacher. Note that objects without a cacher tht do not handle pickle serialization correctly may cause errors.

live_report_path_generated: A flag indicting if the default report’s graphs/reportables folders exist, this helps prevent live displays from breaking if multiple display functions called.

lock(): Necessary to avoid file collisions when run in parallel mode.

logs_path: The path where run logs get stored.

manager_cache_path: The path where the parameter registry and experiment store data are kept.

map_mode: If we’re in map mode, don’t actually execute any stages, we’re only recording the ‘DAG’ (really just the set of stages associated with each record)

map_records(): Run through every record currently stored and grab the stage and stage i/o list and store it. Then clean the records.

mapped_artifacts: The list of MapArtifactRepresentation instances found during the mapping process.

notebooks_path: The path where run notebooks are placed.

notes: User-entered notes associated with a session/run to output into the report etc.

os: The name of the current OS running curifactory.

overwrite: For live session managers where you don’t wish to set overwrite on individual parameter sets, you can universely set the manager to overwrite by changing this flag to True.

overwrite_stages: The list of individual stages for which to ignore the cache.

parallel_lock: If this function is called from a multiprocessing context, use this lock to help prevent files being written to and read simultaneously.

parallel_mode: A flag indicating whether this manager is in a “subproc” run or not. Do not set this manually.

param_file_param_sets: dict

A dictionary of parameter file names for keys, where each value is an array of arrays, each inner array containing the parameter set name and the parameter set hash, for the parameter sets that come from that parameter file. (See the params field in the metadata blocks in ManagerStore.)

e.g. {"my_param_file": [ [ "my_param_set", "44b5e428e7165975a3e4f0d1674dbe5f" ] ] }

parameter_files: list: The list of parameter file names to be used in the experiment.

pip_freeze: The output from a pip freeze command.

prefix: If specified, the name to use for grouping cached data instead of the experiment name.

records: The list of records currently managed by this manager.

report_css_path: The path to a CSS file to copy into each report directory.

reportables: The list of all reportables reported from all records.

reports_path: The path where experiment run reports are saved.

reproduction_line: The CLI command to replicate the experiment this manager is for.

run_info: The metadata block associated with this manager from the ManagerStore.

run_line: The CLI command used to run the current experiment.

run_timestamp: The datetime timestamp for when the manager is initialized (and usually also when the experiment starts running.)

runs_path: The path where full experiment stores (run with --store-full) are kept.

stage_active: Flag indicating whether a stage is actively running or not, used to detect if a stage has directly called another stage. (Technically not disallowed but a big no-no/prohibits any DAG features.)

status

‘incomplete’, ‘complete’, ‘error’, or ‘LIVE’ if run from a notebook, interactive terminal, or non-experiment script.

Type: The current status of the experiment

store(): Update the ManagerStore with this manager’s run metadata.

store_full: bool: Flag for whether to store/copy environment info, log, output report, and all cached files in a run-specific folder.

stored: A flag keeping track of if the experiment store.json has been updated with this run or not.

unlock(): Necessary to avoid file collisions when run in parallel mode.

update_map_progress(record: curifactory.record.Record, update_type: str = ''): Update the rich progress bar for the specified record.

write_run_env_output(): Write all environment metadata to a run folder for a --store-full run.