Reporting

Classes for handling reporting - adding customizable pieces of information to an HTML experiement run report.

This is handled through a base Reportable class, and each reporter class extends it.

Classes:

DFReporter(df[, name, group, float_prec])

Adds an HTML table to the report for the given pandas dataframe.

FigureReporter(fig[, name, group])

Adds a passed matplotlib figure to the report.

HTMLReporter(html_string[, name, group])

Adds the raw string of HTML passed to it to the report.

ImageReporter(image_path[, name, group])

Adds an image to the report from a specified path.

JsonReporter(dictionary[, name, group])

Adds an indented JSON dump in a <pre> tag for a passed dictionary.

LatexTableReporter(df[, name, group])

Output a latex string for a pandas dataframe, useful for copying a table from a report into a latex paper.

LinePlotReporter(y[, x, name, group, ...])

Takes set(s) of data, creates matplotlib line plots for it, and adds to the report.

Reportable([name, group])

The base reporter class, any custom reporter should extend this.

Functions:

map_full_svg(manager[, detailed, colors])

Create a graphviz dot graph for the entire experiment.

map_single_svg(manager, record[, detailed, ...])

Create a graphviz dot graph of the stages/artifacts for the given record.

prepare_report_path(output_path, report_name)

Set up any necessary folders for a report at the given location.

render_graph(graph)

Attempts to return the unicode text for the graph svg.

render_report_all_reportables(manager, ...)

Get the HTML for displaying all reportables output.

render_report_argset_dump(manager)

Dump out the JSON for all args used by manager.

render_report_detailed_stage_maps(manager, ...)

Generate and write out a more detailed graphviz graph for each record and return the html.

render_report_head(manager)

Generates the report head tag.

render_report_info_block(manager)

Generate the header and block of metadata at the top of the report.

render_report_reportables_toc(manager)

Render the table of contents for the reportables.

render_report_stage_map(manager, graphs_path)

Generate and write out the graphviz graph for the stages and return the html to display it.

render_report_toc()

Render table of contents for the overall report.

render_reportable(reportable, manager, ...)

Render a reportable to file and get the HTML to display it.

run_report(manager, output_path, name[, ...])

Generate a full HTML report for the given manager and passed argsets.

update_report_index(experiments_path, ...)

Generate a nice index.html with a summary line for each experiment run report in the passed directory.
class curifactory.reporting.DFReporter(df, name=None, group=None, float_prec=4)

Adds an HTML table to the report for the given pandas dataframe.

Parameters:

  • df (pd.DataFrame) – The pandas dataframe to include in the report.

  • float_prec (int) – the floating point precision to round all values to.

  • name (str)

  • group (str)

Note

If you need to use any of the pandas stylers, apply them and pass df.render() to an HTMLReporter instead.

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.
html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

list[str]

class curifactory.reporting.FigureReporter(fig, name=None, group=None, **kwargs)

Adds a passed matplotlib figure to the report.

Parameters:

  • fig – A matplotlib figure to render.

  • kwargs – All keywords args are passed to the figures savefig() call in render.

  • name (str)

  • group (str)

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.

render()

Any file outputs or calculations that should only run once go here.

Attributes:

kwargs

All keywords args are passed to the figures savefig() call in render.
group: str

If specified, reports group all reportables with the same group value together.

html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

str

kwargs

All keywords args are passed to the figures savefig() call in render.

name: str

if a custom reportable is saving anything in a render function, don’t use just name in the path. self.qualified_name should be preferred, as it is the fully prefixed name.

Type:

The suffix to title the reportable with. NOTE

path: str

Set internally by reporting functions, this variable holds a valid path where a reportable can save files (e.g. images) as needed. This is available to access both in render() and html()

qualified_name: str

The full prefixed name including the stage name and aggregate indicator. This is set by the record when a report() is called.

render()

Any file outputs or calculations that should only run once go here.

rendered: bool

A flag indicating whether this reportable’s render() has been called yet or not.

stage: str

The name of the stage this reportable comes from, used as part of the title.

class curifactory.reporting.HTMLReporter(html_string, name=None, group=None)

Adds the raw string of HTML passed to it to the report.

Parameters:

  • html_string (str) – The raw string of HTML to include.

  • name (str)

  • group (str)

Example

@stage(...)
def report_hello(record: Record ,...):
    record.report(HTMLReporter("<h1>Hello world!</h1>"))

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.

Attributes:

html_string

The raw string of HTML to include.
group: str

If specified, reports group all reportables with the same group value together.

html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

str | list[str]

html_string

The raw string of HTML to include.

name: str

if a custom reportable is saving anything in a render function, don’t use just name in the path. self.qualified_name should be preferred, as it is the fully prefixed name.

Type:

The suffix to title the reportable with. NOTE

path: str

Set internally by reporting functions, this variable holds a valid path where a reportable can save files (e.g. images) as needed. This is available to access both in render() and html()

qualified_name: str

The full prefixed name including the stage name and aggregate indicator. This is set by the record when a report() is called.

rendered: bool

A flag indicating whether this reportable’s render() has been called yet or not.

stage: str

The name of the stage this reportable comes from, used as part of the title.

class curifactory.reporting.ImageReporter(image_path, name=None, group=None)

Adds an image to the report from a specified path.

Parameters:

  • image_path (str) – Path to an image to copy into report folder and display on report.

  • name (str)

  • group (str)

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.

render()

Any file outputs or calculations that should only run once go here.
html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

str

render()

Any file outputs or calculations that should only run once go here.

class curifactory.reporting.JsonReporter(dictionary, name=None, group=None)

Adds an indented JSON dump in a <pre> tag for a passed dictionary.

Parameters:

  • dictionary (Dict) – The python dictionary to write to a JSON string. Technically this can be anything json-serializable, so this includes single values and lists.

  • name (str)

  • group (str)

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.
html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

list[str]

class curifactory.reporting.LatexTableReporter(df, name=None, group=None, **kwargs)

Output a latex string for a pandas dataframe, useful for copying a table from a report into a latex paper.

Parameters:

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.
html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

list[str]

class curifactory.reporting.LinePlotReporter(y, x=None, name=None, group=None, plot_kwargs={}, savefig_kwargs={})

Takes set(s) of data, creates matplotlib line plots for it, and adds to the report.

There are several ways to use this reportable, x (optional) and y are both either a single list/numpy array of data, or are both dictionaries of lists/numpy arrays of data to plot.

Possible combinations:

  • Passing a single list/numpy array of only y data

  • Passing a single list/numpy array of y and x data

  • Passing a dictionary (where keys appear in the legend) of lists/numpy arrays of y data, each to be plotted as its own line.

  • Passing a dictionary (where keys appear in the legend) of lists/numpy arrays of both y and x data, each to be plotted as its own line.

Parameters:

  • y – a single list/numpy array or dictionary of lists/numpy arrays of y data.

  • x – (optional), a single list/numpy array or dictionary of lists/numpy arrays of x data. If specified, this must match y.

  • plot_kwargs (Dict) – The **kwargs to pass to the matplotlib plt.plot() call.

  • savefig_kwargs (Dict) – The **kwargs to pass to the fig.savefig() call on render.

  • name (str)

  • group (str)

Example

@stage(...)
def report_results(record: Record, ...):

    train_loss = [0.8, 0.7, 0.6]
    test_loss = [0.5, 0.3, 0.4]
    acc = [.93, 0.95, 0.96]

    record.report(LinePlotReporter(
        y={
            "Training loss": train_loss,
            "Testing loss": test_loss,
            "Accuracy": acc
        },
        name="training_history"))

The above block will result in the following graph added to the report:

api/line_plot_reporter_example.png

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.

render()

Any file outputs or calculations that should only run once go here.
html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

str

render()

Any file outputs or calculations that should only run once go here.

class curifactory.reporting.Reportable(name=None, group=None)

The base reporter class, any custom reporter should extend this.

Parameters:

  • name (str) – A (optional) reference name to give this piece of reported info, it is used as the title string suffix. If None is supplied, it will be suffixed with the number of the reportable.

  • group (str) – An optional string to use for grouping multiple related reportables together in the report. By default, all reportables are ordered by record. This will create a separate entry on the TOC and put them next to each other in the report.

Note

When subclassing a reportable, html() must be overriden, and render() optionally may be depending on the nature of the reportable. If a reportable relies on some form of external file, such as an image or figure, implement render() to save it (using this class’s path variable as the directory), and then reference it in the output from html(). The internal reporting mechanisms handle calling both of these functions as needed.

A simplified example of the FigureReporter is shown here:

class FigureReporter(Reportable):
    def __init__(self, fig, name=None, group=None):
        self.fig = fig
        super().__init__(name=name, group=group)

    def render(self):
        self.fig.savefig(os.path.join(self.path, f"{self.qualified_name}.png"))

    def html(self):
        return f"<img src='{self.path}/{self.qualified_name}.png'>"

Attributes:

group

If specified, reports group all reportables with the same group value together.

name

if a custom reportable is saving anything in a render function, don't use just name in the path.

path

Set internally by reporting functions, this variable holds a valid path where a reportable can save files (e.g. images) as needed.

qualified_name

The full prefixed name including the stage name and aggregate indicator.

record

The record this reportable came from, automatically populated via record.report()

rendered

A flag indicating whether this reportable's render() has been called yet or not.

stage

The name of the stage this reportable comes from, used as part of the title.

Methods:

html()

When a report is created, the html() function for every reportable is called and appended to the report.

render()

Any file outputs or calculations that should only run once go here.
group: str

If specified, reports group all reportables with the same group value together.

html()

When a report is created, the html() function for every reportable is called and appended to the report. This function should either return a single string of html, or can return a list of lines of html.

Note

Any subclass is required to implement this.

Return type:

str | list[str]

name: str

if a custom reportable is saving anything in a render function, don’t use just name in the path. self.qualified_name should be preferred, as it is the fully prefixed name.

Type:

The suffix to title the reportable with. NOTE

path: str

Set internally by reporting functions, this variable holds a valid path where a reportable can save files (e.g. images) as needed. This is available to access both in render() and html()

qualified_name: str

The full prefixed name including the stage name and aggregate indicator. This is set by the record when a report() is called.

record

The record this reportable came from, automatically populated via record.report()

Type:

Record

render()

Any file outputs or calculations that should only run once go here.

rendered: bool

A flag indicating whether this reportable’s render() has been called yet or not.

stage: str

The name of the stage this reportable comes from, used as part of the title.

curifactory.reporting.map_full_svg(manager, detailed=False, colors=None)

Create a graphviz dot graph for the entire experiment. (Maps out each stage and the input/output artifacts for each.)

Parameters:

  • manager (ArtifactManager) – the manager to base the graph off of.

  • detailed (bool) – Whether to include artifact preview and caching information or not.

  • colors (list[str] | dict[str, str]) – A list or dictionary of colors to use if you wish to override the default.

Return type:

Digraph

Important

For this function to run successfully, graphviz must be installed.

curifactory.reporting.map_single_svg(manager, record, detailed=True, colors=None)

Create a graphviz dot graph of the stages/artifacts for the given record.

Parameters:

  • manager (ArtifactManager) – the manager to base the graph off of.

  • record (Record) – The specific record to make the graph for.

  • detailed (bool) – Whether to include artifact preview and caching information or not.

  • colors (list[str] | dict[str, str]) – A list or dictionary of colors to use if you wish to override the default.

Return type:

Digraph

Important

For this function to run successfully, graphviz must be installed.

curifactory.reporting.prepare_report_path(output_path, report_name)

Set up any necessary folders for a report at the given location. This will not error if the location already has a report in it, but will remove existing reportables and graphs.

Returns:

A tuple containing the folder path to place the report html, the subdirectory within that folder to place the graphviz files, and the subdirectory to place any rendered reportable files.

Parameters:

  • output_path (str)

  • report_name (str)

Return type:

tuple[str, str, str]

curifactory.reporting.render_graph(graph)

Attempts to return the unicode text for the graph svg.

curifactory.reporting.render_report_all_reportables(manager, reportables_path, override_display_path=None, notebook=False)

Get the HTML for displaying all reportables output.

Parameters:

  • reportables_path (str)

  • override_display_path (str)

  • notebook (bool)

Return type:

list[str]

curifactory.reporting.render_report_argset_dump(manager)

Dump out the JSON for all args used by manager.

Return type:

list[str]

curifactory.reporting.render_report_detailed_stage_maps(manager, graphs_path)

Generate and write out a more detailed graphviz graph for each record and return the html.

Return type:

list[str]

curifactory.reporting.render_report_head(manager)

Generates the report head tag.

Return type:

list[str]

curifactory.reporting.render_report_info_block(manager)

Generate the header and block of metadata at the top of the report.

Return type:

list[str]

curifactory.reporting.render_report_reportables_toc(manager)

Render the table of contents for the reportables.

Return type:

list[str]

curifactory.reporting.render_report_stage_map(manager, graphs_path)

Generate and write out the graphviz graph for the stages and return the html to display it.

Return type:

list[str]

curifactory.reporting.render_report_toc()

Render table of contents for the overall report.

Return type:

list[str]

curifactory.reporting.render_reportable(reportable, manager, reportables_path, override_display_path=None, notebook=False)

Render a reportable to file and get the HTML to display it.

Parameters:

  • reportable (Reportable)

  • reportables_path (str)

  • override_display_path (str)

  • notebook (bool)

Return type:

list[str]

curifactory.reporting.run_report(manager, output_path, name, css_path=None)

Generate a full HTML report for the given manager and passed argsets.

Parameters:

  • manager (ArtifactManager) – The manager to base the graph off of.

  • output_path (str) – The string path to the root directory of where you want the report stored.

  • name (str) – The name to store this report under (will generate a folder of this name in the output_path.

  • css_path (str) – The path to a css file to use for styling the report. (This file will get copied into the output_path/name folder.)

Returns:

A tuple containing the folder path to place the report html, the subdirectory within that folder to place the graphviz files, and the subdirectory to place any rendered reportable files.

Return type:

tuple[str, str, str]

curifactory.reporting.update_report_index(experiments_path, reports_root_dir)

Generate a nice index.html with a summary line for each experiment run report in the passed directory.

Parameters:

  • experiments_path (str) – The path to where the experiment scripts are, used to get any top experiment description comments from the file.

  • reports_root_dir (str) – The directory containing the report folders. This is where the output index.html is placed.