Analysis Tools

Overview

The following describes Optimizer Studio functionality for analyzing and gaining insights into a particular knob configuration.

Analysis Actions

All of the actions can be found within the optimizer-ctl utility by typing:

$ optimizer-ctl --help

The available analysis actions are:

1. analyze
2. sample
3. refine

These actions work by assessing data from individual configurations within an experiment.

To access the analysis commands within Conductor, open the experiment and select the desired data point. In the panel to the right of the experiment select the Data Point menu and then select the Actions tab. Within this tab each of the analysis actions can be accessed by a corresponding button.
Each button will generate a pop-up window containing the optimizer-ctl command corresponding to the selected configuration.
Note that not all the command options are listed when copying the command from the pop-up window.

TO VIEW the results of an analysis action, select the desired data point. In the right panel select the Data Point menu and then select the Sensitivty tab.

An analysis action has to be invoked while the optimization still progresses. In case the experiment has completed, retain Optimizer Studio operation in Idle mode, so that it is able to respond to optimizer-ctl requests:

$ optimizer-ctl resume --id {experimentID}
$ optimizer-ctl start

Invoke optimizer-ctl <action> [argument-1 [argument-2 [...]]] in a separate console.

Analyze

This action is able to study a particular knob's impact on a given configuration, by traversing all of the knob's options. If a knob is not specified all of the options of every knob wil be explored within the given configuration. See Analysis Actions section for instructions on how to invoke the following action:

$ optimizer-ctl analyze --config_url <URL> [--knob <name>]

where
--knob <name> - the name of the knob whose options will be tested,
if not specified all knobs and options will be tested

Once this command has been generated it can be invoked from a parrallel terminal to the terminal already running an experiment.

Sample

This action performs additional sampling of a configuration N times. This is done in order to improve the statistical quality of the target metrics value. This is can be significant for experiments with a high level of statistical variance. By running sample users are able to affirm the values of their optimizations by running the additional sampling. See Analysis Actions section for instructions on how to invoke the following action:

$ optimizer-ctl sample --config_url <URL> [--times <N>]

where
--times <N> - the number of samples required, 1 if not specified

Once this command has been generated it can be invoked from a parrallel terminal to the terminal already running an experiment.

Refine

This action is able to analyze which of the knobs of the given configuration can be switched to baseline without negative impact to the performance. This can be useful for determining which knobs have a larger effect on the final optimization levels. Allowing for slimmer testing with fewer knobs in future iterations of testing. See Analysis Actions section for instructions on how to invoke the following action:

$ optimizer-ctl refine --config_url <URL> 

NOTE: The results of the refine command cannot yet be viewed in Conductor. For results please allow refine to finish and then the results will be viewable within the resultant logfile.

Restarting The Experiment For Analysis

Once the type of analysis has been selected the desired experiment must be restarted. The proper way to restart an experiment is by editing the run.sh script in the experiment directory. To do this comment out any lines containg the optimizer-studio then insert the following line:

$ optimizer-ctl resume --id {experimentID}
$ optimizer-ctl start

The --retain option will continue the last stopped experiment The --stages idle option will ensure that only the desired analysis takes place

Once the optimizer studio is running invoke the desired analysis command in the parrallel console as follows:

$ optimizer-ctl <action> --config_url <URL> [argument-1 [argument-2 [...]]

This will automatically start the desired analysis and the experiment can be left to run on its own.

optimizer-ctl MUST be started before executing any of the analysis commands

Analysis action queue

In case an action is invoked while the previous action is still being executed, it will be queued.
The status of the currently queued action(s) (Pending, Running, Completed) can be retrieved by calling optimizer-ctl status.
A pending action can be revoked, if desired, by calling optimizer-ctl revoke <action-ID>, where action-ID is listed by the status call above.

The Sensitivity Tab

This tab will display all of the corresponding configurations generated by any analysis action. As well the tab will display the adjusted knob, the knobs new value, and the resultant of the optimization point. Within the list of configurations, and when hovering over the configuration a '3 dots' menu will appear at the right side of the configurations line. This menu is identical to the menu in the actions tab, and when running the corresponding actions they will be for the selected configuration. If Analyze Knobs is selected in this menu it will have the additional --knob option in the generated command, where the --knob being named is the knob from the selected configuration.

See also

• Retain