Skip to content

Experiment Workflow

Overview

The Experiment Workflow consists of several stages, which can be selected via --stages command line switch, where the desired stages can be listed, separated by comma, e.g. --stages optimization,refinement.

The available workflow stages are:

  1. readiness
  2. optimization [default]
  3. refinement
  4. validation

The stages are pre-ordered irrespective of the order of appearance on the command line, yet the user is free to select which stages should be executed. Some stages require additional configuration parameters in the knob file. Some stages can be selected only when permitted by the license.

It is possible to revisit the workflow of an ongoing experiment. Stop the experiment, then restart it (with --retain command line switch) with only stages of interest selected.

Readiness Stage

The Readines Stage purpose is finding out the correct optimization parameters, before running the optimization process.

Optimizer Studio runs the workload several times using baseline knob configuration, and finds out the approximate time of running a workload, as well as statistical characteristics of the target metric for the baseline configuration - mean value, standard deviation, coefficient of variation (CV), etc.

After this stage, the user can set optimization parameters with more confidence.

The results of the Readiness stage are reported to Experiment Management System.

Readiness Stage Configuration

The following section of a knob file is used for configuration:

stages:
  readiness:
    min_samples: 10
    duration: 15m

The Readiness stage will run until min_samples are acquired or duration has passed, the earlier.

duration values should have the syntax: AhBmCs, where A, B, C are the values, and h, m, s mean "hours", "minutes" or "seconds" correspondingly. For example: 1h30m, 15m, etc.

Optimization Stage

Optimization stage comprises the main optimization process by running the workload and testing different knob configurations. This is the main and default functionality of Optimizer Studio.

Optimization Stage Configuration

The following section of a knob file is used for configuration:

stages:
  optimization:
    min_configs: 30
    timeout: 15m
    configs_without_improvement: 10

Or

stages:
  optimization:
    duration: 15m

The Optimization stage will exercise min_configs (if provided) no matter the progress, but not pass the timeout (if provided).
After testing N configurations, the Optimization stage will run to completion (progress = 100%) or timeout (if provided).
If configs_without_improvement provided, the Optimization stage is considered completed if no improvement is observed for M consecutive configurations or progress reaches 100%.

duration is incompatible with other options, and sets the precise optimization duration.

Refinement Stage

The Refinement Stage runs after the Optimization Stage completes.
The Refinement is applied to the best configuration dicovered throughout the previous stages, and strives to minimize the number of knobs which differ from their baseline values.

It is possible to stop Optimizer Studio, and restart it (with --retain command line switch) with only refinement stage selected. In this case, Optimizer Studio will proceed to refining the best configuration dicovered till this point.

Refinement Stage Configuration

The Refinement stage can be configured via the knob file:

stages:
  refinement:
    std_tolerance: N     # default: 1.0
    improv_tolerance: P  # default: 0.01%

The purpose of the Refinement stage is to identify (and return to baseline) the knobs that do not contribute to the improvement. However, in a noisy environment it is difficult to tell a small improvement from a random noise. The parameters above are used to set this sensitivity limit:

  • std_tolerance - two measurements are considered distinct if the difference between them is more than N times the standard deviation
  • improv_tolerance - an improvement below P% is considered noise

In a completely quiet environment, it makes sense to set improv_tolerance: 0.

Validation Stage

The purpose of the Validation stage is to assess the statistical significance of the best performance achieved throughout the optimization process.
The best configuration is attempted N times while statistical information is collected, such as mean value, standard deviation, coefficient of variation (CV).
In case there are not enough baseline samples for the statistical significance assessment, the additional baseline samples will be made.

Validation Stage Configuration

The following section of a knob file is used for configuration:

stages:
  validation:
    min_samples: 10
    duration: 15m