Introduction

What is Optimizer Studio

Optimizer Studio is a framework of tools and services that together are used for managing and conducting automated configuration tuning experiments in a scientific manner.

Together with the Experiment Management System, Concertio provides a holistic approach to conduct performance tuning experiments.

Features

The framework provides the necessary tools for automating the experiment sampling process, with emphasis on the following three pillars:

1. Statistically derived inference
2. Fast search in the options space - using AI techniques such as reinforcement learning as well as evolution algorithms
3. Continuous Optimization (CO) - via an API for integration with CICD and APM tools

The framework provides an opinionated experimentation workflow methodology, which you can adopt fully or partially as you find convenient for your use case.

The methodology somewhat resembles CICD pipelines, as it is divided into four stages, depicted in the following diagram, taken from the Experiment Management System:

As a user, you can choose which of the stages gets executed in your experiment.

Each stage has its unique contribution to assure the three pillars mentioned above are met.

Let's dig a bit further into each stage to explain their necessity in the optimization workflow:

Readiness

The readiness stage is responsible for testing the environment in which the experiment samples are being taken, without changing the configuration at all. The goal is to determine the target performance metric(s) behavior, as well as the workload validity. For instance, the readiness stage might provide you with clues and recommendations about the system being “noisy” - with high variability, or if the workload is not properly executing, or the tuning knobs are not all well defined nor functional.

Optimization

This is the main stage in which Optimizer Studio runs the feedback loop in tuning mode, where it explores different configurations for the tuning knobs, applying them to the system directly or through your workload, collecting the metrics and trying to find the optimal configuration, using Concertio’s patented algorithms. During that stage, Optimizer Studio reports the samples with all their relevant meta-data, to the Experiment Management System, so you could review and analyze the experiment progress visually, as well as report it to the console and logs on the node where it is executed. The outcome of this stage is the recommended configuration to accomplish the desired performance improvement, accompanied by a script to apply them where applicable.

Refinement

This stage is a direct continuation of the Optimization stage, as it’s role is to filter and refine the recommendation output of Optimization stage, using Concertio’s patent-pending technology for refinement with as few additional samples as possible. The outcome of this stage is usually a new recommended configuration, but with fewer settings that differ from baseline. This stage is available in base, pro and enterprise editions.

Validation

In the validation stage, Optimizer Studio runs statistical tests (e.g. student t-test) on the recommended configuration (best configuration) vs the baseline configuration (default configuration) to determine the overall significance of the findings. It provides a statistical test that helps in determining whether the improvement is beyond doubt, statistically significant and at what significance level (P value).

For more details about how to configure and run Optimizer Studio with the different stages, see the next section in documentation.

Components

The Optimizer Studio and the Conductor ecosystem consists of multiple components:

1. Optimizer-Studio - The CLI tool that orchestrates the experiment, runs the Optimizer Studio Service, and reports back the status to the console.
2. Optimizer-ctl - A CLI tool which sets up the environment, asynchronously communicates with Optimizer-Studio and the Optimizer Studio Service - Conceptually similar to Docker client or kubectl in Kubernetes.
3. Optimizer Studio Service - The core component that includes the algorithm engine and the reporting to Conductor.
4. Conductor - Experiment Management system - SaaS or on-premise set of services to manage, monitor and store experiments via a web UI.

Component Architecture

The Conductor management system is provided as a SaaS service for community (free!), base and pro plan users.

For Enterprise customers, Conductor can be installed on-premise along side optimizer-studio package.

Find out more about our plans here