Overview¶

The configuration files of Optimizer Runtime are located in the installation directory /opt/concertio-optimizer/runtime/. The default location can be overridden via command line switches.
There are two configuration files:

knobs.yaml - Specifies the knobs, input metrics and target function of Optimizer Runtime.
settings.yaml - Specifies the system-wide settings of Optimizer Runtime.

Optimization Target¶

Users can configure Optimizer Runtime to search for the optimal parameters that maximize (or minimize) a specific Target Metric.
The target metric can be defined as follows:

domain:
  common:
    target:
      name: performance
      goal: max

or in shorthand:

domain:
  common:
    target: performance:max

The target metric name can refer any of the sampled metrics, such as proc.diskstats.sda.sectors_written. The metrics definition is further detailed below.
The optimization goal is set to max, if omitted.

Predefined Optimization Target¶

While target metric can refer any of the sampled metrics, Optimizer-Runtime supports a list of predefined targets:

Name	Default Goal	Meaning
duration	min	Total workload execution time.
performance	max	Total number of retired instructions (msr.inst_retired_all) measured in all CPU cores per second.
net.throughput	max	Total number of bytes transferred over the network per second.

knobs.yaml: knob definition¶

Knobs are tunable parameters of the system which Optimizer Runtime will try to change in order to optimize the system.

Concertio Optimizer Runtime ships with embedded set of knob definitions containing many knobs that were tested and benchmarked by Concertio engineers. In order to use these knobs, specify --knobs=embedded command line switch.

It is possible to provide additional knob files by using the --knobs=/path/to/knob-file.yaml command line switch, where you can add custom knobs that are relevant for your system. If you don't supply your knobs file, the embedded knobs will be used automatically.

Optimizer Runtime can accept any number of --knobs=PATH command line switches. In such case Optimizer Runtime loads knob files in the order they appear in the command line. This way a knob files appearing later can overload any settings in previous knob files. Note that this doesn't include embedded knobs which are always processed first.

knobs section: adding a Knob¶

It is possible to add a new or override an existing knob by adding a knob section to the knobs list. Each knob section contains a few values and POSIX shell scripts used for defining and manipulating the knob.

Knob kinds¶

Each knob belongs to one of the supported knob kinds: builtin, scripted, accelerator.
The main difference between the knob kinds is the way the knob is applied.
Below is a list of the supported knob kinds:

builtin knobs
Knobs built into Optimizer Runtime do not need to be defined by the user, but might require configuration parameters.
scripted knobs
The scripted knob is applied via executing a POSIX shell script with $KNOB_VAL, $KNOB_DEV environment variables.
- kind: script
- get_script - returns the current knob value, that will be set as baseline
- set_script - set new knob value
- pre_set_script - when provided, this script is run prior to running any of the set_script(s)
  Note that in case more than one knob refers to the same pre_set_script, identical script invocations will be merged.
- post_set_script - when provided, this script is run after running any of the set_script(s)
  Note that in case more than one knob refers to the same post_set_script, identical script invocations will be merged.
- device_script - when provided, returns the list of applicable devices, and the knob is split into multiple per-device knobs, e.g.
  io.scheduler → io.scheduler.sda, io.scheduler.sdb, io.scheduler.sdc
- Environment variables observable by the knob scripts:
  $KNOB_VAL - knob value being set by set_script
  $KNOB_DEV - device name, in device-specific knobs, as returned by device_script
accelerator knobs
The accelerator knobs encompass all the knob kinds passed directly to Concertio accelerator.
- kind: accel.mem_alloc - provides a selection of malloc implementations

General knob parameters¶

Below is a list of the general purpose parameters supported by all the knob kinds:

kind - knob kind. When omitted, Optimizer software will attempt to identify the knob kind by presence or absence of kind-specific parameters.
enable_script - when provided, a 0 / false indicate that the knob should be discarded, any other value enables the knob. When not provided, the knob is enabled by default.
disable - when provided, the knob will be discarded and erased from internal data structures. This parameter overrides enable_script.
options - a list of available knob options. Options definition is detailed below.
default - when provided, sets the baseline to the value provided. This option is mutually exclusive with get_script.
Notice, that when neither default nor get_script are specified, the 1-st option in the options list will be set as baseline value.
set_policy - knob setting policy:
- on_change (default) - each time the internal algorithm needs to change a knob, its value is applied.
- once:<value> - the value is applied in the beginning of the run, and is reverted to baseline in the end.
- always - the current knob value is applied on each sample.
Once and always setting policies do not participate in the optimization process, and can be considered auxiliary utilities.

Knob options definition¶

The simplest way to define knob options is to list all valid values explicitly:

options:
  values: [10, 15, 20]
default: 15

The second way is to define an options script which returns the list of option values. For example, the following script creates a list of numbers from 0 to 7 for a computer with 8 logical CPUs

options:
  script: cat /proc/cpuinfo | awk '/processor/ {print $3}'

Yet another possibility is to define a list of options using numeric range:

options:
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d
default: 8

This definition creates a list of options "--num-threads=1", --num-threads=2, etc.
Printf style format field is optional. It contains a string with one of supported format specifiers, like in the example:

Format	Numeric Value	Knob Value
"-O%d"	2	-O2
"--pi=%.2f"	3.14159	--pi=3.14
"--pi=%.2f"	3	--pi=3.00
"%g"	5000000	5000000
"%g"	2.718000	2.718
"%x"	2047	7ff
"%X"	2047	7FF

You can combine different ways of specification:

options:
  values: [ "" ]
  range:
    min: 1
    max: 16
    step: 1
    format: --num-threads=%d

The example above adds an empty option value in addition to the ones defined by the range.

Disabling a knob¶

It is possible to disable a knob, either temporarily or permanently, without actually removing its definition from the knob file.
To temporarily disable a knob, add to the knob section the following value:

skip_tuning: ""

To permanently erase a knob from the internal data structures, either add to the knob section the following value:

disable: ""

or define an appropriate enable_script:

enable_script: "echo -n 0"

Note that this approach allows disabling embedded knobs as well.

Examples¶

A: vm.dirty_background_ratio¶

domain:
  common:
    knobs:
      sys.vm.dirty_background_ratio:
        kind: script
        description: "The number of pages at which the pdflush threads begin writeback of dirty data."
        get_script: "/sbin/sysctl -n vm.dirty_background_ratio"
        set_script: "/sbin/sysctl -w vm.dirty_background_ratio=$KNOB_VAL"
        options: [10, 15, 20]

B: io.scheduler¶

Below is a more complex knob example of a device-specific knob. Eventually, this knob definition produces multiple knobs: io.scheduler.sda, io.scheduler.sdb etc.

domain:
  common:
    knobs:
      io.scheduler:
        kind: script
        description: "Block device scheduling algorithm selector."
        get_script: "cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/.*\\[\\([^]]*\\)\\].*/\\1/g'"
        set_script: "echo $KNOB_VAL > /sys/block/$KNOB_DEV/queue/scheduler"
        device_script: "ls -d /sys/block/sd* | cut -d/ -f 4"
        options:
          script: cat /sys/block/$KNOB_DEV/queue/scheduler | sed 's/\[\|\]//g ; s/ $//g ; s/\s/\n/g'

metrics section: named HW and SW metrics¶

Metrics are used by Optimizer Runtime to learn about the system behavior and to detect different phases of execution. Optimizer Runtime will then attempt to find an optimal knob configuration that maximizes a certain sampled metric for each phase. The metrics are sampled periodically.

Metrics definition¶

Comma separated regular expressions define which metrics are sampled, and which metrics are excluded.

domain:
  common:
    include_metrics: [msr.*, proc.*]
    exclude_metrics: [proc.diskstats.sda.sectors_written]

In the above example, all msr metrics and all proc metrics, except for proc.diskstats.sda.sectors_written will be considered by Optimizer Runtime for learning about the system behavior.

User-defined metrics¶

Optimizer Runtime supports user-developed plugins for sampling custom metrics.

Categorized user-defined metrics¶

In version 1.1.0 of optimizer-runtime, a new feature allows the possibility to define metrics which will serve as categorial for workload classification. To make it more comfortable, optimizer supports metrics which return a string rather than only numeric value. This allows for more human-readable category based metrics to be defined.

Following is syntax for declaring such metrics as categorizing metrics using an array for optimizer-runtime to classify category based workloads:

domain:
  common:
    metrics:
      load_avg:
      ...
    categorization_metrics:
    - load_avg

settings.yaml: system-wide settings¶

This file contains runtime settings such as optimization target function, sampling rate, output directory, etc.

Output directory¶

Concertio Optimizer Runtime generates output data such as optimization database file, log files, etc. The default location of all output data is /root/.concertio. The default location can be overridden as follows:

out_directory: "/new/output/directory"

Importing In-Situ Optimizer-Studio Knobs¶

If you ran an in-situ experiment using Optimizer Studio and you wish to import the best settings of this experiment into your Optimizer Runtime environment, you can use the import_accel_knobs command of the optimizer-ctl tool.

This command receives a file path to a json report of the relevant experiment (should be available in your Optimizer Studio output directory) as well as a file path for your target YAML configuration file which will be created.

For example:

$ optimizer-ctl import_accel_knobs --report ~/.concertio/report_2021-02-01_11-09-28_1014287.json --dest insitu.yaml

This will create the configuration file insitu.yaml using the results in ~/.concertio/report_2021-02-01_11-09-28_1014287.json.

Depending on the configuration used in the Optimizer Studio experiment, the content of the output file (insitu.yaml) may look like:

domain:
  common:
    knobs:
      memory:
        kind: accel.mem_alloc
        options:
          - ""
          - mimalloc
        set_policy:
          once: mimalloc

Note: All the knobs in the output configuration file will use set_policy of once. This is because these knobs cannot be changened during the entire session of Optimizer Runtime.

You should use the generated configuration file as either a base for your Optimizer Runtime configuration or you may import this file from your already available configuration file.

Available Setting Parameters¶

Parameter name	Default value	Description
sampling_interval	1s	The interval between samples. Relevant only for asynchronous sampling mode.
knob_ranking_max_num_of_samples	10000	Maximum number of configurations to use for knob ranking calculations.
max_baseline_mean_cv	0.04	The maximum allowed coefficient of variation of the mean of the measurements in baseline settings. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_config_mean_cv	0.04	The maximum allowed coefficient of variation of the mean of the measurements per knob configuration. Lower values imply a stricter convergence threshold, so additional measurements might be required in order to converge.
max_configs_in_report	10	Maximum number of knob configurations to include into best configurations report
max_invalid_samples_per_config	0	The maximum allowed invalid measurements per knob configuration, above which the configuration is considered invalid and will not be further tested.
max_samples_per_config	120	Optimizer will not test any knob configuration more than the number of times specified by this parameter.
metrics_csv_filename	-	If specified, Optimizer Runtime creates a CSV file with the details of all the knob settings and metric measurements.
min_baseline_samples	2	The minimum number of baseline samples. This is used in conjunction with `max_baseline_mean_cv`.
min_samples_per_config	2	The minimum number of samples per knob configuration. This is used in conjunction with `max_config_mean_cv`.
optimization_strategy	evolution	The algorithm employed for searching through the knobs. Available options are greedy and evolution.
optimization_strategy_settings		Settings specific for each optimization strategy. Only settings specific for the selected strategy will be parsed.
out_directory	${HOME}/.concertio	Concertio Optimizer Runtime generates output data such as optimization database file, log files, etc. into this location.
pending_config_timeout	0	Configuration attempt scheduling policy: 0 - attempt configs sequentially, above 0 (in minutes) - attempt configs interleaved with a timeout.
point_estimator	average: <no_value>	Point estimation function. Additional functions: percentile: <percent>, mode: <no_value>
save_interval	120m	The interval between saving the data file to the disk
shell_command	/bin/sh +e	This defines the backend shell of the knobs and metrics. It is possible to run all knobs and metrics scripts on remote hosts using a different shell command
max_idle_time	10m	maximum time for optimizer service to live without any connection from clients after this time the service exits. default value if not set in settings.yaml is 0 meaning no timeout

Deprecated Setting Parameters¶

If you get a parameter deprecation warning when running Optimizer Runtime, please replace the deprecated parameters with new ones in your YAML files, adjusting the parameter values accordingly.

Note that the old parameters still work as before. However they will be dropped in the future versions.

Old parameter	New parameter	Change	Example
interval_seconds	sampling_interval	Time units	1s
save_interval_minutes	save_interval	Time units	2h
pending_config_timeout_minutes	pending_config_timeout	Time units	30m
max_idle_time_minutes	max_idle_time	Time units	10m