Using the scripts in this tarball it is possible to run the following
automated benchmarks:
Sysbench:
---------
Assumes use of the sysbench-0.4.12.5 tarball for MySQL benchmarks
1) Single MySQL Server running InnoDB or other storage engine
2) Single MySQL Server running NDB storage engine with one or many NDB Data nodes
Sysbench can only operate with 1 server instance

DBT2:
-----
1) Single MySQL Server running InnoDB
2) One or multiple MySQL Servers running NDB with one or multiple MySQL Servers

flexAsynch:
-----------
This test uses one or multiple NDB Data nodes and one or multiple NDB API nodes running
flexAsynch. Requires use of MySQL Cluster 7.2.8 and later.

More information about parameters and how to set-up tests can also be found in
examples/autobench.conf which has a set of comments.

The scripts contain code to handle running sysbench and DBT2 with a single
MySQL Server where this server runs on Windows. This can only be done with
source-code tarballs. It does also require careful set-up of a SSH server
with proper privileges on the Windows server. Also this is much less tested
compared to using in particular Linux.

Similarly there is some code to support Drizzle Servers instead of MySQL
Servers, this code is however designed for older Drizzle releases so might
need an update of the code to handle more recent Drizzle releases.

To set-up the infrastructure for running the tests do the following steps:
1) For each server involved one needs to install all build tools and
libraries needed to build and run MySQL, sysbench and dbt2. (e.g. gcc,
CMake).
2) Create a directory for the test to run in (hereafter called DEFAULT_DIR)
3) Place a autobench.conf file in this directory, an example file that
can be used as template is  found in examples/autobench.conf
4) Edit the autobench.conf appropriately
5) Create a directory to place the tarballs in
6) Place a MySQL tarball, this DBT2 tarball and a sysbench tarball
in this tarballs directory and ensure that this directory is referenced
correctly in autobench.conf
7) Execute the script bench_prepare.sh found in the scripts directory as:
./bench_prepare.sh --default-directory DEFAULT_DIR
8) Wait for benchmark to complete
9) Print file final_result.txt in the DEFAULT_DIR for sysbench and
flexAsynch and look for final result of DBT2 in dbt2_results directory
in DEFAULT_DIR.

In the case one runs using Drizzle instead of MySQL one needs to place
a tarball for Drizzle and libdrizzle in the tarballs directory. The
sysbench tarball must be a special drizzle-sysbench tarball. To run
Drizzle set USE_DRIZZLE="y" in autobench.conf.

The automated benchmark is executed on one or two machines, in the case of
NDB benchmarks there can be more nodes running benchmarks, MySQL Servers
and NDB Data nodes. The benchmark process is always executed on the machine
that the above script bench_prepare.sh is started on (except for flexAsynch).
The MySQL/Drizzle server is either executed on the same machine or on a
different server. In the case of a different server it's necessary to get
the binaries also on the remote server. This can be done either by copying
binaries built on the benchmark machine or by building on the remote host.
If BUILD_REMOTE="y" then the script will build the MySQL/Drizzle code on
the remote host.

If the benchmark is to be executed on Windows, only the MySQL Server can be
on Windows, the benchmark machine must still be on a Unix machine. Thus
Windows benchmarks are always using remote servers and remote builds.
For Windows benchmarks thus WINDOWS_REMOTE="y" needs to be set. There
is currently no support for running flexAsynch on Windows servers.

It is also possible to run benchmarks using MySQL Cluster with several
MySQL Servers and several data nodes. Check examples/autobench.conf for
a little bit of information of how to set this up. Currently only DBT2
supports running with multiple drivers towards multiple MySQL Servers
in the benchmark. flexAsynch uses one or multiple NDB API nodes and
one or several NDB Data nodes. There is no need for a MySQL Server in
this test.

Some notes on the variables to assign in autobench.conf:
All variables that can be lists use ; as separator.

1) BENCHMARK_TO_RUN is either sysbench, dbt2 or flexAsynch.
2) TARBALL_DIR is the tarball directory created in 5) above
3) The tarballs placed in the tarball directory should all be
gzipped tarballs, thus MYSQL_VERSION.tar.gz is the name of the
tarball placed in the tarballs directory, when expanded the
tarballs top directory should also be named MYSQL_VERSION.
Similarly for DRIZZLE_VERSION, LIBDRIZZLE_VERSION, SYSBENCH_VERSION
and DBT2_VERSION.
4) SERVER_HOST is the machine where the MySQL/Drizzle Server will run,
localhost or 127.0.0.1 if it's on the same machine. If the SERVER_HOST
is localhost or 127.0.0.1 one doesn't need ssh to run these scripts,
otherwise ssh is needed. Can be a list variable.
5) SERVER_PORT is the port used to communicate to the MySQL/Drizzle
Server. Can be list variable.
6) NDB_MGMD_NODES is a list of the NDB management nodes when NDB
storage engine is used in benchmark.
7) NDBD_NODES is a list of the NDB Data nodes when NDB storage engine
is used in the benchmark.
8) FLEX_ASYNCH_API_NODES is a list of the hosts where NDB API nodes
are placed when running flexAsynch benchmark.
9) DATA_DIR_BASE is the directory under which a directory var_1 will
be created and under here the MySQL/Drizzle Server files are placed,
thus for IO intensive workloads this needs to be a RAID:ed filesystem.
10) REMOTE_BIN_INSTALL_DIR is the directory where MySQL/Drizzle binaries
will be placed on the remote machine (or local machine if one machine
used only).
11) In the case of BUILD_REMOTE="y" or WINDOWS_REMOTE="y" it's necessary
to set REMOTE_SRC_INSTALL_DIR as the directory where remote builds will
be performed to later be installed into REMOTE_BIN_INSTALL_DIR.
12) For any benchmark with InnoDB it's necessary to at least set
INNODB_BUFFER_POOL_SIZE
13) MYSQL_BASE should be set to either "5.1", "5.5" or "5.6".

All variables possible to set are available in the example autobench.conf
file. The values they have there are the default values they will get
if not set. All variables not commented out in the example file are
needed to set.

Other optional parameters are:
1) SSH_PORT if ssh needs to use a non-standard port (not 22 that is)
2) SSH_USER if ssh should log into other user on remote host
3) MALLOC_LIB specifies where to find malloc library
4) USE_MALLOC_LIB specifies that special malloc library to be used
5) TMP_BASE specifies which directory to use as temp directory.
   Normally set to /tmp on Unix:es, but should be set on Windows.
6) USE_FAST_MYSQL set to "y" if one wants to compile with the highest
   optimisation level.
7) TEST_DESCRIPTION is set to a description of the test run which will
be in the title on graphs produced after the end of the test run.

It's recommended to run with jemalloc on Linux, libc malloc
have been verified to have performance bugs on at least Fedora Core 11
which makes the performance drop 4x after about half an hour of heavy
running. tcmalloc can become fragmented and thus cause performance bugs
as well. No performance bugs have been found using jemalloc so far.

When MySQL Server runs on Windows the following optional parameters are
also used:
1) (Mandatory on Windows) REMOTE_SRC_INSTALL_DIR is the directory where
software will be installed to build it.
2) CMAKE_GENERATOR is the code needed to specify to CMake the compiler
used, default is Visual;Studio;9;2008;Win64, the reason for the ; is
that spaces will not be correctly handled in the configuration file,
they will be replaced by spaces before sent to Windows.

There is also a set of parameters that can be set that corresponds to
the same parameters in MySQL/Drizzle servers. These are mostly
InnoDB parameters, but also some MySQL configuration variables that can
be adapted.

The parameters to run Sysbench benchmarks are:
1) RUN_RW
Set this to "y" if a normal Sysbench OLTP readwrite is to be executed
2) RUN_RO
Set this to "y" if a normal Sysbench OLTP readonly is to be executed
3) RUN_RW_WRITE_INT
Set this to "y" to add the parameter --oltp-index-updates=10 to RW test
4) RUN_RW_LESS_READ
Set this to "y" to add the parameter --oltp-range-size=5 and
--oltp-point-select=1 to RW test
5) RUN_WRITE
Set this to "y" to set all parameters relating to number of read queries
to 0 to ensure benchmark only contains write queries.
6) SB_USE_SECONDARY_INDEX and SB_USE_MYSQL_HANDLER
Set these to "y" to add the parameters --oltp-secondary respective
--oltp-point-select-mysql-handler for sysbench tests. --oltp-secondary
replaces the primary key index with a secondary key index.
--oltp-point-select-mysql-handler replaces the SELECT's used to do primary
key selects by HANDLER READ statements.
7) SB_NUM_PARTITIONS
Set this to number of partitions to use in test table. This will set
--oltp-num-partitions=number where the test table will have a
PARTITION BY RANGE(id) ...., where ranges are divided based on number
of ids and number of partitions. This have the effect of creating a
number of low level tables such that some mutexes are split inside the
storage engine if InnoDB is used.
8) SB_NUM_TABLES
Set this to the number of sysbench tables to use in benchmark run. MySQL
and InnoDB have a number of mutexes per table, so this can be useful to
benchmark performance for a more realistic workload where multiple hot
tables are used.
9) SB_USE_FAST_GCC
If set to "y", sysbench will be compiled with -O3 and -m64 to increase speed
of sysbench application itself.
10) SB_TX_RATE and SB_TX_JITTER
SB_TX_RATE is set to a fixed transaction rate that is desired to run
sysbench using. If this is used then SB_TX_JITTER is the variation of
the transaction rate, default is 10% of rate.
11) SB_USE_AUTO_INC
Should the sysbench table use auto increment or not, default is not.
12) SB_DIST_TYPE
Type of data distribution, default is uniform
13) SYSBENCH_ROWS
Size of each sysbench table, default is 1M
14) THREAD_COUNTS_TO_RUN
Provide here the number of threads to execute with, the list should be
separated with ;, so e.g. if we want to run with 1, 4 and 16 threads,
we give "1;4;16"
15) TASKSET and SYSBENCH_CPUS and SERVER_CPUS
By setting TASKSET=taskset and SYSBENCH_CPUS="0xF" we ensure that the
sysbench program will only use CPU with id 0,1,2 and 3.
In the same manner we can set SERVER_CPUS="0xFFF0" to ensure that the
MySQL/Drizzle server stays on CPUs with id 4-15. This feature only
works on Linux where the taskset command can be used.

It is also possible to set TASKSET=numactl, see examples/autobench.conf
for a description of this.
16) MAX_TIME
This is the number of seconds to run each sysbench test.
17) ENGINE
Usually InnoDB or NDB
18) NUM_TEST_RUNS
Number of times to run the test, mean value and standard deviation is
calculated based on all runs.
19) BETWEEN_RUNS
To ensure that the tests starts with a MySQL/Drizzle server that isn't
active in handling previous benchmarks we give a sleep period between
each test run in seconds.
20) AFTER_INITIAL_RUN
This is the time to wait after the initial run (this initial run usually
gives worse performance and is dismissed from calculations).
21) AFTER_SERVER_START
How long time should we wait after server start until we attempt to create
the test database. We are always starting from an initial database so this
means that the MySQL Server will take some time to create all InnoDB
log files, it can take as much as 2-4 minutes.
22) BETWEEN_CREATE_DB_TEST
We attempt to create the sysbench database, if we fail this specifies how
long time in seconds to wait before we attempt again.
23) NUM_CREATE_DB_ATTEMPTS
This specifies the number of attempts to do on creating sysbench database
before we claim defeat.
24) AFTER_SERVER_STOP
How long time to wait after stopping the MySQL/Drizzle server before finishing
the benchmark run.

The following InnoDB parameters can be set:
1) INNODB_ADAPTIVE_HASH_INDEX
2) INNODB_READ_AHEAD_THRESHOLD
3) INNODB_READ_IO_THREADS
4) INNODB_WRITE_IO_THREADS
5) INNODB_THREAD_CONCURRENCY
6) INNODB_LOG_FILE_SIZE
7) INNODB_LOG_BUFFER_SIZE
8) INNODB_FLUSH_LOG_AT_TRX_COMMIT
9) INNODB_IO_CAPACITY
10)INNODB_BUFFER_POOL_INSTANCES
11)INNODB_LOG_DIR
12)INNODB_MAX_PURGE_LAG
13)INNODB_SUPPORT_XA
14)INNODB_FLUSH_METHOD
15)INNODB_USE_PURGE_THREAD
16)INNODB_FILE_PER_TABLE
17)INNODB_CHANGE_BUFFERING
18)INNODB_DOUBLEWRITE
19)INNODB_FILE_FORMAT
20)INNODB_DIRTY_PAGES_PCT
21)INNODB_OLD_BLOCKS_PCT
22)INNODB_SPIN_WAIT_DELAY
23)INNODB_SYNC_SPIN_LOOPS
24)INNODB_STATS_ON_METADATA
+ a number of new ones, see example/autobench.conf

In addition one can also set the MySQL parameters:
1) KEY_BUFFER_SIZE
2) MAX_HEAP_TABLE_SIZE
3) TMP_TABLE_SIZE
4) MAX_TMP_TABLES
5) USE_LARGE_PAGES
6) LOCK_ALL
7) SORT_BUFFER_SIZE
8) MAX_CONNECTIONS
9) TABLE_CACHE_SIZE
10) TMP_BASE
11) TRANSACTION_ISOLATION
+ new ones, see example/autobench.conf

The following NDB parameters can be set:
1) USE_NDBMTD Use ndbmtd (multithreaded NDB Data nodes)
2) NDB_EXECUTE_CPU indicates how to set the locking to CPU for
each NDB Data node. List variable, one per NDB Data node.
3) NDB_DISK_IO_THREADPOOL set to number of IO threads used
in NDB Data nodes.
4) NDB_MULTI_CONNECTION number of API nodes each MySQL Server
will use when connecting to NDB Data nodes.
5) NDB_REPLICAS number of replicas used by NDB Data nodes.
6) NDB_DATA_MEMORY memory used for Data Memory in NDB Data nodes.
7) NDB_INDEX_MEMORY memory used for Index Memory in NDB Data nodes.
8) USE_NDB_O_DIRECT use O_DIRECT by NDB Data nodes.
9) DISK_CHECKPOINT_SPEED speed of NDB Data node checkpointing
+ new ones, see example/autobench.conf

examples/autobench.conf also specifies some parameters that are
set by the scripts that changes defaults in NDB, these parameters
are not configurable other than by changing benchmark scripts.

When running DBT2 benchmarks one need to set the mandatory
parameter:
1) DBT2_DATA_DIR, this is where the data files for DBT2 loading is placed.
If they don't exist, one can build them by using --generate-dbt2-data on
the bench_prepare.sh script.

DBT2 optional parameters are:
1) DBT2_WAREHOUSES, number of warehouses created in database.
2) DBT2_TERMINALS, number of threads executing per warehouse in the
test.
3) DBT2_RUN_WAREHOUSES, this is the number of warehouses used in each
test run, the number is separated by ;.
4) DBT2_TIME, time for each DBT2 test run.
5) DBT2_SPREAD, when running with multiple drivers, each driver will
   only use warehouses with this spread
6) DBT2_LOADERS, number of parallel loaders in load phase
7) USE_MYISAM_FOR_ITEM, useful when running with MySQL Cluster to avoid
   cluster traffic to read the ITEM table which is read-only.
8) DBT2_PARTITION_FLAG set to type of partitioning to be used for DBT2 tables.
9) DBT2_PK_USING_HASH_FLAG set if hash index wanted for Primary Key
10) DBT2_INTERMEDIATE_TIMER_RESOLUTION time between intermediate result reports.
It is also possible to run more complex cases with flexible number of MySQL
Servers involved as explained in examples/autobench.conf.

flexAsynch parameters:
1) FLEX_ASYNCH_API_NODES is list of the NDB API nodes to run the benchmark.
2) FLEX_ASYNCH_NUM_THREADS is the number of threads in each flexAsynch instance
3) FLEX_ASYNCH_NUM_PARALLELISM is the number of parallel transactions handled
by each thread and sent in each batch from the thread.
4) FLEX_ASYNCH_EXECUTION_ROUNDS number of batches sent per thread
5) FLEX_ASYNCH_NUM_ATTRIBUTES number of attributes in benchmark table
6) FLEX_ASYNCH_ATTRIBUTE_SIZE number of words (4 bytes) in each attribute.
7) FLEX_ASYNCH_NO_LOGGING use tables in NDB without logging
8) FLEX_ASYNCH_NO_HINT don't use hints to select transaction coordinator
9) FLEX_ASYNCH_FORCE_FLAG flag whether to use force or not in NDB API. Can
be set to force to send immediately, adaptive to use adaptive sending and
non-adaptive to send every timer interval.
10) FLEX_ASYNCH_USE_WRITE use write in place of update/insert
11) FLEX_ASYNCH_NUM_MULTI_CONNECTIONS number of API nodes used per flexAsynch
program.
12) FLEX_ASYNCH_WARMUP_TIMER number of seconds of warmup before transactions
are counted for result.
13) FLEX_ASYNCH_EXECUTION_TIMER number of seconds to execute the measured part
of the flexAsynch benchmark
14) FLEX_ASYNCH_COOLDOWN_TIMER number of seconds to execute after finishing
the measured part of the benchmark.
+ new ones, see example/autobench.conf
