Benchmarking a deployment

The amc command line utility comes with a subcommand to allow easy benchmarking of an Anbox Cloud deployment. It can be used to measure the time containers take to start up but also for measuring their frames per second. The benchmark enables you to put Anbox Cloud under load. The results can be used to evaluate the performance of Anbox Cloud for a well defined workload.

Warning: If your application is not constantly refreshing the screen by itself, like regular Android applications providing a simple user interface do, you will get a low FPS number. The benchmark can only provide useful information if you know your workload and how it should perform in an ideal scenario.

Run the Benchmark

Have a look at the output of

$ amc benchmark -h

to learn more about the different arguments and their purpose.

Hint: Keep in mind that the success of container startup and the times you will get depend on the underlying resources used on the machines hosting the containers. Latency between the different nodes and the AMS services also play their role.

An example benchmark session looks like this:

$ amc benchmark --fps --network-address=172.31.4.11 --num-containers=15 --containers-per-second=0.1 bh2q90vo3v1lt1ft4mlg
2019/01/21 11:11:49 Test environment:
2019/01/21 11:11:49   AMS version: 1.7
2019/01/21 11:11:49   Available nodes:
2019/01/21 11:11:49     lxd0 (CPU: 48, Memory: 185GB)
2019/01/21 11:11:49
2019/01/21 11:11:49 Launching 15 containers for application bh2q90vo3v1lt1ft4mlg with 0.1 containers per second
2019/01/21 11:11:49
2019/01/21 11:11:59 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:05 Started container bh2qhvvo3v1lt1ft4ndg
2019/01/21 11:12:09 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:15 Started container bh2qi2fo3v1lt1ft4neg
2019/01/21 11:12:19 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:25 Started container bh2qi4vo3v1lt1ft4nfg
2019/01/21 11:12:29 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:35 Started container bh2qi7fo3v1lt1ft4ngg
2019/01/21 11:12:39 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:45 Started container bh2qi9vo3v1lt1ft4nhg
2019/01/21 11:12:49 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:12:55 Started container bh2qicfo3v1lt1ft4nig
2019/01/21 11:12:59 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:05 Started container bh2qievo3v1lt1ft4njg
2019/01/21 11:13:09 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:15 Started container bh2qihfo3v1lt1ft4nkg
2019/01/21 11:13:19 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:25 Started container bh2qijvo3v1lt1ft4nlg
2019/01/21 11:13:29 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:35 Started container bh2qimfo3v1lt1ft4nmg
2019/01/21 11:13:39 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:45 Started container bh2qiovo3v1lt1ft4nng
2019/01/21 11:13:49 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:13:55 Started container bh2qirfo3v1lt1ft4nog
2019/01/21 11:13:59 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:14:05 Started container bh2qitvo3v1lt1ft4npg
2019/01/21 11:14:09 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:14:15 Started container bh2qj0fo3v1lt1ft4nqg
2019/01/21 11:14:19 Launching container for application bh2q90vo3v1lt1ft4mlg ...
2019/01/21 11:14:25 Started container bh2qj2vo3v1lt1ft4nrg
2019/01/21 11:14:25 Done starting all containers. Measuring container statistics for 1m0s ...
2019/01/21 11:15:25 Removing all containers ...
2019/01/21 11:15:25 Removing container bh2qi2fo3v1lt1ft4neg ...
2019/01/21 11:15:25 Removing container bh2qi7fo3v1lt1ft4ngg ...
2019/01/21 11:15:25 Removing container bh2qitvo3v1lt1ft4npg ...
2019/01/21 11:15:25 Removing container bh2qievo3v1lt1ft4njg ...
2019/01/21 11:15:25 Removing container bh2qj2vo3v1lt1ft4nrg ...
2019/01/21 11:15:25 Removing container bh2qj0fo3v1lt1ft4nqg ...
2019/01/21 11:15:25 Removing container bh2qicfo3v1lt1ft4nig ...
2019/01/21 11:15:25 Removing container bh2qihfo3v1lt1ft4nkg ...
2019/01/21 11:15:25 Removing container bh2qijvo3v1lt1ft4nlg ...
2019/01/21 11:15:25 Removing container bh2qimfo3v1lt1ft4nmg ...
2019/01/21 11:15:25 Removing container bh2qiovo3v1lt1ft4nng ...
2019/01/21 11:15:25 Removing container bh2qi4vo3v1lt1ft4nfg ...
2019/01/21 11:15:25 Removing container bh2qirfo3v1lt1ft4nog ...
2019/01/21 11:15:25 Removing container bh2qhvvo3v1lt1ft4ndg ...
2019/01/21 11:15:25 Removing container bh2qi9vo3v1lt1ft4nhg ...
2019/01/21 11:15:39 Containers boot time measurement:
2019/01/21 11:15:39   Launching all 15 containers took 2m36.560310342s
2019/01/21 11:15:39   Out of 15 containers 0 failed to launch
2019/01/21 11:15:39   Average container launch time: 6.149119411s
2019/01/21 11:15:39   Max container launch time: 6.576302043s
2019/01/21 11:15:39   Min container launch time: 5.911184959s
2019/01/21 11:15:39   Android system failed to boot in the following containers:
2019/01/21 11:15:39     None
2019/01/21 11:15:39 Containers statistics:
2019/01/21 11:15:39   FPS avg 58 min 52 max 64 for 15 containers
2019/01/21 11:15:39   Containers below run at low FPS(<30):
2019/01/21 11:15:39     None

Example Stress Test Application

In order to make evaluation and testing easier we provide a GPU stress test application based on an example application from Google.
The application is a stress test which is primarily being used to verify the OpenGL emulation of the Android emulator but serves general stress test purposes as well.

The application was modified to constantly run the benchmark scene with ~1000 objects to drive CPU and GPU utilization of a container. You can find the application APK package here:

Once downloaded you can add the application as regular (see Application Management) to your Anbox Cloud installation. The manifest may look like:

name: gpu-stress-test
instance: a2.3
image: default

Hint: Depending on the selected instance type and how powerful a single core of your machines is you may get better results with a larger instance type.