1 The docker image `tsan11-tsan11rec-image.tar.gz` contains prebuilt LLVMs for tsan11 and tsan11rec. Tsan11 was built based on the instructions given at: https://github.com/mc-imperial/tsan11/. And the source code for tsan11rec was obtained from https://github.com/ChrisLidbury/tsan11/ with commit hash `ecbd6b8`.
3 ## Building the docker container and sshing in:
5 In the repository root:
9 docker image load -i tsan11-tsan11rec-image.tar.gz
11 # Create the container:
12 docker create -it --name tsan11-tsan11rec-container tsan11-tsan11rec-image:asplos
14 # Start the container:
15 docker start -ia tsan11-tsan11rec-container
18 Then in a different terminal in the same directory:
21 # Change permission of the key
22 chmod 600 insecure_key
24 # To find the IP address of the container:
25 docker inspect tsan11-tsan11rec-container
27 # Then copy the `scripts` folder in this directory
28 # to the container (replace 172.17.0.2 by the container’s IP address):
29 scp -i insecure_key -r scripts root@172.17.0.2:/data
31 # ssh into the container as root (replace 172.17.0.2 by the container’s IP address):
32 ssh -i insecure_key root@172.17.0.2
35 ## Compiling benchmarks for tsan11 and tsan11rec:
37 Inside the docker container:
41 # Start the shell that downloads and compiles benchmarks
45 The `setup.sh` will create two benchmark directories `/data/tsan11-benchmarks` and `/data/tsan11rec-benchmarks` for tsan11 and tsan11rec respectively.
47 ## To run benchmarks for tsan11 and tsan11rec:
48 The benchmark directories `tsan11-benchmarks` and `tsan11rec-benchmarks` have the same structures. So we will use `tsan11-benchmarks` as an example.
50 Our benchmarks fall into three categories: application benchmarks, data structure benchmarks (CDSChecker data structure benchmarks) used to evaluate CDSChecker, and data structure benchmarks with injected bugs that both tsan11 and tsan11rec miss.
52 In the `tsan11-benchmarks` directory, the application benchmarks include Gdax (`gdax-orderbook-hpp`), Iris (`iris`), Mabain (`mabain`), Silo (`silo`), and the Javascript Engine of Firefox that runs Jsbench (`jsbench-2013.1`). The `tsan11-benchmarks` repository does not contain the Javascript Engine of Firefox, but the setup scripts download Firefox release 50.0.1, compile the Javascript Engine, and copy the Javascript Engine binary into the benchmark directory (`tsan11-benchmarks/js`) by running the script `build_firefox_jsshell.sh`.
54 The `tsan11-benchmarks/cdschecker_modified_benchmarks` directory contains data structure benchmarks used to evaluate CDSChecker. The `tsan11-benchmarks/tsan11-missingbug` directory contains data structure benchmarks with injected bugs that tsan11 and tsan11rec fail to detect.
56 After the setup, some scripts are copied into the `tsan11-benchmarks` directory, including a unified script `do_test_all.sh` that runs all tests.
60 cd /data/tsan11-benchmarks
64 We also provide instructions on how to run each individual test below.
66 ### To run application benchmarks:
69 cd /data/tsan11-benchmarks
70 ./app_test_all.sh [number of runs]
73 The `tsan11-benchmarks/app_test_all.sh` script runs all of five application benchmarks in both the all-core and single-core configurations. The `app_test_all.sh` script also accepts an integer as an optional parameter that specifies how many times each application benchmark is run, such as `./app_test_all.sh 5`. It runs all of five application benchmarks 10 times by default. After finish running the application benchmarks, the `app_test_all.sh` script executes `python calculator.py all-core` or `python calculator.py single-core` in the `tsan11-benchmarks` directory to print out results.
75 ### To run CDSChecker data structure benchmarks:
78 cd /data/tsan11-benchmarks/cdschecker_modified_benchmarks
82 The `cdschecker_modified_benchmarks/test_all.sh` script tests seven data structures for 500 runs and reports data race detection rates and execution time for each data structure. The results are printed in the console.
84 ### To run data structure benchmarks with bugs that tsan11 and tsan11rec miss:
87 cd /data/tsan11-benchmarks/tsan11-missingbug
91 The `tsan11-missingbug/test_all.sh` script tests two buggy data structure implementations for 1000 runs and reports assertion detection rates and execution time for each data structure. The results are printed in the console.
93 ### To test assertion failures in Silo and Mabain:
96 cd /data/tsan11-benchmarks/
97 ./app_assertion_test.sh [number of runs]
100 The `app_assertion_test.sh` script also accepts an optional integer parameter that specifies how many times Silo and Mabian are tested. Both benchmarks are tested 10 time for assertion failures by default. The assertion detection rates are reported and printed in the console. Both tsan11 and tsan11 detect the assertion failures in Mabain but fail to do so in Silo.