Separated how-to-build -> how-to-build + how-to-use, added details on...

Separated how-to-build -> how-to-build + how-to-use, added details on experiment parallelization, updated fail-structure docs.

git-svn-id: https://www4.informatik.uni-erlangen.de/i4svn/danceos/trunk/devel/fail@1348 8c4709b5-6ec9-48aa-a5cd-a96041d1645a

doc/coverage-experiment-excerpt.cc

-// restore previously saved simulator state
-RestoreEvent ev_restore("sav/p_entry.sav");
-addEventAndWait(&ev_restore);
-
-// breakpoint $instr instructions in the future
-BPEvent ev_instr_reached(ANY_ADDR, instr);
-addEvent(&ev_instr_reached);
-// breakpoint at function exit
-BPEvent ev_func_end(INST_ADDR_FUNC_END);
-addEvent(&ev_func_end);
-// if we reach the exit first, we're done
-if (waitAny() == ev_func_end.id()) { break; }
-
-// commission a register bit-flip FI
-RegisterBitflip fi_bitflip(*reg, bitnr);
-inject(&fi_bitflip);
-// catch traps and timeout
-TrapEvent ev_trap(ANY_TRAP);
-addEvent(&ev_trap);
-BPEvent ev_timeout(ANY_ADDR, 1000);
-addEvent(&ev_timeout);          // [...]
-// wait for function exit, trap or timeout
-if ((id = waitAny()) == ev_func_end.id()) {
-	int result = registers(REG_EAX).cur_value();
-	log(*reg,bitnr,instr,LOG_RESULT,result);
-} else if (id == ev_trap.id()) {
-	log(*reg,bitnr,instr,LOG_TRAP,ev_trap.type());
-} else if (id == ev_timeout.id()) {
-	log(*reg,bitnr,instr,LOG_TIMEOUT); }

doc/coverage-experiment.cc

-for (auto reg = registers.begin(); reg != registers.end(); ++reg) {
-	for (int bitnr = 0; bitnr < (*reg).width(); ++bitnr) {
-		for (int instr = 0; ; ++instr) {
-			clearEvents();
-
-			// restore previously saved simulator state
-			RestoreEvent ev_restore("./bochs_save_point");
-			addEventAndWait(&ev_restore);
-
-			// breakpoint $instr instructions in the future
-			BPEvent ev_instr_reached(ANY_ADDR, instr);
-			addEvent(&ev_instr_reached);
-			// breakpoint at function exit
-			BPEvent ev_func_end(INST_ADDR_FUNC_END);
-			addEvent(&ev_func_end);
-
-			// if we reach the exit first, this round is done
-			if (waitAny() == ev_func_end.id()) {
-				break;
-			}
-
-			// inject bit-flip at bit $bitnr in register $reg
-			RegisterBitflip fi_bitflip(*reg, bitnr);
-			inject(&fi_bitflip);
-
-			// catch traps and timeout
-			TrapEvent ev_trap(ANY_TRAP);
-			addEvent(&ev_trap);
-			BPEvent ev_timeout(ANY_ADDR, 1000);
-			addEvent(&ev_timeout);
-
-			// we assert the succeeded bit-flip is the first thing we hear
-			id = waitAny();
-			assert(id == fi_bitflip.id());
-
-			// wait for function exit, trap or timeout
-			id = waitAny();
-			if (id == ev_func_end.id()) {
-				// log result
-				int result = registers(REG_EAX).cur_value();
-				log_test_result(*reg, bitnr, instr, LOG_RESULT, result);
-			} else if (id == ev_trap.id()) {
-				log_test_result(*reg, bitnr, instr, LOG_TRAP, ev_trap.TrapNr());
-			} else if (id == ev_timeout.id()) {
-				log_test_result(*reg, bitnr, instr, LOG_TIMEOUT);
-			}
-		}
-	}
-}

doc/fail-structure.txt

+=========================================================================================
 Directory structure:
 =========================================================================================
 
 The folders are nested as follow:
------------------------------------
+**********************************************************************
 fail:  Fail* parent directory, containing all source & configuration files (${FAIL_DIR})
 |-cmake:  CMake-related configuration files (e.g. compiler-flags, dependencies, ...)
 |-doc:  Fail*-Framework documentation (e.g., diagrams, how-to's, ...)
@@ -26,13 +27,14 @@ fail:  Fail* parent directory, containing all source & configuration files (${FA
   |-plugins:  plugin code files (within a new dir) need to be located here
 |-[build]: recommended location of your build-tree, generated files will be placed here
 
+
 Some additional useful notes:
------------------------------------
+**********************************************************************
  - The source files have been documented using Doxygen comments. The Doxygen
    documentation can be generated by typing "make doc" in your build-tree. The
    resulting HTML and LaTeX files will be located in "${BUILD_DIR}/src/core/doc/".
  - CMake supports an "in-source" build-tree. We recommend you not to use this
-   feature, because it leads to a cluttered directory structure (mixing original
+   "feature", because it leads to a cluttered directory structure (mixing original
    source files and generated code/config files). (Since it is still possible, however,
    the "build" directory is optional.)
  - CMake invokes the compiler with the following include directories:
@@ -42,8 +44,7 @@ Some additional useful notes:
     * ${FAIL_DIR}/simulators/bochs
    These definitions simplify and shorten the include paths.
 
-
-
+=========================================================================================
 Namespace structure:
 =========================================================================================
 All classes, functions, constants, etc. are encapsulated in the namespace "fail".

doc/how-to-build.txt

+=========================================================================================
+Additional libraries/packages/tools needed for Fail*:
+=========================================================================================
+
+Required anyway:
+**********************************************************************
+ - libprotobuf-dev
+ - libpthread
+ - libpcl1-dev
+ - libboost-dev
+ - libboost-all-dev (or at least libboost-thread-dev)
+ - protobuf-compiler
+ - cmake
+ - cmake-curses-gui
+ - AspectC++ (ag++, ac++): AspectC++ 1.1 or newer is known to work and can be
+   obtained from http://www.aspectc.org; nightlies can be downloaded from
+   http://akut.aspectc.org
+
+
+For distribution/parallelization:
+**********************************************************************
+ - rsync
+ - tmux
+
+
+32-bit FailBochs on x86_64 Linux machines:
+**********************************************************************
+ - libc6-i386 + all libraries listed by 
+     $ ldd bochs|awk '{print $3}'
+   in ~/bochslibs (client.sh will add these to LD_LIBRARY_PATH)
+
+=========================================================================================
+Compiling, building and modifying: Simulators and Fail*
+=========================================================================================
+
+Building Fail*:
+**********************************************************************
+
+For the first time:
+------------------------------------------------------------
+ 1. Enter the "fail/" directory (${FAIL_DIR}, see also "fail-structure.txt"):
+      $ cd fail/
+ 2. (Optional) Cleanup previous CMake remnants:
+      $ find -name CMakeCache.txt | xargs rm
+ 3. Create out of source build directory (${BUILD_DIR}, see also "fail-structure.txt"):
+      $ mkdir build
+ 4. Enter out-of-source build directory.  All generated files end up there.
+      $ cd build
+ 5. Generate CMake environment. 
+      $ cmake ..
+ 6. Setup build configuration by opening the CMake configuration tool
+      $ ccmake .
+    Select "BUILD_BOCHS" or "BUILD_OVP".  Select an experiment to enable by naming its
+    "experiments/" subdirectory under "EXPERIMENTS_ACTIVATED".  Configure Fail* features
+    you need for this experiment by enabling "CONFIG_*" options.  Press 'c', 'g' to
+    regenerate the build system.  (Alternatively use 
+      $ cmake-gui .
+    for a Qt GUI.)  To enable a Debug build, choose "Debug" as the build type.
+ 7. Additionally make sure Bochs is at least configured (see below).
+
+
+After changes to Fail* code:
+------------------------------------------------------------
+Prerequisite, if you're building with Bochs: configure Bochs (see below).
+Compile (in ${BUILD_DIR}, optionally "add -jN" for parallel building):
+  $ make
+CMake will build all Fail* libraries, merge them into a libfail.a and put it into
+"${FAIL_DIR}/src".  (As the current Bochs Makefile expects it there.)  The static
+library contains all core components and activated experiments/plugings.
+You may use the shell script
+  $ ${FAIL_DIR}/scripts/rebuild-bochs.sh [-]
+to speed up repetitive tasks regarding Fail/Bochs builds.  This script contains a
+concise documentation on itself.
+
+
+Add new Fail* sources to build chain:
+------------------------------------------------------------
+To add new source files to the build, see CMakeLists.txt in the subdirectory of the
+corresponding component in "${FAIL_DIR}/src/core/", and probably consultate the
+CMake docs: http://cmake.org/cmake/help/documentation.html
+
+
+Add new experiment/plugin/campaign:
+------------------------------------------------------------
+Look at "${FAIL_DIR}/src/experiments/coolchecksum/" as a template.  After creating a
+new subdirectory in "experiments/", activate it in the CMake configuration step (see
+above).
+
+
+Generating the Doxygen documentation for Fail*:
+**********************************************************************
+To generate the Doxygen documentation, type:
+  $ make doc
+The documentation files (HTML and LaTeX) are located in "${BUILD_DIR}/src/core/doc/".
+To open the HTML documentation, type:
+  $ firefox src/core/doc/html/index.html
+(You may want to replace "firefox" with your favourite browser.) The LaTeX docs need
+to be compiled previously:
+  $ cd src/core/doc/latex; make
+
+
+Building Bochs:
+**********************************************************************
+
+For the first time:
+------------------------------------------------------------
+ 1. Change to the Bochs simulator directory (expects to be in ${FAIL_DIR}):
+      $ cd simulators/bochs
+ 2. Configure the Bochs simulator (auto-tools).
+     - Sufficient:
+         $ ./configure --prefix=$(echo ~/localroot/usr) --enable-{cpu-level=6,ne2000,trace-cache,gdb-stub} --disable-docbook
+     - More optimised:
+         $ ./configure --prefix=$(echo ~/localroot/usr) --enable-{a20-pin,x86-64,cpu-level=6,ne2000,acpi,pci,usb,repeat-speedups,trace-cache,fast-function-calls,host-specific-asms,disasm,all-optimizations,readline,clgd54xx,fpu,vmx=2,monitor-mwait,cdrom,sb16=linux,gdb-stub} --disable-docbook --with-all-libs
+    Instead of --with-all-libs, you could use --with-nogui for "headless" experiments,
+    additionally removing lots of library dependencies (thus reducing startup overhead).
+    --with-x11 enables only the "x" (X11), --with-wx only the "wx" (wxWidgets) GUI.
+    Note that "wx" does not play well together with Fail*'s "restore" feature (FailBochs
+    will fall back to "x" if available, or die trying.)
+    Once you know everything works as it should, you may want to add the
+    following flags before running larger campaigns:
+      --disable-logging --disable-assert-checks
+    FIXME: Remove more redundant flags/libraries
+
+
+After changes to Bochs code or Bochs-affecting aspects:
+------------------------------------------------------------
+ - Compiling: The make call from the make-ag++.sh is now invokable by calling
+   (still in ${BUILD_DIR}, optionally adding -jN for parallel building):
+     $ cd ../build %% FIXME: involviert make bochs (im build-Ver.) wirklich make-ag++.sh?
+     $ make bochs
+   (Of course, this requires a configured Bochs/Fail*.)
+ - Cleaning up: The former make all-clean is now invokable by
+     $ make bochsallclean
+ - Installing: For installing the bochs executable (former "make install")
+     $ make bochsinstall
+   (See "make help" for a target listing.)
+ - Note: You may use scripts/rebuild-bochs.sh to speed up repetitive tasks regarding
+   Fail/Bochs builds.  This script contains a concise documentation on itself.
+
+
+Debug build:
+------------------------------------------------------------
+Configure Bochs to use debugging-related compiler flags (expects to be in ${BUILD_DIR}):
+  $ cd ../simulator/bochs
+  $ CFLAGS="-g -O0" CXXFLAGS="-g -O0" ./configure --prefix=... ... (see above)
+
+
+Profiling-based optimization build:
+------------------------------------------------------------
+FIXME: ag++ needs to be run with --keep_woven
+Configure Bochs to use compiler flags to enable profiling:
+  $ cd ../simulator/bochs
+  $ CFLAGS="-fprofile-generate" CXXFLAGS="-fprofile-generate" LDFLAGS="-fprofile-generate" ./configure --prefix=... ... (see above)
+Build Bochs normally, and run it.  Configure Bochs to use compiler flags to enable
+optimizations based on profiling results:
+  $ CFLAGS="-fprofile-use -Wcoverage-mismatch" CXXFLAGS="-fprofile-use -Wcoverage-mismatch" LDFLAGS="-fprofile-use" ./configure --prefix=... ... (see above)
+
+
+Benchmarking:
+------------------------------------------------------------
+Simple instructions/second measurement:
+ - Configure Bochs with --enable-show-ips (see above)
+ - Modify the bochsrc: print_timestamps: enabled=1
+Comparison IPS numbers are shown in the default "bochsrc". Headless bochsrc configuration,
+all aspects disabled, guest system executes endless loop, host CPU Xeon X5470 (3.33GHz):
+IPS: 66124283 average = 66964789
+
+Building OVP for ARM Cortex-M3:
+**********************************************************************
+
+For the first time:
+------------------------------------------------------------
+ 1. Get a license from http://www.ovpworld.org/
+ 2. Download the following files:
+     - Downloads -> Main OVP Download including OVPsim Simulator
+     - Downloads -> ARM -> ARM OVP Cortex-M Profile Model
+                        -> Self contained ARM Cortex-M examples
+                        -> ARM GNU GCC and GDB tools
+ 3. Install OVP by running the self-extracting archives.
+ 4. Get Sourcery CodeBench Lite Edition from
+    http://www.mentor.com/embedded-software/codesourcery
+    (ARM processors -> EABI release)
+ 5. Install the self-extracting archive, or use the installation in
+    /fs/proj/compiler/codesourcery-arm-eabi (DO, ios/kos)
+TODO: Source setup.sh, setupImperas ...
+TODO: Fix Hard-coded paths

doc/how-to-use.txt

+=========================================================================================
+Steps to run a boot image in Fail* using the Bochs simulator backend:
+=========================================================================================
+Follow the Bochs documentation, and start your own "bochsrc" configuration file
+based on the "${PREFIX}/share/doc/bochs/bochsrc-sample.txt" template (or
+"/usr/share/doc/bochs/examples/bochsrc.gz" on Debian systems with Bochs installed).
+ 1. Add your floppy/cdrom/hdd image in the floppya/ata0-master/ata0-slave
+    sections; configure the boot: section appropriately.
+ 2. Comment out com1 and parport1.
+ 3. The following Bochs configuration settings (managed in the "bochsrc" file) might
+    be helpful, depending on your needs:
+     - For "headless" experiments:
+         config_interface: textconfig
+         display_library: nogui
+     - For an X11 GUI:
+         config_interface: textconfig
+         display_library: x
+     - For a wxWidgets GUI (does not play well with Fail*'s "restore" feature):
+         config_interface: wx
+         display_library: wx
+     - Reduce the guest system's RAM to a minimum to reduce Fail*'s memory footprint
+       and save/restore overhead, e.g.:
+         memory: guest=16, host=16
+     - If you want to redirect FailBochs's output to a file using the shell's
+       redirection operator '>', make sure "/dev/stdout" is not used as a target
+       file for logging.  (The Debian "bochsrc" template unfortunately does this
+       in two places.  It suffices to comment out these entries.)
+     - To make Fail* terminate if something unexpected happens in a larger
+       campaign, be sure it doesn't "ask" in these cases, e.g.:
+         panic: action=fatal
+         error: action=fatal
+         info: action=ignore
+         debug: action=ignore
+         pass: action=ignore
+     - If you need a quick-and-dirty way to pass data from the guest system to the
+       outside world, and you don't want to write an experiment utilizing
+       GuestEvents, you can use the "port e9 hack" that prints all outb's to port
+       0xe9 to the console:
+         port_e9_hack: enabled=1
+     - Determinism:  (Fail)Bochs is deterministic regarding timer interrupts,
+       i.e., two experiment runs after calling simulator.restore() will count the
+       same number of instructions between two interrupts.  Though, you need to be
+       careful when running (Fail)Bochs with a GUI enabled:  Typing "bochs -q<return>"
+       on the command line may lead to the GUI window receiving a "return key
+       released" event, resulting in a keyboard interrupt for the guest system.
+       This can be avoided by starting Bochs with "sleep 1; bochs -q", or
+       disabling the GUI (see "headless experiments" above).
+
+=========================================================================================
+Example experiments and code snippets
+=========================================================================================
+
+Experiment "hsc-simple":
+**********************************************************************
+A simple standalone experiment (without a separate campaign). To compile this
+experiment, the following steps are required:
+ 1. Add "hsc-simple" to ccmake's EXPERIMENTS_ACTIVATED.
+ 2. Enable CONFIG_EVENT_BREAKPOINTS, CONFIG_SR_RESTORE and CONFIG_SR_SAVE.
+ 3. Build Fail* and Bochs, see "how-to-build.txt" for details-
+ 4. Enter experiment_targets/hscsimple/, bunzip2 -k *.bz2
+ 5. Start the Bochs simulator by typing
+      $ bochs -q
+    After successfully booting the eCos/hello world example, the console shows
+    "[HSC] breakpoint reached, saving", and a hello.state/ subdirectory appears.
+    You probably need to adjust the bochsrc's paths to romimage/vgaromimage.
+    These by default point to the locations installed by the Debian packages
+    "bochsbios" and "vgabios"; for example, you alternatively may use the
+    BIOSes supplied in "${FAIL_DIR}/simulators/bochs/bios/".
+ 6. Compile the experiment's second step: edit
+    fail/src/experiments/hsc-simple/experiment.cc, and change the first "#if 1"
+    into "#if 0".  Make an incremental build, e.g., by running
+    "${FAIL_DIR}/scripts/rebuild-bochs.sh -" from your ${BUILD_DIR}.
+ 7. Back to ../experiment_targets/hscsimple/ (assuming, your are in ${FAIL_DIR}),
+    run 
+      $ bochs -q
+    After restoring the state, the hello world program's calculation should
+    yield a different result.
+
+
+Experiment "coolchecksum":
+**********************************************************************
+An example for separate campaign/experiment implementations. To compile this
+experiment, the following steps are required:
+ 1. Run step #1 (and if you're curious how COOL_ECC_NUMINSTR in
+    experimentInfo.hpp was figured out, then step #2) of the experiment
+    (analogous to what needed to be done in case of the "hsc-simple" experiment,
+    see above).  The experiment's target guest system can be found under
+    ../experiment_targets/coolchecksum/.
+    (If you want to enable COOL_FAULTSPACE_PRUNING, step #2 is mandatory because
+    it generates the instruction/memory access trace needed for pruning.)
+ 2. Build the campaign server: make coolchecksum-server
+ 3. Run the campaign server: bin/coolchecksum-server
+ 4. In another terminal, run step #3 of the experiment ("bochs -q").
+
+Step #3 of the experiment currently runs 2000 experiment iterations and then
+terminates, because Bochs has some memory leak issues.  You need to re-run
+Bochs for the next 2k experiments.
+
+The experiments can be significantly sped up by
+ a) parallelization (run more FailBochs clients and
+ b) a headless (and more optimized) Fail* configuration (see above).
+
+
+Experiment "MHTestCampaign":
+**********************************************************************
+An example for separate campaign/experiment implementations.
+ 1. Execute Campaign (job server): ${BUILD_DIR}/bin/MHTestCampaign-server
+ 2. Run the FailBochs instance, in properly defined environment:
+      $ bochs -q
+
+=========================================================================================
+Parallelization
+=========================================================================================
+Fail* is designed to allow parallelization of experiment execution allowing to reduce
+the time needed to execute the experiments on a (larger) set of experiment data (aka
+input parameters for the experiment execution, e.g. instruction pointer, registers, bit
+numbers, ...). We call such "experiment data" the parameter sets. The so called "campaign"
+is responsible for managing the parameter sets (i.e., the data to be used by the experiment
+flows), inquired by the clients. As a consequence, the campaign is running on the server-
+side and the experiment flow are running on the (distributed) clients.
+First of all, the Fail* instances (and other required files, e.g. saved state) are
+distributed to the clients. In the second step the campaign(-server) is started, preparing
+it's parameter-sets in order to be able to answer the requests from the clients. (Once
+there are available parameter-sets, the clients can request them.) In the final step,
+the distributed Fail* clients have to be started. As soon as this setup is finished,
+the clients request new parameter-sets, execute their experiment code and return their
+results to the server (aka campaign) in an iterative way, until all paremeter-sets have
+been processed successfully. If all (new) parameter-sets have been distributed, the
+campaign starts to resend unfinished parameter-sets to requesting clients in order to
+speed up the overall campaign execution. Additionally, this ensures that all parameter
+sets will produce a corresponding result set. (If, for example, a client terminates
+abnormally, no result is send back. This scenario is managed by this "resend-mechanism"
+of the campain, too.)
+
+
+Shell scripts supporting experiment distribution:
+**********************************************************************
+These can be found in ${FAIL_DIR}/scripts/ (for now have a look at the script files
+themselves, they contain some documentation):
+ - fail-env.sh: Environment variables for distribution/parallelization host
+                lists etc.; don't modify in-place but edit your own copy!
+ - distribute-experiment.sh: Distribute necessary FailBochs ingredients to
+                             experiment hosts.
+ - runcampaign.sh: Locally run a campaign server, and a large amount of
+                   clients on the experiment hosts.
+ - multiple-clients.sh: Is run on an experiment host by runcampaign.sh,
+                        starts several instances of client.sh in a tmux session.
+ - client.sh: (Repeatedly) Runs a single FailBochs instance.
+
+
+Some useful things to note:
+**********************************************************************
+ - Using the distribute-experiment.sh script causes the local bochs binary to
+   be copied to the hosts. If the binary is not present in the current directory
+   the default bochs binary (-> $ which bochs) will be used. If you have modified
+   some of your experiment code (i.e., your bochs binary will change), don't
+   forget to delete the local bochs binary in order to distribute the *new* binary.
+ - The runcampaign.sh script prints some status information about the clients
+   recently started. In addition, there will be a few error messages concerning
+   ssh, tmux and so on. They can be ignored for now.
+ - The runcampaign.sh script starts the coolchecksum-server. Note that the server
+   instance will terminate immediatly (without notice), if there is still an
+   existing coolcampaign.csv file.
+ - In order to make the performance gains (mentioned above) take effect, a "workload
+   balancing" between the server and the clients is mandatory. This means that
+   the communication overhead (client <-> server) and the time, needed to execute
+   the experiment code on the client-side should be in due proportion. More
+   specifically, for each experiment there will be exactly 2 TCP connections
+   (send parameter-set to client, send result to server) established. Therefore
+   you should ensure that the execution time of the experiment is "long enough"
+   (heuristic). (See existing experiments for examples.)

doc/howto-build.txt

-Additional libraries/packages/tools needed for Fail*:
-
-	libprotobuf-dev
-	libpthread
-	libpcl1-dev
-	libboost-dev
-	libboost-all-dev (or at least libboost-thread-dev)
-	protobuf-compiler
-
-	cmake
-	cmake-curses-gui
-
-	AspectC++ (ag++, ac++)
-	  (AspectC++ 1.1 or newer is known to work and can be obtained from
-	   http://www.aspectc.org ; nightlies can be downloaded from
-           http://akut.aspectc.org)
-
-For distribution/parallelization:
-	rsync
-	tmux
-
-32-bit FailBochs on x86_64 Linux machines:
-	libc6-i386
-	+ all libraries listed by "ldd bochs|awk '{print $3}'" in ~/bochslibs
-	(client.sh will add these to LD_LIBRARY_PATH)
-
-===========
-Build steps
-===========
-
-Build Fail*
-===========
-For the first time
-------------------
-cd failbochs/
-
-> (optional) Cleanup previous cmake remnants
-find -name CMakeCache.txt | xargs rm
-
-> Create out of source build directory
-mkdir build
-
-> Enter out-of-source build directory
-> All generated files end up there.
-cd build
-
->> Generate cmake environment. 
-cmake ..
->> Setup build configuration
-> Select BUILD_BOCHS or BUILD_OVP
-> Select an experiment to enable by naming its experiments/ subdirectory under
->   EXPERIMENTS_ACTIVATED
-> Configure Fail* features you need for this experiment by enabling CONFIG_*
-> options.
-> Press 'c', 'g' to regenerate the build system
-> (alternatively use "cmake-gui ." for a Qt GUI)
-ccmake .
-
-> Additionally make sure Bochs is at least configured (see below).
-
-After changes to Fail* code
----------------------------
-> Prerequisite, if you're building with Bochs: configure Bochs (see below).
->
-> Compile (optionally add -jN for parallel building)
-make
-
-> cmake will build all Fail* libraries, merge them into a libfail.a and put it
-> into failbochs/fail.  (as the current Bochs Makefile expects it there..)
-
-> You may use scripts/rebuild-bochs.sh to speed up repetitive tasks regarding
-> Fail/Bochs builds.  This script contains a concise documentation on itself.
-
-Doxygen documentation
----------------------
-make doc
-firefox core/doc/html/index.html
-
-Debug build
------------
-> Choose "Debug" as the build type:
-ccmake .
-
-Add new (fail*) sources to build chain
---------------------------------------
-To add new files to the build, see CMakeLists.txt in fail/core, and probably
-consultate the CMake docs: http://cmake.org/cmake/help/documentation.html
-
-Add new user-defined experiment/campaign
-----------------------------------------
-Look at experiments/coolchecksum/ as a template.  After creating a new
-subdirectory in experiments/, activate it in the cmake configuration step (see
-above).
-
-
-Build Bochs
-===========
-For the first time
-------------------
-cd ../simulators/bochs
-> Sufficient:
-./configure --prefix=$(echo ~/localroot/usr) --enable-{cpu-level=6,ne2000,trace-cache,gdb-stub} --disable-docbook
-> More optimised:
-./configure --prefix=$(echo ~/localroot/usr) --enable-{a20-pin,x86-64,cpu-level=6,ne2000,acpi,pci,usb,repeat-speedups,trace-cache,fast-function-calls,host-specific-asms,disasm,all-optimizations,readline,clgd54xx,fpu,vmx=2,monitor-mwait,cdrom,sb16=linux,gdb-stub} --disable-docbook --with-all-libs
-> Instead of --with-all-libs, you could use --with-nogui for "headless"
-> experiments, additionally removing lots of library dependencies (thus
-> reducing startup overhead).  --with-x11 enables only the "x" (X11), --with-wx
-> only the "wx" (wxWidgets) GUI.  Note that "wx" does not play well together
-> with Fail*'s "restore" feature (FailBochs will fall back to "x" if available,
-> or die trying.)
->
-> Once you know everything works as it should, you may want to add the
-> following flags before running larger campaigns:
-> --disable-logging --disable-assert-checks
->
-> FIXME remove more redundant flags/libraries
-
-After changes to Bochs code or Bochs-affecting aspects
-------------------------------------------------------
-> The make call from the make-ag++.sh is now invokable by calling (still in
-> your build dir, optionally adding -jN for parallel building):
-cd ../build
-make bochs
-
-> The former make all-clean is now invokable by
-make bochsallclean
-
-> For installing the bochs executable (former make install)
-make bochsinstall
-> (see >make help< for a target listing.)
-
-> You may use scripts/rebuild-bochs.sh to speed up repetitive tasks regarding
-> Fail/Bochs builds.  This script contains a concise documentation on itself.
-
-Debug build
------------
-> Configure Bochs to use debugging-related compiler flags:
-cd ../bochs
-CFLAGS="-g -O0" CXXFLAGS="-g -O0" ./configure --prefix=... ... (see above)
-
-Profiling-based optimization build
-----------------------------------
-> FIXME: ag++ needs to be run with --keep_woven
-> Configure Bochs to use compiler flags to enable profiling:
-cd ../bochs
-CFLAGS="-fprofile-generate" CXXFLAGS="-fprofile-generate" LDFLAGS="-fprofile-generate" ./configure --prefix=... ... (see above)
-> Build Bochs normally, and run it.
-> Configure Bochs to use compiler flags to enable optimizations based on profiling results:
-CFLAGS="-fprofile-use -Wcoverage-mismatch" CXXFLAGS="-fprofile-use -Wcoverage-mismatch" LDFLAGS="-fprofile-use" ./configure --prefix=... ... (see above)
-
-Benchmarking
-------------
-> Simple instructions/second measurement:
-> - configure Bochs with --enable-show-ips (see above)
-> - bochsrc: print_timestamps: enabled=1
-> Comparison IPS numbers are shown in the default bochsrc.
-> Headless bochsrc configuration, all aspects disabled, guest system executes
-> endless loop, host CPU Xeon X5470 (3.33GHz): IPS: 66124283 average = 66964789
-
-Steps to run a boot image in FailBochs:
----------------------------------------
-Follow the Bochs documentation, and start your own "bochsrc" configuration file based on the
-$PREFIX/share/doc/bochs/bochsrc-sample.txt template (or
-/usr/share/doc/bochs/examples/bochsrc.gz on Debian systems with Bochs
-installed).
- -  Add your floppy/cdrom/hdd image in the floppya/ata0-master/ata0-slave
-    sections; configure the boot: section appropriately.
- -  Comment out com1 and parport1.
- -  For "headless" experiments:
-    config_interface: textconfig
-    display_library: nogui
- -  For an X11 GUI:
-    config_interface: textconfig
-    display_library: x
- -  For a wxWidgets GUI (does not play well with Fail*'s "restore" feature):
-    config_interface: wx
-    display_library: wx
- -  Reduce the guest system's RAM to a minimum to reduce FailBochs's memory
-    footprint and save/restore overhead, e.g.:
-    memory: guest=16, host=16
- -  If you want to redirect FailBochs's output to a file using the shell's
-    redirection operator '>', make sure /dev/stdout is not used as a target
-    file for logging.  (The Debian bochsrc template unfortunately does this in
-    two places.  It suffices to comment out these entries.)
- -  To make FailBochs terminate if something unexpected happens in a larger
-    campaign, be sure it doesn't "ask" in these cases, e.g.:
-    panic: action=fatal
-    error: action=fatal
-    info: action=ignore
-    debug: action=ignore
-    pass: action=ignore
- -  If you need a quick-and-dirty way to pass data from the guest system to the
-    outside world, and you don't want to write an experiment utilizing
-    GuestEvents, you can use the "port e9 hack" that prints all outb's to port
-    0xe9 to the console:
-    port_e9_hack: enabled=1
- -  Determinism:  (Fail)Bochs is deterministic regarding timer interrupts,
-    i.e., two experiment runs after calling simulator.restore() will count the
-    same number of instructions between two interrupts.  Though, you need to be
-    careful when running (Fail)Bochs with a GUI enabled:  Typing "bochs -q<return>"
-    on the command line may lead to the GUI window receiving a "return key
-    released" event, resulting in a keyboard interrupt for the guest system.
-    This can be avoided by starting Bochs with "sleep 1; bochs -q", or
-    disabling the GUI (see "headless experiments" above).
-
-
-Build OVP for Cortex-M3
-=======================
-For the first time
-------------------
-> Get a license from ovpworld.org
-> Download:
->  - Downloads -> Main OVP Download including OVPsim Simulator
->  - Downloads -> ARM -> ARM OVP Cortex-M Profile Model
->  -                  -> Self contained ARM Cortex-M examples
->  -                  -> ARM GNU GCC and GDB tools
-> Install OVP by running the self-extracting archives.
->
-> Get Sourcery CodeBench Lite Edition from
-> http://www.mentor.com/embedded-software/codesourcery
->  - ARM processors -> EABI release
-> Install the self-extracting archive, or use the installation in
-> /fs/proj/compiler/codesourcery-arm-eabi (DO, ios/kos)
->
-> TODO source setup.sh, setupImperas ...
-> TODO hard-coded paths
-
-=====================================
-Example experiments and code snippets
-=====================================
-
-hscsimple
----------
-A simple standalone experiment (without a separate campaign).
-
-1. Add "hscsimple" to ccmake's EXPERIMENTS_ACTIVATED.
-2. Enable CONFIG_EVENT_BREAKPOINTS, CONFIG_SR_RESTORE and CONFIG_SR_SAVE.
-3. Build Fail* and Bochs as described.
-4. Enter experiment_targets/hscsimple/, bunzip2 -k *.bz2
-5. Run "bochs -q".  After successfully booting the eCos/hello world example,
-   the console shows "[HSC] breakpoint reached, saving", and a hello.state/
-   subdirectory appears.
-   - You probably need to adjust the bochsrc's paths to romimage/vgaromimage.
-     These by default point to the locations installed by the Debian packages
-     "bochsbios" and "vgabios"; for example, you alternatively may use the
-   BIOSes supplied in fail/bochs/bios/.
-6. Compile the experiment's second step: edit
-   fail/core/experiments/hscsimple/experiment.cc, and change the first "#if 1"
-   into "#if 0".  Make an incremental build, e.g., by running
-   "fail/scripts/rebuild-bochs.sh -".
-7. Back to experiment_targets/hscsimple/, run "bochs -q".  After restoring the
-   state, the hello world program's calculation should yield a different
-   result.
-
-coolchecksum
-------------
-An example for separate campaign/experiment implementations.
-
-1. Run step #1 (and if you're curious how COOL_ECC_NUMINSTR in
-   experimentInfo.hpp was figured out, then step #2) of the experiment
-   (analogous to what needed to be done in case of the hscsimple experiment,
-   see above).  The experiment's target guest system can be found under
-   experiment_targets/coolchecksum/.
-   (If you want to enable COOL_FAULTSPACE_PRUNING, step #2 is mandatory because
-   it generates the instruction/memory access trace needed for pruning.)
-2. Build the campaign server: make coolchecksum-server
-3. Run the campaign server: bin/coolchecksum-server
-4. In another terminal, run step #3 of the experiment ("bochs -q").
-
-Step #3 of the experiment currently runs 2000 experiment iterations and then
-terminates, because Bochs has some memory leak issues.  You need to re-run
-Bochs for the next 2k experiments.
-
-The experiments can be significantly sped up by a) parallelization (run more
-FailBochs clients; TODO document fail/scripts/* stuff) and b) a headless (and
-more optimized) FailBochs configuration (see above).
-
-MHTestCampaign
---------------
-An example for separate campaign/experiment implementations.
-
-1. execute Campaign (job server):
- <your_build_dir>/bin/MHTestCampaign-server
-2. run the FailBochs instance, in properly defined environment:
- bochs
-
-Dwarf example (tests/dwarf.cc)
-------------------------------
-Dependencies: 
-	LibDwarf: (debian/ubuntu package: libdwarf-dev)
-	LibElf:	  (debian/ubuntu package: libelf-dev)
-
-Build with cmake:
-cd {your-out-of-source-dir}
-cmake {path to fail dir}
-	-> cmake checks if libelf and libdwarf are present
-
-make dwarf
-./tests/dwarf main.elf
-
-===============
-Parallelization
-===============
-FIXME more details
-
-Shell scripts supporting experiment distribution/parallelization
-----------------------------------------------------------------
-These can be found in fail/scripts/ (for now have a look at the script files
-themselves, they contain some documentation):
- -  fail-env.sh -- Environment variables for distribution/parallelization host
-    lists etc.; don't modify in-place but edit your own copy!
- -  distribute-experiment.sh -- Distribute necessary FailBochs ingredients to
-    experiment hosts
- -  runcampaign.sh -- Locally run a campaign server, and a large amount of
-    clients on the experiment hosts
- -  multiple-clients.sh -- Is run on an experiment host by runcampaign.sh,
-    starts several instances of client.sh in a tmux session
- -  client.sh -- (Repeatedly) runs a single FailBochs instance
-
-Some useful things to note:
----------------------------
- -  Using the distribute-experiment.sh script causes the local bochs binary to
-    be copied to the hosts. If the binary is not present in the current directory
-    the default bochs binary (-> which bochs) will be used. If you have modified
-    some of your experiment code (id est, your bochs binary will change), don't
-    forget to delete the local bochs binary in order to distribute the *new* binary.
- -  The runcampaign.sh script prints some status information about the clients
-    recently started. In addition, there will be a few error messages concerning
-    ssh, tmux and so on. They can be ignored for now.
- -  The runcampaign.sh script starts the coolchecksum-server. Note that the server
-    instance will terminate immediatly (without notice), if there is still an
-    existing coolcampaign.csv file.