README

Run a bunch of OpenBSD regression tests in a row.  The intension
is to have a reliable set of tests and run them on a regular basis.
The result summary is written to a file, log output of each test
is available.

A second prupose is to run OpenBSD performance tests.  A certain
cvs checkout corresponding to a date is created.  The build environment
is based on the previous release and adapted with commands from
current.html.  This is mostly updating the build tools to compile
a kernel from that cvs checkout date.  The results are grouped by
run date, cvs checkout date, and repetitions.  The tests are run
multiple times to detect deviations.  Tests consist of iperf3,
tcpbench, and kernel compile time.

Additional features are running ports tests, building release, and
testing network interfaces in multiple configurations.

Buildquirks.pm	- create tool chain to compile kernel from certain date
Hostctl.pm	- run commands on all tests hosts to configure them
Html.pm		- utility functions to create html pages with results
Logcmd.pm	- common log funtions for run.pl, setp.pl, and setup.pl
Machine.pm	- functions to setup a single machine
Netstat.pm	- parse netstat statistics and show coounter differences
Testvars.pm	- variables with test names for perform and netlink
bsdcons.pl	- collect kernel output from console of selected machine
build-ot14.list	- setup builds special tools needed on test machine ot14
cleanup.pl	- remove old test logs to recover disk space
cmd-ot1.list	- setup executes commands after installation of ot1
cvsbuild.pl	- checkout kernel at certain cvs date and install it
cvslog.pl	- collect all cvs commit messages between two dates
env-ot1.sh	- environment variables to be set on the test machine ot1
favicon.ink.svg	- favicon for bluhm.genua.de created with inkscape
favicon.svg	- favicon converted to small SVG
force-ssh.pl	- restrict ssh commands on linux perform servers
gnuplot.pl	- generate gnuplot graphics for perform html
keys.sh		- generate gnuplot icons for legend in html graphics
		  this is started by gnuplot.pl
mktestlist.pl	- find possible names of tests in regress directory
make.pl		- run make build and release
makealign.sh	- align object files of kernel on perform machine
mktestlist.pl	- print the Makefile leaves of the regress directory
net.pl		- run network regression tests
netlink-html.pl	- convert netlink test results into html table
netlink.pl	- set up and run one network regression test on test machine
		  this is started by net.pl
once.pl		- run performance test once, used for manual testing
perform-html.pl	- convert performance test results into html table
perform.pl	- run one set up performace test on test machine,
		  this is started by step.pl or once.pl
pkg-ot1.list	- setup installs packages on test machine ot1
plot.gp		- gnuplot script used by gnuplot.pl
ports.list	- list of /usr/ports subdirectories that should be tested
portstest.pl	- run port test for set of ports on test machine
		  this is started by test.pl
power.pl	- power up and down machine to save cooling power
reboot.pl	- reboot machine, after optionally reordering kernel
regress-html.pl	- convert all regression test results into html table
regress.pl	- run one openbsd regression test on test machine,
		  this is started by run.pl
release.pl	- build release from /usr/src
		  this is started by make.pl
run.pl		- do regression tests, add this to cron job
running-html.pl	- create html table with all tests currently running
setup-html.pl	- convert machine setup logs into html table
setup.pl	- install one test machine,
		  this is started by run.pl and step.pl
site.list	- list of config files that must be installed on test machine
step.pl		- do performance tests, add this to cron job
test.pl		- do ports tests, add this to cron job
test.list	- list of /usr/src/regress subdirectories that should be tested
testsuite-html.pl - create html of test suites run during regress

Several modes for run.pl and setup.pl exist to prepare the test machines.

build	- build system from source /usr/src
cvs	- cvs update /usr/src and make obj
install	- install from snapshot
keep	- keep existing software on machine and skip setup
kernel	- build kernel from source /usr/src/sys
upgrade	- upgrade with snapshot

Daily test results are published here:
http://bluhm.genua.de/regress/results/regress.html
Performance timelines are published here:
http://bluhm.genua.de/perform/results/perform.html

Thanks to genua GmbH https://www.genua.de/ for sponsoring this work.