All tests in /usr/src/regress are run every day by cron(8). The test machines are automatically installed from an i386 or amd64 snapshot, the current source tree is checked out and necessary packages are installed. The results are shown over the time in a HTML table. Complete log output from the setup and test run exist. Tests are skipped if packages are missing, if they are specific for another architecture, or if additional remote machines are missing. The framework to run everything is available at github.
dmesg i386, dmesg amd64An OpenBSD release is installed. Based on that, the kernel sources for a specific date are checked out from cvs. A new kernel is built, installed, and performace tests are executed with that. The checkout, build, test is iterated in a loop while advancing the cvs checkout date in steps. Currently iperf TCP and UDP, tcpbench TCP and kernel build times are measured. To reach over incompatibilities between the kernel and user land or build enviroment, so called build quirks are used. Then the necessary parts of the source tree are updated or patched. After building and installing the relevant tools, checkout and building kernel continues.
dmesg with 8 cores, dmesg with 4 coresThe os-test test suite of the Sortix project is designed to find differences in system call behavior of various POSIX operating systems. Run the tests during OpenBSD regress and display the results every day. The date in the table is taken from the snapshot kernel's build time.
The open POSIX test suite provides good test coverage for system calls. It tries to compile varoius test programs and runs a lot of binaries. The output of the build and run with a current OpenBSD snapshot is reported. The test suite is mature and unmaintained, so the result is not significant. The important part is that it executes many error paths in the kernel and libc.
Every day performance numbers are generated with an amd64 snapshot. The machine has 10 GBit/sec interfaces. The routing tests measure the forwarding throughput with TCP streams and UDP packets with pf turned on. For proxy testing relayd(8) is accepting TCP connections, connects to a server and splices the content. For testing IPsec two OpenBSD machines encapsulate traffic in tunnel mode. Although the results are fluctuating, trends over the time can be seen.
Traffic is generated by multiple GNU/Linux machines. They have three 1 GBit/sec interfaces each and can be combined as client and server clusters. Testing TCP is done with iperf and a maximum of 30 parallel streams. The UDP tests are also based on iperf and use 3 to 9 streams. The rate of packet generation is adjusted with bisection up to a precision of 5 MBit/sec. The maximum allowed packet loss for valid measurements is 0.1%. In proxy mode each TCP stream from a Linux client is connected to its own listen socket at relayd and from there forwarded to the corresponding Linux server. The socket buffer size of relayd is set to 262144. Each measurement is performed for 60 seconds.
dmesgThanks to genua GmbH for work time, rack space, hardware, network, and electricity.