All tests in /usr/src/regress are run every day by cron(8). The test machines are automatically installed from an i386 or amd64 snapshot, the current source tree is checked out and necessary packages are installed. The results are shown over the time in a HTML table. Complete log output from the setup and test run exist. Tests are skipped if packages are missing, if they are specific for another architecture, or if additional remote machines are missing. The framework to run everything is available at github.
from 6.2 to 6.3, daily steps, 4 core machine, TCP and make
from 6.3 to 6.4, daily steps, 4 core machine, TCP and make
from 6.4 to 6.5, daily steps, 4 core machine, TCP and make
from 6.5 to 6.6, daily steps, 4 core machine, TCP, UDP and make
5 days of 6.4, 10 repetitions, keep running between tests
5 days of 6.4, 10 repetitions, reboot machine between tests
5 days of 6.4, 10 repetitions, reorder kernel between tests
5 days of 6.4, 10 repetitions, sorted kernel objects without gap
5 days of 6.4, 10 repetitions, random gap before sorted kernel objects
dmesg with 8 cores, dmesg with 4 cores
The os-test test suite of the Sortix project is designed to find differences in system call behavior of various POSIX operating systems. Run the tests during OpenBSD regress and display the results every day. The date in the table is taken from the snapshot kernel's build time.
The open POSIX test suite provides good test coverage for system calls. It tries to compile varoius test programs and runs a lot of binaries. The output of the build and run with a current OpenBSD snapshot is reported. The test suite is mature and unmaintained, so the result is not significant. The important part is that it executes many error paths in the kernel and libc.
Every day performance numbers are generated with an amd64 snapshot. The machine has 10 GBit/sec interfaces. The routing tests measure the forwarding throughput with TCP streams and UDP packets with pf turned on. For proxy testing relayd(8) is accepting TCP connections, connects to a server and splices the content. For testing IPsec two OpenBSD machines encapsulate traffic in tunnel mode. Although the results are fluctuating, trends over the time can be seen.
Traffic is generated by multiple GNU/Linux machines. They have three 1 GBit/sec interfaces each and can be combined as client and server clusters. Testing TCP is done with iperf and a maximum of 30 parallel streams. The UDP tests are also based on iperf and use 3 to 9 streams. The rate of packet generation is adjusted with bisection up to a precision of 5 MBit/sec. The maximum allowed packet loss for valid measurements is 0.1%. In proxy mode each TCP stream from a Linux client is connected to its own listen socket at relayd and from there forwarded to the corresponding Linux server. The socket buffer size of relayd is set to 262144. Each measurement is performed for 60 seconds.
Thanks to genua GmbH for work time, rack space, hardware, network, and electricity.