The openQA-based testing system has recently been integrated into KDE Linux
(hooray!), and I thought it was about time I did a little write-up.
The nature of KDE Linux, in which the whole system ships as a single signed
image rather than a pile of packages, is (in theory) wonderful for reliability.
However, this raises an uncomfortable question: how do we make sure that image
actually works before we ship it to people? OpenQA is the answer!
TL;DR: we boot each build in a virtual machine, run tests that interact with it
to ensure the system installs and upgrades properly and that desktop
functionality works. Once the tests pass, the user gets an end-to-end
tested image. This replaces the rather rudimentary basic-test machinery, which
simply booted up the live image and checked if the boot was blessed and if any
units failed.
The test flow
A single build goes through three stages.
install-system takes the live ISO, boots it in a VM, and runs a real
installation onto an empty virtual disk, just like a real user would.
sanity-test then boots that freshly installed disk and verifies the system
actually comes up and behaves. In between, while we're testing the upgrade path
in parallel, an upgrade-system stage installs the previous release and upgrades
it to the current build to check whether the previous release can actually be
upgraded to the new build. Each stage hands its disk to the next.
They're wired together as a dependency chain, so in the openQA web interface
the whole run shows up as a single connected graph. If installation fails, the
later stages don't bother running, as there's nothing to test.
Our CI pipelines now approximately look like this:
Interesting architectural tidbits
We do a few things differently compared to your stock-standard openSUSE or
Fedora openQA instances.
Selenium testing instead of needle testing
Normal openQA tests operate through "needles". These aren't sewing needles;
rather, they're screenshots of the virtual machine in some desired state with
some JSON metadata attached. This metadata defines certain areas to match or
ignore, and the test code can click matched areas. While needles are certainly
effective at interacting with the system exactly how a user would, they have
drawbacks. It's quite annoying to make and constantly update needles, as well
as keep them from breaking every time there are slight changes in user
interfaces.
Luckily for us, we already have a battle-tested way of interacting with user
interfaces for testing:
selenium-webdriver-at-spi
. It's already widely used across unit tests in KDE projects, hence our
decision to use it affords us a lot more flexibility, maintainability, and
consistency. It also enables app developers to run their own tests on KDE Linux
with openQA down the track.
Essentially, we have a Python unittest script on the system that we're testing
(see the sysext section below for details), which attaches itself to an
application. It then interacts with the app by leveraging the AT-SPI2
accessibility API to send clicks and read the screen, in a similar vein to
screen-reading software such as Orca.
Ephemeral workers in CI jobs
openQA instances usually have long-running workers that are hosted on servers.
It's a bit of a painful ordeal to get all that infrastructure up and running.
On top of that, hosted workers need to do an upload-download rigmarole
involving large assets from the server, such as the .iso files and the
generated hard disk. This makes things very slow for no good reason.
We already have CI runners that work perfectly well for this and can be spun up
when needed, giving us effortlessly simple scaling. So, we spin up an openQA
worker container in a CI runner, which submits jobs to the openQA server. It
has its own UUID, which is shared with the job, so the worker running in CI is
always assigned the right job.
This saves us from the bandwidth rigmarole because all the assets are generated
and consumed within the one container, so we can simply keep all the assets on
the worker and never upload them to the server. As a result, we save a lot of
storage space on the openQA server, so we can run it with fairly minimal
hosting requirements.
The use of systemd system extensions to inject tests
How do we actually get our Selenium tests on the system, you may ask? Enter the
humble
system extension
, or sysext, for short.
We include a few things in our sysext:
- The Python
unittests themselves. - A bootstrap script with some system configuration, so we have an appropriate
environment set up for testing.
- A
venv, so we can make use of the Python ecosystem. This is created in the
bootstrap script.
All of this is packaged up into an EROFS .img file, which we mount to the
worker's VM. This is then automounted by an associated udev rule in upstream
KDE Linux, with the bootstrapping script being triggered by an associated
service shortly afterward.
Since we have the capability to inject tests and configuration into the system,
we're able to test things that would otherwise be impossible to test with
needles. For example, we test if essential desktop processes have ever crashed,
if any systemd services failed, if networking works, and if commands we ship
with KDE Linux work properly. All of these tests leverage direct access to the
innards of the system.
Interaction with the system through SSH
To actually poke at the system and have the worker run these tests
sequentially, we need some way of interacting with the system. openQA provides
some facilities to interact with a serial terminal, but this proved to be very
fragile and unreliable, with buffering issues everywhere.
Instead, we set up SSH with our sysext and use the facilities provided by the
Python library Fabric to run all our tests in a
robust manner.
Each test runs in a transient systemd service created by systemd-run. This
runs the test as the intended user, groups its processes in a cgroup, gives it
an isolated journal stream for output, and returns its service exit status
synchronously. The harness can then collect the unit's journal even when the
test fails, and we keep everything neat and tidy.
Staging images before we publish, and how we test updates
To prevent users from downloading an image that still needs to be tested, we
create a staging directory on storage.kde.org, scoped to the imaging stage's
job ID, that stores the built artifacts in a directory tree. It has a layout
that mirrors the public-facing tree, so we can simply merge it in once tests
pass.
However, this throws a spanner in the works when we try to test system upgrades
because we obviously can't upgrade to an image that hasn't been published yet!
To fix this, the solution is simple. In the sysext, we simply point
systemd-sysupdate to the staging directory we've already created. This has
some drawbacks, though. For the moment, we can't test delta updates through
kde-linux-sysupdated. That shouldn't be too difficult to fix in the near
future, but we're waiting on KDE Linux to be entirely hosted on storage.kde.org
before we jump on that. The bigger issue here is that we really don't have a
good way of testing updates from the chunk store. A better story for this still
needs to be worked out, but for the time being the upgrade test is good enough.
What's to come
We have a few things we're aiming towards:
- As mentioned above, testing delta/chunked upgrades.
- Leveraging openQA to allow app developers to test their own apps atop KDE
Linux.
- Generalizing all our openQA glue so other projects can use the architecture
we've built.
- By extension, porting the aforementioned glue from admittedly fragile bash
scripts to Python or some other more appropriate language.
- Testing installs using manual partitioning and Full Disk Encryption.
…and probably many more things that we haven't thought of yet. Exciting times!
pinheiro
@xevrion:matrix.org
GSoC
And someday after that, Razer. I think they’d be receptive. And then Lenovo, HP, Dell, and Asus. I don’t believe this is far-fetched.
