Adding integration tests to your IOC¶

Note

This tutorial is a continuation of the Creating a StreamDevice IOC tutorial.

If you haven’t already, follow the StreamDevice tutorial first.

In this tutorial, you’ll learn how to test your created StreamDevice IOC by running it against the simulator, inside a declared NixOS VM, and checking that it behaves as expected.

This method of testing can then be automated by running it inside a Continuous Integration (CI) system.

Pre-requisites¶

Warning

Nix assumes you can run hardware-accelerated VMs, through KVM.

Make sure that you have KVM on your Linux machine by checking if the file /dev/kvm is present.

If the file is present, you can proceed to the next section.

If you don’t have KVM, and you’re running Nix on a physical machine, examine your firmware settings to see if you can enable hardware-accelerated virtualization. The setting can show up as:

Virtualization
Intel Virtualization Technology
Intel VT
VT-d
SVM Mode
AMD-v

If you don’t have KVM, and you’re running Nix on a virtual machine, check your firmware settings as said before, and look up your hypervisor documentation to enable nested virtualization.

If this doesn’t work, you can still proceed without hardware acceleration by adding this line to your nix.conf:

/etc/nix/nix.conf¶

extra-system-features = kvm

Note that this means much slower integration tests.

Writing the test¶

Through the NixOS testing framework, EPNix provides a way of specifying a machine configuration, and running a Python script that can do various kinds of testing.

With your IOC created during the Creating a StreamDevice IOC tutorial, you’ll see a checks/ directory, which is the place to add your integration tests.

These tests are imported using the epnix.checks.imports option.

For example, in the EPNix template, you’ll see in your flake.nix file:

flake.nix: importing an integration test¶

checks.imports = [ ./checks/simple.nix ];

The ./checks/simple.nix file should contain a NixOS test such as this:

checks/simple.nix: structure of a test¶

{ build, pkgs, ... }:

pkgs.nixosTest {
  name = "simple";

  nodes.machine = {config, ...}: {
    # Description of the NixOS machine...
  };

  testScript = ''
    # Python script that does the testing...
  '';
}

Running this test creates a NixOS virtual machine from the given configuration, and runs the test script.

The test script can, among other things, run commands on the machine, start, shut down, or reboot the machine.

Tip

The Python test script does not run on the virtual machine, but communicates with it.

If you want to run Python code on the VM machine, you need to package it and run it as a command.

For a more detailed overview of what you can put in the machine configuration, examine the NixOS documentation, or the Creating an Archiver Appliance instance tutorial.

Starting your IOC through systemd¶

First, you need to ensure that your IOC will start inside the VM.

In the default template, you’ll see this particular configuration:

checks/simple.nix: config for starting an IOC¶

  nodes.machine = {config, ...}: {
    imports = [
      epnix.nixosModules.ioc
      epnixConfig
    ];
    environment.systemPackages = [pkgs.epnix.epics-base];

    systemd.services.ioc = config.epnix.nixos.services.ioc.config;
  };

The first two emphasized lines are about importing the ability to define an IOC, and then importing your IOC configuration that you defined in your flake.nix.

Then, on the last emphasized line, the systemd service configuration generated by EPNix is used to generate ioc.service.

EPNix uses the configuration epnix.nixos.services from your flake.nix to figure out the name of your app and the name of your iocBoot folder.

Make sure yours is correct in your flake.nix:

flake.nix: configuring the name of your app and iocBoot folder for the test systemd service¶

        # Used when generating NixOS systemd services, for example for
        # deployment to production, or for the NixOS tests in checks/
        # ---
        nixos.services.ioc = {
          app = "example";    # Name of your app
          ioc = "iocExample"; # Name of your iocBoot folder
        };

Also take note of the package epics-base being installed, with the environment.systemPackages option. This enables you to use the caget, caput commands inside the VM.

Running the test¶

To run the test, run this command:

Running the test “simple”¶

nix build -L '.#checks.x86_64-linux.simple'

If you left the test script as-is, you should see that the test fails. That’s because the test script is currently not adapted to our IOC.

We’ll change it afterward, but for now in the logs you should see your IOC being run.

If you have several tests, you can run them all using:

Running all tests¶

nix flake check -L

Running the test interactively¶

It’s often desirable to run the VM interactively, to figure out what works and what doesn’t, before writing the test.

To do so, run:

Running the test “simple” interactively¶

nix run -L '.#checks.x86_64-linux.simple.driverInteractive'

This runs a Python shell prompt in the same environment as the test script. Any command run here is the same as running it in the test script, but interactively.

You can use the start_all() functions to start all VMs that you declared in nodes:

>>> start_all()

In our case, we only defined machine, so this starts a single VM, and runs your IOC inside it.

You can log in to that VM with the user root and no password. You can then run any command you want to inspect the state of the VM.

Integration VM screenshot showing the IOC running¶

Tip

If you have a non-English-language keyboard, change your keyboard layout inside the VM by using loadkeys.

For example, to set the keyboard to “french”:

[root@machine:~]# loadkeys fr

Tip

To exit the Python shell prompt, press Ctrl-d, then y.

Exiting the Python shell prompt automatically shuts down the VMs.

Adding the simulator¶

The simulator is a program listening on port 9999. Inside the test VM, it should be a program run by a systemd service.

Same as the IOC, you should use the systemd.services options.

Change your Nix test file like this:

Adding the simulator as systemd service, important changes emphasized¶

nodes.machine = {config, lib, ...}: {
  imports = [
    epnix.nixosModules.ioc
    epnixConfig
  ];
  environment.systemPackages = [pkgs.epnix.epics-base];

  systemd.services = {
    ioc = config.epnix.nixos.services.ioc.config;
    simulator = {
      serviceConfig.ExecStart = lib.getExe pkgs.epnix.psu-simulator;
      wantedBy = ["multi-user.target"];
    };
  };
};

The first emphasized line is about adding the lib argument used below.

The second set of emphasized lines is about creating the simulator.service systemd service. These lines will generate the following service file:

generated /etc/systemd/system/simulator.service¶

[Unit]

[Service]
# ...
ExecStart=/nix/store/...-psu-simulator/bin/psu-simulator

And this service is automatically started at boot, by being a dependency of multi-user.target.

The serviceConfig option adds configuration keys to the [Service] section. Here, we set ExecStart to main executable program of the psu-simulator package, by using the lib.getExe function.

A unitConfig for the [Unit] section also exists.

The [Install] section isn’t present in NixOS, because managed differently, by using options such as wantedBy, requiredBy, etc.

For more information, see the systemd.services options in the NixOS manual.

With this configuration, you can run the VM interactively (see Running the test interactively), and you should see the simulator up and running after booting.

Tip

If you make changes to your configuration, or your IOC, you don’t need to rebuild anything before running the nix run command.

Nix will by itself figure out what it needs to rebuild, and rebuild it before running the test.

Integration VM screenshot showing the simulator running¶

Writing the test¶

Now that the VM configuration is appropriate, you can start writing your test script.

Here is a sample of useful Python functions:

start_all()¶: Start all defined VMs

Machine.wait_for_unit(self, unit: str, user: str | None = None, timeout: int = 900)¶

Wait for a systemd unit to get into “active” state. Throws exceptions on “failed” and “inactive” states as well as after timing out.

Example¶

machine.wait_for_unit("ioc.service")

Machine.succeed(self, command: str, timeout: int | None = None)¶

Execute a shell command, raising an exception if the exit status is not zero, otherwise returning the standard output

Example¶

machine.succeed("caput VOLT 42")

Machine.wait_until_succeeds(self, command: str, timeout: int = 900)¶

Repeat a shell command with 1-second intervals until it succeeds.

Be careful of the s in succeeds.

Example¶

machine.wait_until_succeeds("caget -t my:stringout | grep -qxF 'expected value'")

Machine.fail(self, command: str, timeout: int | None = None)¶

Like succeed(), but raising an exception if the command returns a zero status.

Example¶

machine.fail("caget unknown-PV")

Machine.wait_for_open_port(self, addr: int | str, timeout: int = 900)¶

Wait until a process is listening on the given TCP port and IP address (default localhost).

Example¶

machine.wait_for_open_port(9999)

retry(fn: Callable, timeout: int = 900)¶

Call the given function repeatedly, with 1-second intervals, until it returns True or a timeout is reached.

Example¶

def check_value(_last_call: bool) -> bool:
    """Check whether the VOLT-RB PV is 42."""
    value = float(machine.succeed("caget -t VOLT-RB"))
    return value == 42.

retry(check_value, timeout=10)

subtest(name: str)¶

Group logs under a given test name.

To be used with the with syntax.

Example¶

with subtest("check voltage"):
    test_setting_voltage()
    test_voltage_readback()
    ...

You can also read more about the Python functions available in the test script in the NixOS tests documentation.

Example test script¶

Here an example test script that should work with your StreamDevice IOC:

checks/simple.nix: Example test script¶

start_all()

with subtest("check services"):
    machine.wait_for_unit("ioc.service")
    machine.wait_for_unit("simulator.service")
    machine.wait_for_unit("default.target")

    machine.wait_for_open_port(9999)

# Prefer using 'wait_until_succeeds',
# since the 'ioc.service' being active doesn't necessarily means
# that the IOC is initialized
machine.wait_until_succeeds("caget VOLT-RB", timeout=10)
machine.fail("caget unknown-PV")

with subtest("check voltage"):
    # Initial value is zero
    machine.succeed("caget -t VOLT-RB | grep -qxF '0'")

    machine.succeed("caput VOLT 42")

    def check_value(_last_call: bool) -> bool:
        """Check whether the VOLT-RB PV is 42."""
        value = float(machine.succeed("caget -t VOLT-RB"))
        return value == 42.

    retry(check_value, timeout=10)

Note that the script uses the wait_until_succeeds method and the retry function. This is because EPICS has few guarantees about whether it propagates changes immediately. It’s better to encourage the use of retries, instead of hoping the timing lines up.

After changing your test script, run your test as explained in Running the test.

Next steps¶

You can examine other NixOS test examples:

In the EPNix’ ioc/tests folder, for IOC tests,
In the EPNix’ nixos/tests folder, for EPICS-related NixOS services tests,
Or in the nixpkgs’ nixos/tests folder.

If you’d like to run a complete python script on the test VM, which can use Python dependencies such as pyepics, examine the guide Packaging Python scripts.

If you’re interested in adding unit tests, examine the Unit testing guide.

For all testing related guides, see Testing.