Installation and development

The ci-tron container can in turn be run from a host OS, booted directly using boot2container or netbooted over the internet using iPXE boot server.

Requirements

The CI-tron gateway container is meant to be run on the gateway machine of a CI farm, and comes with the following requirements:

  • Hardware: Two network interfaces, one connected to the internet while the other one is connected to a switch where all the test machines to be exposed by the gateway are connected.

  • Volumes: The container requires a volume to store persistent data (mounted at /config), and optionally a temporary volume which acts as a cache across reboots (mounted at /cache).

  • Container: The container needs to be run as privileged, and using the host network

Building the container

The container image is built using rootless podman , and is provisioned using Ansible recipes (see the ansible/ subproject).

The following is a (likely incomplete) list of dependencies for a system running Debian/Ubuntu:

$ apt install buildah \
    git \
    jq \
    make \
    netcat-openbsd \
    podman \
    python \
    qemu-utils \
    qemu-system \
    skopeo \
    socat \
    wget

Note: In order to reduce round-trips to an external registry, a local registry is automatically started when running certain make targets.

The container image can be built with:

$ make gateway

Build options

  • V=1 Turn on more verbose logging messages in the build process

  • ANSIBLE_EXTRA_ARGS="-vvv ..." Pass any custom flags to ansible-playbook. Helpful for re-running only tagged roles in the ansible build, for example. You can also use this to override variables used by the ansible playbook, for example: ANSIBLE_EXTRA_ARGS="-e foo=bar"

  • IMAGE_NAME=localhost:8088/my/image The container name to tag the image with. WARNING: The image will automatically be pushed to the registry that got tagged! Defaults to localhost:8088/gfx-ci/ci-tron/gateway:latest.

Once completed, a container image will be generated, for example,

Successfully tagged localhost:8088/gfx-ci/ci-tron/gateway:latest
60cc3db9bedd2a11d8d61a4433a2a8e8daf35a59d6229b80c1fdcf9ece73b7ab

Notice that it defaults to a localhost registry. This is to save on bandwidth, since the ci-tron container is quite big.

Running the infrastructure

While not strictly required, enabling nested KVM on the host will dramatically improve the performance of a DUT VM booted from the VPDU. It can be enabled by configuring the relevant module parameter:

$ cat /etc/modprobe.d/kvm.conf
options kvm-intel nested=Y
options kvm-amd nested=1

Nested KVM support can be verified on the host by reading the “nested” parameter from sysfs:

$ cat /sys/module/kvm_*/parameters/nested
Y

We are now ready to start our virtual gateway machine, which will boot directly into the container we built in the previous section:

$ make vivian [GITLAB_REGISTRATION_TOKEN=...] [GITLAB_URL=https://gitlab.freedesktop.org]

Note 1: The GITLAB_REGISTRATION_TOKEN should be retrieved from the GITLAB_URL GitLab instance (documentation).

Note 2: options to vivian can be passed by setting VIVIAN_OPTS, for example: $ make VIVIAN_OPTS="--ssh-port=50022" ... vivian. See vivian/vivian -h for the full list of options it supports.

For information on starting virtual DUTs in the VM, see the section on “Spawning virtual DUTs” below.

The virtual testing recipes will fetch a Linux kernel and a boot2container ramdisk, and start the system. After the kernel boots and loads the ramdisk, the ramdisk will then pull the ci-tron container, and hand control to it.

N.B: Due to the stateful nature of the permanent partition in the VM’s tmp disk, it is wise to occasionally delete said disk and check a fresh boot continues to work as expected. This can be done using make clean.

By default, the boot logs will be redirected to your console until the SSH service comes up. At this point, vivian will switch the console to an SSH-based one.

In the boot logs, you should see a Linux kernel booting, then boot2container downloading the container we built, then systemd logs, and finally a dashboard which looks mostly green.

On some distros, you may receive the following error message when vivian tries to run qemu:

failed to create tun device: Operation not permitted
qemu-system-x86_64: -nic bridge,br=vivianbr0,mac=DE:AD:BE:EF:00:12,model=virtio-net-pci: bridge helper failed

Re-try after enforcing the user ownership on /usr/lib/qemu/qemu-bridge-helper on your test system:

# chmod u+s /usr/lib/qemu/qemu-bridge-helper

There are several options for getting a shell on the gateway:

  • create one from the dashboard by pressing Ctrl-b c in tmux

  • connect via SSH using make vivian-connect

  • use netcat to connect to the qemu console with nc localhost 4321

Spawning virtual DUTs

Right now, our gateway has no idea about any potential machine connected to its private network interface. Let’s boot one!

If the gateway is running in vivian, then virtual DUTs can be booted by using the vivian-dut make target. The OUTLET variable can be set to select the outlet on the VPDU:

$ make OUTLET=4 vivian-add-dut

Alternatively, you can use the dashboard to add/start a virtual DUT by selecting one with the arrow keys and submitting the DISCOVER button.

Once a DUT VM has started, output from it is redirected into a log file located at /cache/vpdu/dut-log-<mac address>-<date and time stamp>.log.

This log file is also tail’d in a new tmux window on the gateway’s console dashboard (CTRL + B in tmux then 0..n to select the window by ID).

If all went well, you should now see a machine appear in the “Machines” column of the dashboard, with the state DISCOVERING. After a short while, it should transition into TRAINING.

During the TRAINING step, the machine will be booted in a loop for a set amount of times in order to test the boot reliability. After the boot loop is complete, the machine will be marked as ready for testing and its state should change to IDLE.

Your first virtual test machine is now available for testing both locally, and on your chosen Gitlab instance.

Running a job on the virtual gateway

Now that you have at least one machine available, you may run jobs on it using the following command:

$ executorctl -e http://localhost:8000 run -c 10.0.2.2 -t virtio:family:VIRTIO $JOB_FILE

If all went well, congratulations! You seem to have a functional setup! Most of the steps above are amenable to further configuration. You are now in a position to play around and modify defaults to your testing requirements.

Hacking on the infrastructure

After making changes to Ansible or any other component of the gateway image, it is recommended to test the changes by either re-generating the gateway image, or running the ansible playbook against an already-running vivian instance:

$ make vivian-provision

If you want to iterate faster, you may limit the ansible job to some roles by using the TAGS= parameter:

$ make TAGS=dashboard,minio,executor vivian-provision

You can get a full list of available tags by running:

$ ansible-playbook --list-tags ansible/gateway.yml

Production deployment

The ci-tron infrastructure is meant to be netbooted over HTTPS, with the boot configuration being stored in a git repository for easy of update.

All of the steps are outlined at iPXE boot server.