Jump to content

Category:Hardware CI

From postmarketOS Wiki
CI-tron
Our testing harness PCB over the CI-tron dashboard, the infrastructure used to deploy test environments on real hardware
Our testing harness PCB over the CI-tron dashboard, the infrastructure used to deploy test environments on real hardware
Name CI-tron
License MIT
Latest Version https://gitlab.freedesktop.org/gfx-ci/ci-tron
FOSS

This category is meant to provide a collection of pages related to running tests on bare-metal hardware.

Overall architecture

Overview of the HW/SW testing architecture
Overview of the HW/SW testing architecture

There are 3 main components needed to provide hardware testing:

  • Devices under test: Devices connected to a gateway through a testing harness sufficient for remote control;
  • A testing gateway (CI-tron): Receives CI jobs from the different projects hosted by gitlab.postmarketos.org, deploys the suitable test environment to the devices under test, then deals with the power sequencing and boot sequence;
  • PostmarketOS's GitLab instance: Where the code to be tested, the test definitions, and the CI logic is located.

In order to add testing on a particular device that is not currently available in our GitLab instance, you will first need to deploy a CI-tron instance at your home, then add the device under test (DUT) to it.

CI-tron: The testing gateway

The logo of the CI-tron project. License: CC BY 4.0 Deed, Akseli Haapoja

The CI-tron project started in 2020 to provide an easier-to-use infrastructure for bare-metal testing than existing solutions by:

  • Providing a plug-and-play experience by automating best practices and minimizing manual configuration;
  • Using containers rather than rootfses for test environments;
  • Allowing both interactive and CI sessions.

This enables developers and power users to share their local devices, either for development or CI. The project's documentation is located at https://gfx-ci.pages.freedesktop.org/ci-tron/.

Installation

Warning WARNING: The installation instructions are currently for advanced-users only. A simpler system is currently being developed that will allow generating a boot image for your gateway from a website.

The installation instructions are located at https://gfx-ci.pages.freedesktop.org/ci-tron/docs/installation_and_development.html.

Adding devices under test to your gateway

Please follow the instructions found at https://gfx-ci.pages.freedesktop.org/ci-tron/docs/adding_duts.html.

Enabling device testing in GitLab

Warning WARNING: This section assumes that the devices under test you want to use are already exposed on gitlab.postmarketos.org using CI-tron

Now that our devices under test have been added to our GitLab instance, we need to instruct it how to submit test jobs to your CI-tron instance. The official way of doing so is documented at https://gfx-ci.pages.freedesktop.org/ci-tron/docs/forges_integration.html, and forms the basis of our implementation.

There are however a couple of differences compared to the official documentation:

  • The provided template files are vendored as-is in our GitLab instance to reduce our reliance on external projects
  • Common patterns have been moved to hidden jobs that any job can extend from in order to reduce code duplication

The common patterns can be found in the ci-common's ci-tron/common.yml, and can be included by adding the following excerpt at the top of your projects' .gitlab-ci.yml:

variables:
  # WARNING: it is strongly advised to use a commit hash instead of a branch name:
  CI_TRON_TEMPLATE_PROJECT: &ci-tron-template-project postmarketOS/ci-common
  CI_TRON_JOB_TEMPLATE_PROJECT_URL: $CI_SERVER_URL/$CI_TRON_TEMPLATE_PROJECT
  CI_TRON_JOB_TEMPLATE_COMMIT: &ci-tron-template-commit 7c95b5f2d53533e8722abf57c73e558168e811f3

include:
  - project: *ci-tron-template-project
    ref: *ci-tron-template-commit
    file: '/ci-tron/common.yml'

You can now use extends: to inherit from .pmos-ci-tron-job-v1, and can use all of the official options.

Generate and run the boot artifacts

To generate the boot artifacts needed by the devices under test, you may use the .pmos-ci-tron-build-boot-artifacts job from the ci-common repo (ci-tron/common.yml). The job requires you to set 3 variables:

  • DEVICE_NAME: The device name, in the wiki format (eg. qemu-amd64)
  • INSTALL_PACKAGES: The list of packages that should be installed
  • KERNEL_VARIANT: The kernel name (eg. lts, or virt), if applicable. Leave empty if only one kernel is supported.

The job also makes it possible to use packages or binaries built in previous CI steps into the generated kernel an initramfs. For that it considers 3 locations:

  • keys: Anything located under this directory will be copied to pmbootstrap local keys.
  • packages: Any package located under this directory will be copied to pmbootstrap local package repository. Together with keys, this can be used to use new packages instead of the ones in the binary repo.
  • rootfs: Any file located under this directory will be copied to the pmbootstrap rootfs chroot under the same relative path. This can be used to run custom binaries (e.g: mkinitfs, boot-deploy) instead of the ones provided by packages.

To boot the kernel and initramfs generated in the previous stage, you may use the .pmos-ci-tron-initramfs-test job from the ci-common repo (ci-tron/common.yml). The job requires to set the following elements:

  • tags:: The list of tags the Gitlab runner should have to run the job
  • variables:: All the CI-tron variables needed to specify which kernel and initramfs to load

Example

The following example will build the boot artifacts for the lts kernel of the qemu-amd64, then boot them on the device.

prepare-qemu-amd64:
  stage: hardware tests
  extends:
    - .pmos-ci-tron-build-boot-artifacts
  variables:
    DEVICE_NAME: qemu-amd64
    KERNEL_VARIANT: lts
    INSTALL_PACKAGES: device-${DEVICE_NAME} device-${DEVICE_NAME}-kernel-${KERNEL_VARIANT} postmarketos-mkinitfs-hook-ci
 
test-qemu-amd64:
  stage: hardware tests
  extends:
    - .pmos-ci-tron-initramfs-test
 
  # List of tags the device/runner should have to be picked by GitLab for testing. 
  # Alternatively you can use .pmos-ci-tron-runner-qemu-amd64 from common.yml
  tags:
    - cpu:arch:x86_64
    - virtio:codename:VIRTIO
  
  # Wait for the preapre-qemu-amd64 job to be over before executing this job
  dependencies: []
  needs:
    - job: 'prepare-qemu-amd64'
      artifacts: false
    
  # Specify how to get the kernel and initramfs to use for testing
  variables:
    CI_TRON_KERNEL__URL: "glartifact://build-qemu-amd64/${CI_TRON__PMB_EXPORT_PATH}/vmlinuz-${KERNEL_VARIANT}"
    CI_TRON_INITRAMFS__INITRAMFS__URL: "glartifact://build-qemu-amd64/${CI_TRON__PMB_EXPORT_PATH}/initramfs"

Enabling device testing in pmaports

Enabling testing on a particular device can be done by adding in your device's package a file called gitlab-ci.yml.j2 which defines all the test jobs for your device. This file will get included when generating the .ci/build-jobs.yaml.j2 template if the device needs testing (its package or one of its dependencies got modified).

Your jobs are expected to set all the necessary attributes to run jobs on a CI-tron-provided runner, but the following template variables and hidden jobs are provided to you as helpers:

  • Template variables (defined in .ci/lib/generate_build_jobs.py):
    • device: Instance of the Device class which represents the device under test
  • Hidden jobs (defined in .ci/build-jobs.yaml.j2):
    • .test-{{ device.name }}: Defines all the common parameters for testing this device (;
    • .ci-tron-device-boot-flow-separate-artifacts: Sets the kernel and initramfs URLs when the build job generates separate kernel and initramfs files as artifacts. If your device supports multiple kernel variants, make sure to set the KERNEL_NAME by referencing the KERNEL_VARIANT variable that is set in .test-{{ device.name }}.

As an example, here is how qemu-{amd64|aarch64} define their test job:

test-{{ device.name }}:
  extends:
    # Extended from jobs defined in ci-common's ci-tron/common.yml
    - .ci-tron-runner-{{ device.name }}  # Sets the runner tags

    # Extended from build-jobs.yaml.j2
    - .ci-tron-device-boot-flow-separate-artifacts
    - .test-{{ device.name }}
  variables:
    KERNEL_NAME: vmlinuz-$KERNEL_VARIANT
Note Rather than duplicating the gitlab-ci.yml.j2 file for all the devices that can make use of it, you may save the file in .gitlab-ci/packages/ then link to it in the device folder.

Debugging

If something goes wrong on device testing in CI, it might be needed to go do some debugging related to the actual devices.

Hardware tests always run at the end of the pipelines, and often depend on previous building and testing jobs to be successful. This guide does not cover how to debug those previous steps, like building packages in pmaports from MR changes, or compiling mkinitfs. However, once those are successful, there are 3 possible thing that can go wrong:

Only pmaports: dynamic pipeline generation

Testing pipelines in pmaports are generated dynamically, so that changing an armv7 devices only builds for that architecture, and does not run ci-tron tests for any other device. The pipeline is generated by the .ci/lib/generate_build_jobs.py script. The script requires jinja2 and pmbootstrap to be installed, but once they are it can be run passing a jinja template (unless doing changes, it should be .ci/build-jobs.yaml.j2) as a first argument, and the output file as a second argument. This way, it is possible to run the code generation locally, and inspect the resulting pipeline.

Preparing ci-tron artifacts for booting on devices

These artifacts should be generated using the .pmos-ci-tron-build-boot-artifacts hidden job from ci-common's ci-tron/common.yml, which in turn calls the build-boot-artifacts.sh script in the same repository.

If something goes wrong in this step, it should be possible to:

  • Run the build-boot-artifacts.sh script locally, making sure the required variables and paths contain the right data.
  • Push your changes to a branch in the ci-common repository (or a fork of it), and modify the following variables at the top of the .gitlab-ci.yml file to test your changes:
    • CI_TRON_TEMPLATE_PROJECT: &ci-tron-template-project my-user/ci-common
    • CI_TRON_JOB_TEMPLATE_COMMIT: &ci-tron-template-commit my-branch-name-or-commit-sha

Testing the artifacts on the real devices

If the testing itself fails, it might be necessary to look into the devices themselves. In this case, the test jobs should upload job_script.sh as an artifact. The script contains the details on the code run on the ci-tron gateway (see the architecture on top of this page). Therefore, anybody with access to the gateway can download that script, modify variables at wish (e.g: maybe you want to test on a different device in the same farm, or build and pass your own initramfs), and run on the device. For example, it might be desired to build a custom initramfs with the debug-shell mkinitfs hook installed. Then it can be built locally, and fetched in the device by providing a different value in CI_TRON_INITRAMFS__INITRAMFS__URL. For further details on variables to use, refer to the official documentation.

To be able to access the gateway, you must have access to it, preferably through wireguard. Giving access depends on the administrator. For farms managed by the postmarketOS-team, any team member can request access to the infra team. The details can be found in the private infra repo.

Subcategories

This category has only the following subcategory.