lava-users

lava-users@lists.lavasoftware.org

2 participants
949 discussions

LAVA 2026.06

by Remi Duraffort

Hi folks, The 2026.06 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Device-types ## New device-types * lenovo-500e-gen5-rev3-nissa * imx937-frdm * imx941-19x19-lpddr4-evk * imx942-19x19-lpddr4-evk * tegra234-3737-0000+3701-0000 * devices deriving from the new `base-qdl` template (including qcs6490-rb3gen2, which now supports minimal, u-boot and fastboot boot on top of qdl flashing) ## Renamed and removed device-types * `e850-96` renamed to `exynos850-e850-96` to follow the device tree name. * `juno-r2` dropped in favour of `juno`. The `juno-r2` template was an empty alias with no overrides, and a single `juno` device-type covers all Juno revisions. # Breaking changes ## VLANd support dropped The VLANd code has been finally removed from LAVA. This feature has been unmaintained for years and was no longer used by any active deployment. # Dispatcher ## QDL deploy and boot LAVA now supports [`qdl`](https://github.com/linux-msm/qdl/), the tool used to flash Qualcomm devices. The typical device life cycle is: * **deploy to qdl**: download a tarball containing all artifacts and add the LAVA overlay * **boot to qdl**: flash the board using the tarball contents. This step can be repeated to flash the various storage types (spinor, nvme, ufs, emmc, ...) * boot using another method depending on the software build (minimal, u-boot, fastboot, ...) `boot-to-qdl` also detects the installed `qdl` version and enables options accordingly (for example `--skipblock`, which is only available on v2.7 and above). The minimal version required to work with modern boards is 2.0. `qdl` is not yet packaged in Debian, so it must be available on the dispatcher or inside the container. It has been added to the suggested packages of `lava-dispatcher`. Example job flashing and booting a `qcs6490-rb3gen2`: ```yaml device_type: qcs6490-rb3gen2 job_name: qcs6490 qdl boot timeouts: job: minutes: 210 visibility: public actions: - deploy: to: qdl rootfs_image: "disk-sdcard.img2" overlay_path: "/" qcomflash: url: 'https://example.com/qcomflash.tar.gz' apply-overlay: true timeout: minutes: 200 - boot: method: qdl firehose_program: "prog_firehose_ddr.elf" rawprogram: "rawprogram0.xml" patch: "patch0.xml" storage: "ufs" path: "." timeout: minutes: 5 - boot: method: minimal prompts: - root@debian timeout: minutes: 3 ``` The device dictionary should include the right configuration to enter EDL mode. ```jinja {% set board_qdl_id = 'BA9B2FEB' %} {% set qdl_enter_commands = ['/usr/local/bin/tac-api.py --serial DP05DD00 --command bootToEDL'] %} ``` ## rclone download support A new `rclone` download backend allows downloading images via [rclone](https://rclone.org/), giving access to 70+ cloud storage backends (S3, Google Drive, Azure, SFTP, ...) through a unified URL scheme: ```yaml - deploy: to: tmpfs images: rootfs: url: rclone://s3remote/bucket/images/rootfs.img.gz compression: gz ``` Configuration can be provided from environment variable or the secret block: ```yaml secrets: rclone_env: RCLONE_CONFIG_S3REMOTE_TYPE: "s3" RCLONE_CONFIG_S3REMOTE_PROVIDER: "AWS" RCLONE_CONFIG_S3REMOTE_ACCESS_KEY_ID: "AKIAIOSFODNN7EXAMPLE" RCLONE_CONFIG_S3REMOTE_SECRET_ACCESS_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" RCLONE_CONFIG_S3REMOTE_REGION: "us-east-1" actions: - deploy: to: tmpfs images: rootfs: url: rclone://s3remote/bucket/images/rootfs.img.gz compression: gz ``` ## Docker login support LAVA is now able to pull docker image from private docker registry using username and password credentials provided in the job definition. To pull images from a password protected registry add a login section with the registry domain name, username and password. ```yaml - test: docker: image: example.com/my-adb-image login: registry: example.com user: foobar password: my_password ``` ## Test definition tarballs: strip-components Tarballs fetched via `from: url` (such as GitHub and GitLab archives) often wrap their contents in a top-level directory. LAVA now support a new optional `strip-components` parameter (passed through to `tar --strip-components`), so the test definition path can be given relative to the repository root. This can be used with: ```yaml - test: definitions: - name: smoke-tests from: url repository: https://github.com/Linaro/test-definitions/archive/2019.03.tar.gz compression: gz strip-components: 1 path: automated/linux/smoke/smoke.yaml ``` ## Faster tarball extraction Test definition tarballs are now unpacked using the `tar` binary instead of the pure-Python `tarfile` module. This is significantly faster and also allows unpacking Zstd-compressed archives. ## flasher: skip image download Add an option to the `flasher` deploy action to not download an image but to make it available in the substitution. ```yaml - deploy: to: flasher images: direct_url_image: url: https://example.com/image.img.xz download: false ``` The image is not downloaded; the URL from the job is passed directly to the substitution. This avoids extra downloads and uploads when the flasher device can fetch the image itself. ## Device metadata in the shell environment LAVA test action now export the device hostname and device type as environment variables: * `LAVA_DEVICE_HOSTNAME` * `LAVA_DEVICE_TYPE` ## fastboot: pre-os-command always run `pre-os-command` is now always run on `boot-to-fastboot`, including when the action uses a Docker container. This fixes booting boards (such as RB1) using fastboot with an NFS-mounted rootfs. ## Bug fixes * **lava-test-runner**: use KMSG in the runner when the test definition uses it, so the `EXIT` signal cannot arrive before `ENDRUN` and cause a run to be wrongly marked as failed. * **Overlay path**: `CompressOverlay` now stores the overlay under the full `lava_test_results_dir` instead of just its basename, fixing a path mismatch when the variable is overridden in the context. * **QEMU overlay**: the guest overlay is now created as `ext2` again, so DUTs with older kernels can mount it (a recent `e2fsprogs` enables `metadata_csum_seed` by default). * **vexpress**: default `test_character_delay` to `boot_character_delay` so tests are paced like boot and deploy, helping slow consoles (such as Juno's). # Server ## Notifications: remote artifact tokens Notification callbacks can now resolve secret tokens using the `RemoteArtifactsAuth` table in addition to the legacy XML-RPC `AuthToken` table. Previously only XML-RPC tokens could be used as callback secrets, which required administrative privileges to set. Regular users can now manage their own GitLab/API tokens for job notifications without administrative intervention. # Packaging ## PEP 621 project metadata The build now uses PEP 621 compliant project metadata and `setuptools-scm` for automatic version derivation from git tags. Dependencies are organised by component (dispatcher, server, dispatcher-host, coordinator) so only what is needed can be installed, with a `full` extra pulling in everything: ```sh uv sync --extra full --extra test pip install lava[server] ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

1 week, 3 days

LAVA 2026.05

by Remi Duraffort

Hi folks, The 2026.05 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # New debugfs implementation for generating mountable images Instead of using `libguestfs` LAVA uses `debugfs` command from `e2fsprogs`. This should in theory result in fewer bugs compared to `libguestfs`. To fallback to `libguestfs` implementation is in place if `debugfs` command is not found. An optional `overlay_backend` key in the deploy stage allows explicit selection of the backend (`auto`, `e2fsprogs`, or `guestfs`): ```yaml - deploy: overlay_backend: e2fsprogs images: rootfs: url: https://example.com/rootfs.ext4 format: ext4 overlays: lava: true ``` # Device-types ## New device-types * r9a09g057-rzv2h-evk * imx952-15x15-evk # Server ## Job cancellation tracking Admin log entries are now recorded when a job is cancelled, making it easier to track who cancelled a job and when. ## Database New indexes on `state` and `health` columns improve performance when filtering jobs. The deprecated `index_together` on `TestJob` has been replaced with dedicated per-column indexes. ## 500 error responses Server 500 errors now return JSON instead of HTML when the client sends an `Accept: application/json` header (as `lava-worker` and `lava-run` now do). This prevents raw HTML from appearing in job logs. ## UI improvements Enable search for device-type and all workers pages. Also order tags by name (case insensitive) in the devices page. ## Future proofing A lot of changes has been done recently to remove deprecated and old django and python features from the LAVA source code. This was possible due to the removal of support for Debian 10. Multiple fixes for Debian 13 are also part of this LAVA release: * Fixed `text_offset` not being part of device type schema even thought it was used by some u-boot devices. * Fixed REST API pages having invalid layout on Debian 13. * Fixed various deprecation warnings. * Fixed LDAP settings not working on Debian 13. * Fix OIDC settings not working on Debian 13. # Dispatcher ## Docker test action Docker test actions no longer require the boot action to stop at a shell prompt. This allows booting minimal systems without a proper shell and running tests exclusively from the Docker test shell on the worker. ## fastboot ### fastboot autodetect When using fastboot auto-detect feature, LAVA will now run fastboot in a container if that's requested in the job definition. Prio to this release, LAVA was ignoring the docker configuration for running the auto-detection step. ### DUT power sequence LAVA is now downloading and preparing image prior to powering DUT on. This aligns fastboot deployment with the typical download-then-deploy pattern used by the other deployment methods in LAVA. ## Test services Environment variables from the device dictionary and job definition are now written to a `.env` file in the test service repository. Docker Compose reads this file automatically, making it easy to pass per-device information to service containers. Device dictionary example: ```jinja2 {% set environment = { 'BOARD_QDL_ID': 'D902AA38'} %} ``` Docker Compose example to expose the variable to the container: ```yaml services: srv1: environment: - BOARD_QDL_ID ``` ## u-boot `#conf@XXX` is no longer required for every device when using the `bootm` command with FIT images. The config suffix is only appended when `bootm_device_tree_config` is set in the device dictionary. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

1 month, 2 weeks

Issue: "Secret filtering" feature leads to "Unable to parse invalid logs" error when using empty secret

by armand.beneteau＠iot.bzh

Hello all, I am reporting here an issue we observed, in my company, that has started with the "2026.02" release, and more accurately, with the "Secret filtering" feature. Starting from this release, we've noticed that some of our jobs logs were not displayed in the UI. Instead we just had the following error message: "Unable to parse invalid logs: This is maybe a bug in LAVA that should be reported.". After looking for the source of the problem, we found out that the corresponding temporary logs files (example: /var/lib/lava/dispatcher/worker/tmp/2/logs.yaml) were full of "[MASKED]" string. This leads to a problem during the YAML parsing done a bit further by Lava. And this is at this moment that we understood that the problem happens when one of our "secrets" variable was empty. So, I guess this is a case that you did not cover in your tests, and this is why I wanted to warn you. Best regards, Armand Beneteau

1 month, 3 weeks

LAVA 2026.04

by Remi Duraffort

Hi folks, The 2026.04 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Security fixes ## Private job information leaked in device table The device table was displaying the submitter and description of the currently running job even when the user did not have permission to view that job. Now users without view permission will only see that the device is "Running" a job, without any job details. # Breaking changes ## LXC support dropped LXC support has been removed. It has been unmaintained and deprecated for several years. Users of LXC jobs must migrate to [Docker-based alternatives]( https://lava.readthedocs.io/en/latest/user/advanced-tutorials/running-arbit… ). ## Database migrations squashed (upgrade path limited) `lava_results_app` migrations have been squashed to remove non-portable PostgreSQL-specific SQL and Python. The squashed migration is now the new initial migration. **Direct upgrades from LAVA 2020.09 or earlier are no longer supported.** The oldest supported upgrade path is from LAVA 2020.12 (the version shipped with Debian 11 Bullseye). Administrators running older instances must upgrade to at least 2020.12 before upgrading to this release. ## Worker auto-registration disabled by default Worker auto-registration is now disabled by default due to security implications. If `WORKER_AUTO_REGISTER_NETMASK` is set and a HTTP proxy is in use, LAVA sees the proxy's IP address (possibly localhost), which could allow unauthorized workers to register and obtain tokens. To re-enable auto-registration, explicitly configure `WORKER_AUTO_REGISTER_NETMASK` in your settings. ## Debian 11 support As of 2026.04 release Debian 11 (Bullseye) is no longer supported. Administrators are requested to migrate to at least Debian 12 (Bookworm). # LAVA Server ## REST API New endpoints for managing groups and users are available in the v0.2 REST API: * `GET/POST /api/v0.2/groups/` * `GET/PUT/PATCH/DELETE /api/v0.2/groups/<id>/` * `GET/POST /api/v0.2/users/` * `GET/PUT/PATCH/DELETE /api/v0.2/users/<id>/` ## XMLRPC API Fix device owner not being updated when calling `lavacli devices update --user`. The `physical_owner` field was not being updated by the API call. # LAVA Worker ## Test services LAVA now supports running test services on the worker. Test services are Docker Compose-based containers started on the LAVA worker, enabling custom services for test definitions. For example, LAVA can now start an OTA server that a DUT under test can connect to. Job definition example: ```yaml - test: services: - name: my-service compose: version: "3" services: ota-server: image: my-ota-server:latest ports: - "8080:8080" definitions: - repository: https://example.com/tests.git from: git path: testdefs/ota.yaml name: ota-test ``` ## Expected test results When an expected test list is provided, any test case not in that list now results in a **fail** (previously it was only logged). This makes it easier for regression-detection tools to spot unexpected test cases that should be investigated. ## GRUB ## Overriding `net_commands` Device dictionaries can now override `net_commands` to set custom GRUB network commands. This is useful for debugging network boot issues (e.g. adding `net_ls_addr`): ```jinja2 {% set net_commands = ['net_ls_addr', 'dhcp'] %} ``` ### Error detection LAVA will now automatically catch a large set of error raised by Grub while trying to boot a kernel. The following error patterns are now matched: * `missing (.*) symbol.` * `couldn't send network packet` * `destination unreachable` * `couldn't autoconfigure` * `timeout reading` * `file '(.*)' not found` * `no such device` * `disk '(.*)' not found` * `unknown filesystem` ## udev: kernel netlink fallback for containers The udev monitoring code now falls back to kernel netlink if no udev events are received. This allows containerized dispatchers to detect USB devices without requiring `--network=host` or `CAP_NET_ADMIN`, since containers don't propagate udev events across namespaces. sysfs attributes are used as a fallback for `ID_SERIAL_SHORT` when udev properties are unavailable. ## LAVA_JOB_ID environment variable `LAVA_JOB_ID` is now available in the `lava-run` environment, accessible to `user_commands` the same way as secrets declared in the job definition. This is useful for running commands that are job-specific. # Documentation The documentation has been again vastly improved with changes to: * Admin docs: hardware recommendations, typical deployment topologies, LDAP configuration, first steps, recommended security settings * Developer docs: contribution guide, code of conduct, guide for adding new actions, new device type guide, developer first steps * User docs: job debugging guide, job results, LAVA auth, user permissions, common LAVA use cases # CI More pylint checks enabled and code brought into compliance: `W1510`, `W0611`, `W1115`, `E0307`, `E0611`, `E1136`. The minimum checked Python version has been increased to **3.11**. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

2 months, 3 weeks

Issue: Log processor bottleneck and deploy-flasher timeouts on high-frequency output (LAVA 2026.02 )

by pawanKumar

Dear LAVA Community, I am reaching out to report a regression (or significant change in behavior) regarding how the LAVA dispatcher handles high-frequency terminal output and escape sequences during flashing operations. Background: After upgrading from LAVA 2022.06 to a recent version LAVA 2026.02 , our flashing jobs—which utilize mfgtoolcli (NXP) to write images to i.MX6/8 devices—have begun failing consistently with a deploy-flasher timeout. The Issue: The flashing tool outputs a high-frequency progress bar using carriage returns (\r) and ANSI escape codes (e.g., \e[1F, \e[2K). 1. Log Bloat: In the older version, these were handled gracefully. In the current version, every escape sequence is captured as a new log entry, resulting in logs exceeding 5MB for a single flash. 2. Dispatcher Lag: The dispatcher appears to bottleneck while processing this flood of data. This "processing lag" causes the internal action timer to hit the default 500s limit, even if the physical flashing process succeeds locally on the worker. 3. Timeout Overrides: We’ve observed that deploy-flasher often ignores the timeout values specified in the Job YAML, defaulting to 500 seconds unless explicitly overridden in the Device Dictionary. Error Examples: finish programming rootfs-a>> [1F [2K 1 100% [============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= ... continues >] Donedeploy-flasher timed out after 500 seconds>> uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.243-0-ged48c51>> >> Wait for Known USB Device Appear... [?25l [1F [1F [1F [1F [1F [1F1:241312 1/ 0 [ ] >> [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [ Steps we have attempted without success: - Setting TERM=dumb and piping to cat (the tool continues to output sequences). - Using tr -d '\r' (reduces lines but the data volume still triggers the timeout). - Increasing timeouts in the Job YAML (often ignored by the sub-action). Question: Has there been a change in the dispatcher's log-collection priority or buffering logic that would cause this bottleneck? Are there recommended "best practices" for handling tools that force interactive progress bars in the newer LAVA architecture? Best Regards Pavan Kumar

3 months

LAVA 2026.02

by Remi Duraffort

Hi folks, The 2026.02 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Security fixes Some security issues has been fixed in this release and we advice to upgrade as soon as possible. The details will be available later on. # Device-types ## New device-types * asus-CX3402CVA-brya * lenovo-chrome-2in1-14iru10-brox * mt8196-rauru-navi-sku1 * r9a07g043u11-smarc * r9a07g044c2-smarc * sc7180-trogdor-wormdingler-rev1-boe # Documentation migration Documentation v3 continues to be updated. All `deploy`, `boot` and `test` methods are now documented and published at https://lava.readthedocs.io/en/latest under **Technical references**. # Debian support Debian 11 (Bullseye) is being phased out. 2026.02 is the last LAVA release to support it. Starting from 2026.02 the oldest supported Debian release will be 12 (Bookworm) # Docker * All LAVA base images are now based on Debian 13 (Trixie) `slim` * Debian 11 (Bullseye) was dropped from all CI jobs * `wget` tool was added to lava-dispatcher-base image # Job logs ## Secret filtering LAVA is now filtering the job logs to remove sensitive data. This is still a work in progress that will be improved in the following releases. With this new feature, values of the variables defined in `secrets` dictionary will be replaced in the job output log with string `[MASKED]`. # Dispatcher changes ## Downloading artifacts Set max_retries to 1 by default when downloading an artifacts. The max_retries value can be set in the job definition using failure_retry key. Example: ```yaml - deploy: failure_retry: 2 images: image: url: https://example.com/foo ``` ## Deploy character delay Add support for `deploy_character_delay` to add delay between each character when interacting with the serial during deployment. To use it add to the device dictionary (in milliseconds): ```jinja2 {% set deploy_character_delay = 30 %} ``` Defaults to 30ms delay for vexpress devices. ## fastboot Improve fastboot device detection by making parsing more generic. This should now work with multiple fastboot versions. ## FVP Use `find` to locate armlm binary instead of hardcoded path to provide compatibility with more FVP models. ## multinode Fix `lava-role list` to properly match the documentation. Also fix multinode collate function used to replace placeholdeers like `$ipaddr` in synchronization messages. ## pyocd Fix flashing with pyocd with latest pycocd versions. ## Transfer overlay Allow to transfer the overlay using zmodem when network is not available. In the job definition, add: ```yaml - boot: transfer_overlay: transfer_method: zmodem unpack_command: tar -C / -xzf ``` The worker and target device **must** have respectively the `sz` and `rz` utilities installed. This is typically provided by: `lrzsz` package on Debian-based or Fedora-based systems. ## uuu Allow to use `uniquify` parameter for uuu deployment action. This is useful when using `downloads://` urls. # Server changes ## Restricting callback URLs By default, notification callbacks can target any URL. Administrators can restrict which hosts are permitted by setting `CALLBACK_ALLOWED_HOSTS` in `/etc/lava-server/settings.yaml` or a file in `/etc/lava-server/settings.d/`: ```yaml CALLBACK_ALLOWED_HOSTS: - "example.com" - "*.example.com" - "*.ci.example.com" ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

3 months, 3 weeks

EXT4 officially supported, but are you using anything else?

by Ben Copeland

Hello all, I am preparing a pull request in LAVA and would like to know which filesystems people are using. The PR aims to drop libguestfs and eventually move to e2fsprogs-based tooling. LAVA officially supports ext4, but unofficially, are people using anything else? Could you let me know if any users are using XFS, BTRFS (or other variants)? Any feedback would be appreciated! Regards, Ben

4 months, 1 week

DRF-filters deprecation

by Milosz Wasilewski

Hi, When LAVA introduced REST framework, it allowed for cross model filtering using djangorestframework-filters package. v0.2 LAVA API still allows that. Unfortunately the package has been unmaintained for the last 5 years: https://pypi.org/project/djangorestframework-filters/#history Without updates it will soon become incompatible with DRF and Django itself. In fact Debian has to carry patches to make it still working. On top of that there are performance issues with queries made by the DFR-filters. For these reasons it was decided that DRF-filters will be removed as LAVA dependency starting with version 0.3 of the API. v0.2 will remain supported for now. There is no end of life date agreed yet. In practice this means cross-model queries won't be supported any more in v0.3. As a tradeoff performance of single model queries should improve. If there are any concerns regarding this change, please reply to this thread. Best Regards, Milosz

5 months, 1 week

Auto-registration disabled by default

by Milosz Wasilewski

Hi, Starting from 2026.02 worker autoregistration will be disabled by default. Details can be found in this merge request: https://gitlab.com/lava/lava/-/merge_requests/2081 This may break some setups, so please be aware when upgrading. Local setting can still overwrite the default if needed. Best Regards, Milosz

5 months, 1 week

Download action default retry

by Milosz Wasilewski

Hi, As of now the default retry for download action is set to 3 (three). This is an outlier as all other actions have retry set to 1. Some time ago I complained about it in the list. There is also a MR posted to address the problem: https://gitlab.com/lava/lava/-/merge_requests/3012 I just rebased it on top of 2026.01 and I'm planning to merge it in 2026.02 release. It will still be possible to set retry to 3 in download action with addition of failure_retry in a test job: - deploy: failure_retry: 2 images: image: url: 'https://example.com/foo.tgz' timeout: minutes: 20 to: downloads Please reply here if this change is going to fundamentally break your setup. IMHO the worst that can happen is the download timeouts increase 3x which in my experience should not cause problems. I don't know all potential setups, so please reply if this is a breaking change for you. Best Regards, Milosz

5 months, 2 weeks

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users