Hello Everyone, I am seeking assistance with a performance regression observed on an i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm. The Issue: A test suite containing 90 test cases, which previously took 10 minutes to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware. Observations:
- I'm not using any test_character_delay externally in my YAML. - Identical Logs: The "Fast" and "Slow" jobs produce identical serial output. - Recent Hangs: The test run has recently started to get stuck/hang mid-run, even though CPU usage is not at 100%.
Debugging Performed:
- Verified that CPU thermal throttling is not active. - Network latency and DNS resolution times appear normal. - No hardware errors are reported in dmesg. - Swapped power supplies and network cables with no improvement.
Questions:
1. What is the best way to debug "dead time" between lava-test-case signals? 2. Could the LAVA worker be experiencing serial buffer delays on this specific node that don't appear in the logs?
Thank you for your time.
Best Regards Pavan Kumar
Hi Pawan,
On Fri, Jan 16, 2026 at 9:45 AM pawanKumar pawan4749@gmail.com wrote:
Hello Everyone, I am seeking assistance with a performance regression observed on an i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm. The Issue: A test suite containing 90 test cases, which previously took 10 minutes to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware. Observations:
I'm not using any test_character_delay externally in my YAML. Identical Logs: The "Fast" and "Slow" jobs produce identical serial output. Recent Hangs: The test run has recently started to get stuck/hang mid-run, even though CPU usage is not at 100%.
Debugging Performed:
Verified that CPU thermal throttling is not active. Network latency and DNS resolution times appear normal. No hardware errors are reported in dmesg. Swapped power supplies and network cables with no improvement.
Questions:
What is the best way to debug "dead time" between lava-test-case signals?
I'm not sure what you mean by that, but you could try dowloading raw log. It contains timestamps for each line in the log.
Could the LAVA worker be experiencing serial buffer delays on this specific node that don't appear in the logs?
Very unlikely. Did you try to run the same test outside of LAVA (manually)?
Best Regards, Milosz
Hi Milosz, Thanks for your response! I ran the test manually outside of LAVA, and it runs as expected without any delay. I have identified that the issue is that the automation is interrupting at the first U-Boot instance instead of the second. I have other boards which have only a single *U-Boot* and are working fine. Because LAVA interrupts at the first U-Boot, the board is failing to enter the correct Factory Mode. As a result, background services remain active during the test, causing the delays and failures I am seeing. I need help configuring LAVA to ignore the first U-Boot prompt and only interrupt during the second U-Boot. Here are the differences in the boot logs: First U-Boot (Ignore this):
Reset cause: POR DRAM: 256 MiB ... Hit any key to stop autoboot: 0
Second U-Boot (Interrupt here):
Reset cause: WDOG DRAM: 512 MiB ... Hit any key to stop autoboot: 0
In my Jinja2 device configuration, I have tried using a regex to anchor the interrupt to the second boot's unique strings, but LAVA still triggers on the very first "Hit any key" it sees in the serial buffer: jinja2
{% set uboot_autoboot_prompt = '(?s)Reset cause: WDOG.*Hit any key to stop autoboot' %}{% set uboot_interrupt_character = ' ' %}{% set uboot_interrupt_newline = False %}{% set bootloader_prompt = '=>' %}
Could you please advise on the correct way to specify the interrupt-prompt in the Jinja2 template so that it specifically waits for the second instance? Best regards, Pavan Kumar Best Regards *Pavan Kumar*
On Fri, Jan 16, 2026 at 3:46 PM Milosz Wasilewski < milosz.wasilewski@foundries.io> wrote:
Hi Pawan,
On Fri, Jan 16, 2026 at 9:45 AM pawanKumar pawan4749@gmail.com wrote:
Hello Everyone, I am seeking assistance with a performance regression observed on an
i.MX6SOLO rev 1.4 (996 MHz) board in our LAVA farm.
The Issue: A test suite containing 90 test cases, which previously took 10 minutes
to complete, is now consistently taking 20 minutes. This issue is unique to this one specific board; other boards of the same type are still completing the run in 10 minutes using the same build/firmware.
Observations:
I'm not using any test_character_delay externally in my YAML. Identical Logs: The "Fast" and "Slow" jobs produce identical serial
output.
Recent Hangs: The test run has recently started to get stuck/hang
mid-run, even though CPU usage is not at 100%.
Debugging Performed:
Verified that CPU thermal throttling is not active. Network latency and DNS resolution times appear normal. No hardware errors are reported in dmesg. Swapped power supplies and network cables with no improvement.
Questions:
What is the best way to debug "dead time" between lava-test-case signals?
I'm not sure what you mean by that, but you could try dowloading raw log. It contains timestamps for each line in the log.
Could the LAVA worker be experiencing serial buffer delays on this
specific node that don't appear in the logs?
Very unlikely. Did you try to run the same test outside of LAVA (manually)?
Best Regards, Milosz
On Tue, Jan 20, 2026 at 8:58 AM pawanKumar pawan4749@gmail.com wrote:
Hi Milosz, Thanks for your response! I ran the test manually outside of LAVA, and it runs as expected without any delay. I have identified that the issue is that the automation is interrupting at the first U-Boot instance instead of the second. I have other boards which have only a single U-Boot and are working fine. Because LAVA interrupts at the first U-Boot, the board is failing to enter the correct Factory Mode. As a result, background services remain active during the test, causing the delays and failures I am seeing. I need help configuring LAVA to ignore the first U-Boot prompt and only interrupt during the second U-Boot. Here are the differences in the boot logs: First U-Boot (Ignore this):
Reset cause: POR DRAM: 256 MiB ... Hit any key to stop autoboot: 0
Second U-Boot (Interrupt here):
Reset cause: WDOG DRAM: 512 MiB ... Hit any key to stop autoboot: 0
In my Jinja2 device configuration, I have tried using a regex to anchor the interrupt to the second boot's unique strings, but LAVA still triggers on the very first "Hit any key" it sees in the serial buffer: jinja2
{% set uboot_autoboot_prompt = '(?s)Reset cause: WDOG.*Hit any key to stop autoboot' %} {% set uboot_interrupt_character = ' ' %} {% set uboot_interrupt_newline = False %} {% set bootloader_prompt = '=>' %}
1st issue with your regex is that it tries to match a single line. The text comes in multiple lines.
With a better regex it might work. This is what I tried: ^Reset cause:\s*WDOG[\s\S]*?Hit any key to stop autoboot:\s*0$
This is the site I use for prototyping regexes: https://regex101.com/
I'm not sure it works in LAVA, but it's worth to give it a try.
Best Regards, Milosz
Hi Milosz
This worked for me. {% set interrupt_prompt = '(?s)DRAM:[ ]+512 MiB.*?Hit any key to stop autoboot' %}
Best Regards, Pavan Kumar
On Tue, Jan 20, 2026 at 5:26 PM Milosz Wasilewski < milosz.wasilewski@foundries.io> wrote:
On Tue, Jan 20, 2026 at 8:58 AM pawanKumar pawan4749@gmail.com wrote:
Hi Milosz, Thanks for your response! I ran the test manually outside of LAVA, and it runs as expected without
any delay. I have identified that the issue is that the automation is interrupting at the first U-Boot instance instead of the second. I have other boards which have only a single U-Boot and are working fine.
Because LAVA interrupts at the first U-Boot, the board is failing to
enter the correct Factory Mode. As a result, background services remain active during the test, causing the delays and failures I am seeing.
I need help configuring LAVA to ignore the first U-Boot prompt and only
interrupt during the second U-Boot. Here are the differences in the boot logs:
First U-Boot (Ignore this):
Reset cause: POR DRAM: 256 MiB ... Hit any key to stop autoboot: 0
Second U-Boot (Interrupt here):
Reset cause: WDOG DRAM: 512 MiB ... Hit any key to stop autoboot: 0
In my Jinja2 device configuration, I have tried using a regex to anchor
the interrupt to the second boot's unique strings, but LAVA still triggers on the very first "Hit any key" it sees in the serial buffer:
jinja2
{% set uboot_autoboot_prompt = '(?s)Reset cause: WDOG.*Hit any key to
stop autoboot' %}
{% set uboot_interrupt_character = ' ' %} {% set uboot_interrupt_newline = False %} {% set bootloader_prompt = '=>' %}
1st issue with your regex is that it tries to match a single line. The text comes in multiple lines.
With a better regex it might work. This is what I tried: ^Reset cause:\s*WDOG[\s\S]*?Hit any key to stop autoboot:\s*0$
This is the site I use for prototyping regexes: https://regex101.com/
I'm not sure it works in LAVA, but it's worth to give it a try.
Best Regards, Milosz
lava-users@lists.lavasoftware.org
-
Milosz Wasilewski -
pawanKumar