lava-users

lava-users@lists.lavasoftware.org

2 participants
936 discussions

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Neil Williams

On Mon, 3 Jul 2017 23:50:25 +0300 Paul Sokolovsky <paul.sokolovsky(a)linaro.org> wrote: > Hello Milosz, > > I appreciate getting at least some response ;-). Some questions > however could use a reply from LAVA team, I guess. > > On Mon, 3 Jul 2017 13:34:49 +0100 > Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: > > [] > > > > jobs submit a number of tests to LAVA (via > > > https://qa-reports.linaro.org/) for the following boards: > > > arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an > > > example of cumulative test report for these platforms: > > > https://qa-reports.linaro.org/lite/zephyr-upstream/tests/ > > > > > > That's really great! (Though the list of tests to run in LAVA > > > seems to be hardcoded: > > > https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_t…) > > > > > > > It is, as I wasn't really sure what to test. The build job needs to > > prepare the test templates to be submitted to LAVA. In case of > > zephyr each tests is a separate binary. So we end up with the > > number of file paths to substitute in the template. Hardcoding was > > the easiest thing to get things running. But I see no reason why it > > wouldn't be changed with some smarter code to discover the > > binaries. The problem with this approach is that some of these > > tests are just build time. They have no meaning when running on the > > board and need to be filter out somehow. Running the build tests within the Jenkins build makes a lot of sense. Typically, the build tests will have a different command syntax to the runtime tests (otherwise Jenkins would attempt to run both), so filtering should be possible. If the build tests are just a different set of binary blobs from the runtime tests, that may need a fix upstream in Zephyr to distinguish between the two modes. > I see, that makes some sense. But thinking further, I'm not entirely > sure about "build only" tests. Zephyr's sanitycheck test has such > concept, but I'd imagine it comes from the following reasons: a) > sanitycheck runs tests on QEMU, which has very bare hardware support, > so many tests are not runnable; b) sanitycheck can operate on > "samples", not just "tests", as sample can be interactive, etc. it > makes sense to only build them, not run. > > So, I'm not exactly sure about build-only tests on real HW boards. The > "default" idea would be that they should run, but I imagine in > reality, some may need to be filtered out. But then blacklisting > would be better approach than whitelisting. And I'm not sure if > Zephyr has concept of "skipped" tests which may be useful to handle > hardware variations. (Well, I actually dunno if LAVA supports skipped > tests!) Yes, LAVA has support for pass, fail, skip, unknown. For POSIX shell tests, the test writer just calls lava-test-case name --result skip For monitor tests, like Zephyr, it's down to the pattern but skip is as valid as pass and fail (as is unknown) for the result of the matches within the pattern. > Anyway, these are rough ideas for the future. I've spent couple of > weeks of munging with LITE CI setup, there're definitely some > improvements, but also a Pandora box of other ideas and improvements > to make. I'm wrapping up for now, but hope to look again in some time > (definitely hope to look before the Connect, so we can discuss further > steps there). In the meantime, I hope that more boards will be > installed in the Lab and stability of them improves (so far they seem > to be pretty flaky). There are known limitations with the USB subsystem and associated hardware across all architectures, affecting test devices and the workers which run the automation. LAVA has to drive that subsystem very hard for both fastboot devices and IoT devices. There are also problems due to the design of methods like fastboot and some of the IoT support which result from a single-developer model, leading to buggy performance when used at scale and added complexity in deploying workarounds to isolate such protocols in order to prevent interference between tests. The protocols themselves often lack robust error handling or retry support. Other deployment methods which rely on TFTP/network deployments are massively more reliable at scale, so comparing reliability across different device types is problematic. > [] > > > > - test: > > > monitors: > > > - name: foo > > > start: "" > > > end: Hello, ZJS world! > > > pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\. > > > > > > So, the "start" substring is empty, and perhaps matches a line > > > output by a USB multiplexer or board bootloader. "End" substring > > > is actually the expected single-line output. And "pattern" is > > > unused (dunno if it can be dropped without def file syntax > > > error). Is there a better way to handle single-line test > > > output? > > > > You're making a silent assumption that if there is a matching line, > > the test is passed. In case of other tests (zephyr unit tests), it's > > not the case. The 'start' matches some line which is displayed when > > zephyr is booting. End matches the line which is displayed after all > > testing is done. The pattern follows the unit test pattern. > > Thanks, but I'm not sure I understand this response. I don't challenge > that Zephyr unittests need this support, or the way they're handled. > LITE however needs to test more things than "batch" Zephyr unittests. > I present another usercase which albeit simple, barely supported by > LAVA. (That's a question to LAVA team definitely.) LAVA result handling is ultimately a pattern matching system. Patterns must have a unique and reliable start string and a unique and reliable end string. An empty start string is just going to cause misleading results and bad pattern matches as the reality is that most boards emit some level of random junk immediately upon connection which needs to be ignored. So there needs to be a reliable, unique, start string emitted by the test device. It is not enough to *assume* a start at line zero, doing so increases the problems with reliability. > > > > Well, beyond a simple output matching, it would be nice even for > > > the initial "smoke testing" to actually make some input into the > > > application and check the expected output (e.g., input: "2+2", > > > expected output: "4"). Is this already supported for LAVA "v2" > > > pipeline tests? I may imagine that would be the same kind of > > > functionality required to test bootloaders like U-boot for Linux > > > boards. > > > > I didn't use anything like this in v2 so far, but you're probably > > best off doing sth like > > > > test 2+2=4 PASS. > > > > than you can easily create pattern that will filter the output. In > > case of zephyr pattern is the only way to filter things out as there > > is no shell (?) on the board. > > So, the problem, for starters, is how to make LAVA *feed* the > input, as specified in the test definition (like "2+2") into a board. That will need code changes, so please make a formal request for this support at CTT https://projects.linaro.org/servicedesk/customer/portal/1 so that we can track exactly what is required. > > As there were no reply from LAVA team (I may imagine they're busy with > other things), I decided to create a user story in Jira for them, as I > couldn't create a LAVA-* ticket, I created it as > https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go > unnoticed and LAVA team would get to it eventually. That JIRA story is in the LITE project. Nobody in the LAVA team can manage those stories. It needs a CTT issue which can then be linked to the LITE story and from which a LAVA story can also be linked. Sadly, any story in the LITE project would go completely unnoticed by the LAVA software team until it is linked to CTT so that the work can be prioritised and the relevant LAVA story created. That's just how JIRA works. > > > > > milosz > > Thanks! > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > _______________________________________________ > linaro-validation mailing list > linaro-validation(a)lists.linaro.org > https://lists.linaro.org/mailman/listinfo/linaro-validation -- Neil Williams ============= http://www.linux.codehelp.co.uk/

8 years, 4 months

Re: [Lava-users] Fwd: OE/Yocto testcase

by Robert Berger

Hi, On 2017-07-04 15:18, Agustin Benito Bethencourt wrote: > Dear CIP friends, > > please check below. This is a use case we will meet in a few > weeks/months. It is important to see others walking the same route. My simple tests are running now thanks to the help of the nice people from the #linaro-lava chat. 1) Crazy me decided to use upstream u-boot 2017.05 instead of running some ancient version from 2014;) 1.1) which happens to have a different AUTOBOOT_PROMPT than the one Lava expects "Press SPACE to abort autoboot in %d seconds\n" instead of "Hit any key to stop autoboot" so (since) I would like to be able to stay as close as possible to upstream LAVA 2017.06 I patched u-boot[1]. Note that this could be fixed with LAVA as well - interrupt_prompt: {{ interrupt_prompt|default('Hit any key to stop autoboot') }} 1.2) also the SYS_PROMPT of upstream u-boot is different than the one expected by LAVA and again I made a u-boot patch[2]. Note that this could be fixed with LAVA as well - {% set bootloader_prompt = bootloader_prompt|default('U-Boot') %} 2) After some searching it turned out that LAVA set some funny variables in u-boot which made my kernel crash. (Crazy me decided to use a 4.9.x uImage without baked in address). Adding this: {% set base_high_limits = false %} to my bbb03.jinja2 file fixed it. ... obviously ;) Regards, Robert [1] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b… [2] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b…

8 years, 4 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Paul Sokolovsky

Hello Milosz, Thanks for routing this thread to lava-users - when I made initial post to linaro-validation, I check my archive and so that e.g. Neil posts there frequently, but I missed that it's not official LAVA list. On Mon, 3 Jul 2017 22:25:31 +0100 Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: [] > > So, I'm not exactly sure about build-only tests on real HW boards. > > The "default" idea would be that they should run, but I imagine in > > reality, some may need to be filtered out. But then blacklisting > > would be better approach than whitelisting. And I'm not sure if > > Zephyr has concept of "skipped" tests which may be useful to handle > > hardware variations. (Well, I actually dunno if LAVA supports > > skipped tests!) > > As far as I can tell they acutely run on the board, but usually output > just 'Hello world!' or sth similar. As we discussed with Kumar, this > is still OK. What Kumar requested (and I still didn't deliver) is that > whenever the LAVA test job completes, the test should be considered > 'passed'. So we wouldn't have to do any parsing of patterns. I'm not > sure if that will work, but it's worth to try. Hmm, I wonder what would be criteria for being "failed" for such tests... Anyway, thanks for sharing - I'm not familiar with all Zephyr tests/samples myself, will keep in mind such issues when looking into them. [] > > more boards will be installed in the Lab and stability of them > > improves (so far they seem to be pretty flaky). > > > > You're absolutely right. This is a pretty big task to work on and IMHO > requires someone to work full time at least for couple of weeks. The > second part is also true, the boards don't behave as they should. I > guess Dave can elaborate more on that. I can only see the result - > boards (frdm-kw41z) don't run the tests they're requested. Matt Hart actually showed me a ticket on that, so at least it's confirmed/known issue in works. But even with arduino_101 and frdm_k64f, I hit cases more than once when board(s) were stuck for extended time, but still were routed jobs to (which either failed or timed out). So, there may be problem with health checks, which either don't run frequently enough or aren't robust enough. arduino_101 is all the lone one, so if something happens to it, there's no backup. Etc, etc. [] > > So, the problem, for starters, is how to make LAVA *feed* the > > input, as specified in the test definition (like "2+2") into a > > board. > > Right. What I proposed was coding all the inputs in the test itself. Well, that would require bunch of legwork, but the biggest problem is that it wouldn't test what's actually required. E.g. both JerryScript and MicroPython Zephyr ports are actually interactive apps working over serial connection. And functional testing of them would be feeding something over this serial connection and checking that results are as expected. I'll keep in mind idea of "builtin" tests though. Thanks! -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

8 years, 4 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Milosz Wasilewski

I was too quick to hit reply. CCing lava-users for comments from LAVA team. milosz On 3 July 2017 at 21:50, Paul Sokolovsky <paul.sokolovsky(a)linaro.org> wrote: > Hello Milosz, > > I appreciate getting at least some response ;-). Some questions however > could use a reply from LAVA team, I guess. > > On Mon, 3 Jul 2017 13:34:49 +0100 > Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: > > [] > >> > jobs submit a number of tests to LAVA (via >> > https://qa-reports.linaro.org/) for the following boards: >> > arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an >> > example of cumulative test report for these platforms: >> > https://qa-reports.linaro.org/lite/zephyr-upstream/tests/ >> > >> > That's really great! (Though the list of tests to run in LAVA seems >> > to be hardcoded: >> > https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_t…) >> > >> >> It is, as I wasn't really sure what to test. The build job needs to >> prepare the test templates to be submitted to LAVA. In case of zephyr >> each tests is a separate binary. So we end up with the number of file >> paths to substitute in the template. Hardcoding was the easiest thing >> to get things running. But I see no reason why it wouldn't be changed >> with some smarter code to discover the binaries. The problem with this >> approach is that some of these tests are just build time. They have no >> meaning when running on the board and need to be filter out somehow. > > I see, that makes some sense. But thinking further, I'm not entirely > sure about "build only" tests. Zephyr's sanitycheck test has such > concept, but I'd imagine it comes from the following reasons: a) > sanitycheck runs tests on QEMU, which has very bare hardware support, > so many tests are not runnable; b) sanitycheck can operate on "samples", > not just "tests", as sample can be interactive, etc. it makes sense to > only build them, not run. > > So, I'm not exactly sure about build-only tests on real HW boards. The > "default" idea would be that they should run, but I imagine in reality, > some may need to be filtered out. But then blacklisting would be better > approach than whitelisting. And I'm not sure if Zephyr has concept of > "skipped" tests which may be useful to handle hardware variations. > (Well, I actually dunno if LAVA supports skipped tests!) > > Anyway, these are rough ideas for the future. I've spent couple of > weeks of munging with LITE CI setup, there're definitely some > improvements, but also a Pandora box of other ideas and improvements to > make. I'm wrapping up for now, but hope to look again in some time > (definitely hope to look before the Connect, so we can discuss further > steps there). In the meantime, I hope that more boards will be > installed in the Lab and stability of them improves (so far they seem > to be pretty flaky). > > [] > >> > - test: >> > monitors: >> > - name: foo >> > start: "" >> > end: Hello, ZJS world! >> > pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\. >> > >> > So, the "start" substring is empty, and perhaps matches a line >> > output by a USB multiplexer or board bootloader. "End" substring is >> > actually the expected single-line output. And "pattern" is unused >> > (dunno if it can be dropped without def file syntax error). Is >> > there a better way to handle single-line test output? >> >> You're making a silent assumption that if there is a matching line, >> the test is passed. In case of other tests (zephyr unit tests), it's >> not the case. The 'start' matches some line which is displayed when >> zephyr is booting. End matches the line which is displayed after all >> testing is done. The pattern follows the unit test pattern. > > Thanks, but I'm not sure I understand this response. I don't challenge > that Zephyr unittests need this support, or the way they're handled. > LITE however needs to test more things than "batch" Zephyr unittests. I > present another usercase which albeit simple, barely supported by LAVA. > (That's a question to LAVA team definitely.) > >> > Well, beyond a simple output matching, it would be nice even for the >> > initial "smoke testing" to actually make some input into the >> > application and check the expected output (e.g., input: "2+2", >> > expected output: "4"). Is this already supported for LAVA "v2" >> > pipeline tests? I may imagine that would be the same kind of >> > functionality required to test bootloaders like U-boot for Linux >> > boards. >> >> I didn't use anything like this in v2 so far, but you're probably best >> off doing sth like >> >> test 2+2=4 PASS. >> >> than you can easily create pattern that will filter the output. In >> case of zephyr pattern is the only way to filter things out as there >> is no shell (?) on the board. > > So, the problem, for starters, is how to make LAVA *feed* the > input, as specified in the test definition (like "2+2") into a board. > > As there were no reply from LAVA team (I may imagine they're busy with > other things), I decided to create a user story in Jira for them, as I > couldn't create a LAVA-* ticket, I created it as > https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go > unnoticed and LAVA team would get to it eventually. > >> >> milosz > > Thanks! > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

8 years, 4 months

OE/Yocto testcase

by Robert Berger

Hi, I succeeded in running the example Lava tests[1][2] on a Beagle Bone Black in my lab. Now I would like to test a core-image-minimal I built with Yocto and cooked up something like this[3][4]. The result is unfortunately this[5]. I guess one of the problems I have is this: lava-test: # ls -l /lava-38/ export SHELL=/bin/sh /lava-38/bin/lava-test-runner /lava-38/0ls: cannot access '/lava-38/': No such file or directory lava-test: # export SHELL=/bin/sh lava-test: # /lava-38/bin/lava-test-runner /lava-38/0 -sh: /lava-38/bin/lava-test-runner: No such file or directory Regards, Robert [1] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [2] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [3] https://github.com/RobertBerger/git.linaro.org-lava-team-refactoring/blob/m… [4] https://github.com/RobertBerger/git.linaro.org-qa-test-definitions/blob/mas… [5] https://pastebin.com/14pjxhiN

8 years, 4 months

job yaml

by Li, Xuelian

Hi,all: If devive have boot successed，can I skip deploy and boot step in job.yaml? This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

8 years, 4 months

Is it possible to get data in device dictionary inside a job?

by Li, Yifan2

Hello LAVA experts, I am currently working with LAVA and was asked to find is there a way to get data from device dictionary inside a running job. Details like below: I have a device name demo-01, and its device-dictionary have one line like "{% set my_property = 'my_prop' %}". Then I have a job running on demo-01 device, and I would like to use string 'my_prop' to passed into a script during running. Is it possible to get device-dictionary data directly from job definition(Job Submitter webpage) or test definition(yaml file)? If yes, how could I do this? If not, is there any good way you would like to share to solve this problem? Thanks and Best Regards, Yifan

8 years, 4 months

About Test Duration

by kingboli

Hi all, Is there any limitation on test duration when using LAVA to do tests? Recently, I found my tests were automatically canceled after running for 24 hours and I could not find any clue about why the job was canceled. Can anybody give me some help? (Very sorry my LAVA version is V1.) 12794.0 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM INFO: Cancel operation 12794.1 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM DEBUG: finally status fail 12794.2 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM WARNING: [ACTION-E] boot_image is finished with error (Cancel). 12794.3 <LAVA_DISPATCHER>2017-06-25 02:31:42 PM WARNING: Target : power_off being call Thanks.

8 years, 4 months

libzmq issue between Debian Jessie and Stretch

by Guillaume Tucker

Hi, We've hit an issue when running lava-master on Debian Jessie and lava-slave on Debian Stretch, after a few minutes the slave would stop working. After some investigation, it turned out to be due to a difference of the libzmq versions in Jessie (4.0.5+dfsg-2) and Stretch (4.2.1-4) causing some protocol errors. The line that detects the error in Stretch is: https://github.com/zeromq/libzmq/blob/7005f22726d4a6ca527f27560a0a132394fdb… This appears to be due to how the "once" counter gets written into memory and into the zmq packets: the libzmq version from Jessie uses memcpy whereas the one in Stretch calls put_uint64. As a result the byte endianness has changed from little to big, causing the packets to work until "once" reaches 255 which translates into 0xff << 56, after which it overflows to 0 and causes the error. This is not a LAVA bug as such, rather a libzmq one, but it impacts interoperability between Jessie and Stretch for LAVA so it may need to be documented or resolved somehow. We've installed the new version of libzmq onto our Jessie servers to align them with Stretch; doing this does fix the problem. Best wishes, Guillaume

8 years, 4 months

Lab tool for DUT manual usage,

by Li, Yongtao

Hi All, We have a Lab and the DUTs in the Lab will be used for both automation testing(LAVA) and manual usage(development, debug, manual testing). And we will develop a tool for the manual usage, please find the basic requirements of the tool in the attachment. I list possible solutions about how to develop the lab tool and let the tool cooperate with LAVA on the same Lab hardwares. which one is better? could you give your suggestions? 1. Develop the lab tool based on LAVA framawork. Need to modify LAVA DB: add some new tables for network ports, port attribution, port connection, usage log, notes, also need to add columns to existing tables lava_scheduler_app_device. Need to modify LAVA command line, add functions like lava-server reserve dut1, lava-server connect dut1. Need to add new codes for the features which LAVA doesn't support, part of codes may be reused, I need to look into LAVA and check how to reuse them. The tool will be developed based on https://github.com/Linaro/lava-server and will be installed on LAVA master, right? Most probably we will maintain the codes change in local repository because it is difficult to upstream thousands of codes changes to linaro master repository. We need to merge the changes in master repository to local repository. 2. Develop the lab tool as a separated tool and use separated DB. There are two ways to avoid the DUT conflicts between the lab tool and LAVA: a) Lab tool DB maintain all DUTs, LAVA DB also maintain all DUTs, when a user want to do automation testing on one DUT, he need to reserve this DUT via lab tool in advance, then he can run automation testing on the specified DUT. b),Divide the DUTs to two groups, one group is for automation testing and it will be added to LAVA DB, the other one group is for manual usage and will be added to lab tool DB. Another question about how to link/unlink two network ports dynamically(see requirement#7 in attachment) in automation testing. I am not sure whether LAVA can support this feature later, the simple way is supporting it in test script: Do the link/unlink in test script: subprocess.Popen(["lab-tool","link","dut1.eth_1g_1","dut2.eth_1g_1"]) , and test script needs to get DUT name from LAVA in advance. Does this work? BR Yongtao This email is intended only for the named addressee. It may contain information that is confidential/private, legally privileged, or copyright-protected, and you should handle it accordingly. If you are not the intended recipient, you do not have legal rights to retain, copy, or distribute this email or its contents, and should promptly delete the email and all electronic copies in your system; do not retain copies in any media. If you have received this email in error, please notify the sender promptly. Thank you.

8 years, 4 months

← Newer
1
...
82
83
84
85
86
87
88
...
94
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users