On Mon, 2 Sep 2019 at 14:47, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Milosz Wasilewski milosz.wasilewski@linaro.org Gesendet: Montag, 2. September 2019 15:32 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker?
On Mon, 2 Sep 2019 at 14:18, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Milosz Wasilewski milosz.wasilewski@linaro.org Gesendet: Montag, 2. September 2019 15:00 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker?
On Mon, 2 Sep 2019 at 13:47, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Lava-users lava-users-bounces@lists.lavasoftware.org Im Auftrag von Tim Jaacks Gesendet: Montag, 2. September 2019 10:41 An: Milosz Wasilewski milosz.wasilewski@linaro.org Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker?
>-----Ursprüngliche Nachricht----- >Von: Milosz Wasilewski milosz.wasilewski@linaro.org >Gesendet: Montag, 2. September 2019 10:16 >An: Tim Jaacks tim.jaacks@garz-fricke.com >Cc: lava-users@lists.lavasoftware.org >Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker? > >On Mon, 2 Sep 2019 at 07:45, Tim Jaacks tim.jaacks@garz-fricke.com wrote: >> >> >-----Ursprüngliche Nachricht----- >> >Von: Milosz Wasilewski milosz.wasilewski@linaro.org >> >Gesendet: Dienstag, 27. August 2019 17:42 >> >An: Tim Jaacks tim.jaacks@garz-fricke.com >> >Cc: Axel Lebourhis axel.lebourhis@linaro.org; >> >lava-users@lists.lavasoftware.org >> >Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker? >> > >> >On Tue, 27 Aug 2019 at 15:37, Tim Jaacks tim.jaacks@garz-fricke.com wrote: >> >> >> >> -----Ursprüngliche Nachricht----- >> >> >Von: Milosz Wasilewski milosz.wasilewski@linaro.org >> >> >Gesendet: Dienstag, 27. August 2019 16:32 >> >> >An: Tim Jaacks tim.jaacks@garz-fricke.com >> >> >Cc: Axel Lebourhis axel.lebourhis@linaro.org; >> >> >lava-users@lists.lavasoftware.org >> >> >Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker? >> >> > >> >> >On Tue, 27 Aug 2019 at 15:10, Tim Jaacks tim.jaacks@garz-fricke.com wrote: >> >> >> >> >> >> -----Ursprüngliche Nachricht----- >> >> >> >Von: Milosz Wasilewski milosz.wasilewski@linaro.org >> >> >> >Gesendet: Dienstag, 27. August 2019 15:59 >> >> >> >An: Tim Jaacks tim.jaacks@garz-fricke.com >> >> >> >Cc: Axel Lebourhis axel.lebourhis@linaro.org; >> >> >> >lava-users@lists.lavasoftware.org >> >> >> >Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker? >> >> >> > >> >> >> >On Tue, 27 Aug 2019 at 14:44, Tim Jaacks tim.jaacks@garz-fricke.com wrote: >> >> >> >> >> >> >> >> Hi Axel, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> thanks for your reply. I don’t want to force job submission to a SPECIFIC worker, I just want to make sure that all of the devices are on the SAME worker. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> When I submit a job, I don’t care on which specific worker it is scheduled, as long as all of the nodes are on the same worker. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Or differently spoken: as soon as one node is scheduled, can we define scheduling rules for all the remaining nodes? >> >> >> > >> >> >> >I don't think this is currently possible. So far the main objective was to grab devices as they're available to avoid deadlocks. >> >> >> > >> >> >> >Having said that I'm wondering why are you using multinode with LXC and HW? Is this for synchronization only? Maybe the test can be re-written in such way to not require multinode? By default LXC container is launched on the same worker that the DUT is connected to. >> >> >> >> >> >> Well, we need to access several hardware interfaces and software services running on the DUT from the outside in order to validate their functionality. We use an LXC device in order to achieve this. The LXC is the "outside world" and connects to the DUT, e.g. via telnet to validate the telnet server running on the DUT. >> >> >> >> >> >> Is this the wrong approach? How would the correct LAVA way be for this? >> >> > >> >> >I don't think it's wrong, but it drives you to the troubles with networks. As I wrote scheduling 'on the same worker' is not implemented in LAVA and I doubt it will change soon. Did you try 'secondary connections'? >> >> >https://master.lavasoftware.org/static/docs/v2/connections. >> >> >htm >> >> >l#i >> >> >n >> >> >dex >> >> >-12 >> >> >> >> How would this work in my scenario? As I understand it, a secondary connection is another login shell on the device. I don't need another login shell, I want to actually execute commands on a remote side (which is the worker) as part of the test. Is this possible with secondary connections? If yes, do you have an example which is similar to my use case? >> > >> >In order to connect from DUT to LXC you would need to configure SSH on the worker and LXC. I'm not sure if it's possible to do dynamically. >> >Surely it adds complexity to the setup. Maybe running the tests scripts 'on the other end' is a better approach? >> > >> >> >> >> >> >> >> >> >Main use case for LXC was flashing boards with fastboot over USB and accessing android with adb over USB. So this implicitly assumes there is a proper USB connection available in LXC container running the job. >> >> >> >> >> >> We are not using LXC for establishing the initial serial connection, as it is done with Android devices. The LXC is a separate device from LAVA's perspective, running test scripts in parallel to the test scripts on the DUT. They are synchronized using the MultiNode API. Is that understood? >> >> > >> >> >Yes, I got that. I'm just suggesting that you might try to reimplement the test and to the synchronization yourself. This way you avoid the problem of LXC 'device' being in different network than your DUT. >> >> >> >> But how would I execute commands on the worker then as part of the test? >> > >> >Ok, let's back up one step. In the 1st email you wrote the >> >test consists of 1. DUT running SFTP server 2. Client (LXC) >> >sending a file 3. comparison of md5 sum of source file and >> >transmitted copy >> > >> >I would do the following >> >1. boot DUT >> >2. start sftp server on DUT. Testing script has to determine >> >whether the daemon is running (pass) or not (fail) 3. start >> >LXC and transfer the file (you should know IP address of DUT >> >from it's device >> >dictionary) 4. using secondary connection/second uart connect >> >from LXC to DUT and check md5 checksum of the file 5. in LXC >> >check the checksum of the file and compare with checksum from >> >4 >> >> This is exactly how we implemented it, except for the "using secondary connection". How do I do this? We are using two LAVA devices for this purpose: the DUT (which has an actual hardware board type) and the LXC (which has board type lxc), resulting in a multi-node job. >> >> I still have no idea how to rewrite this without using multi-node. How do I start the LXC and execute commands on it in this case? Are there any examples where this is used in practice? > >I didn't want to create artificial example. Here is real life test job using LXC and running Android VTS: >https://paste.debian.net/1098366/ >You could add a test running on the target. For that you would need to add a 'test' section with namespace 'droid'. This would result in tests executed directly on the target. In this case namespaces are used to tell LAVA where to run the test. Some more details on that in the docs: >https://master.lavasoftware.org/static/docs/v2/deploy-lxc.html#in >dex >-1 There is an example on how to run tests either on LXC or on >target. >This is a single node test.
Thanks very much, Milosz. I will take a look at this.
I have experimented with this and got a single node job running with my DUT and an LXC. So far, so good. And good to see: the environment is applied to both of the devices, which comes in quite handy for my use case.
So I see the following options for my use case now:
- Use multiple test actions. To achieve this, the test script has to be split into parts, depending on which lines run on the DUT and which run on the LXC. Each block of lines has to be a separate test action and therefore a separate test script. This is not optimal, because the script gets scattered and semantically tied lines would be broken across multiple files. This leads to code which is hard to read and to maintain.
(correct so far?)
this sounds about right and I understand your pain with this approach.
- Use a single test action. The complete test script runs on either the DUT or the LXC. The commands for the other side have to be sent via SSH. While I can configure an SSH server on the LXC without problems, I do not see a way to tell the device the LXC's ip address, since we do not have the multi node API here. Any clues?
The other way round it might work, if and only if the DUT uses a static ip configuration. The test script runs on the LXC in this case. For commands executed on the DUT, the LXC can connect via SSH to the DUT and execute the command there.
What is your opinion on these two options? Do you see other ways?
I think this is a pretty good summary. I don't know of any way to tell DUT the LXC's IP address. In this case I would run all tests scripts on LXC.
Thanks for your quick and good replies. Is this a scenario that is used anywhere in practice? Are there any experiences regarding this technique? I wouldn’t consider this optimal either. There are tests where only one command is executed on the LXC (e.g. setting a relay configuration). Running the whole script on the LXC, while 99% of the commands are redirected to the DUT, seems a bit weird to me.
I'm not sure how much Android CTS/VTS is similar to your scenario. It uses this approach only.
Good to know, thank you.
Are relays switched during the tests? If they're switched before the test you can probably use user_commands and avoid LXC completely. Example here: https://staging.validation.linaro.org/scheduler/device/staging-lces2-01/devi... and here: https://staging.validation.linaro.org/scheduler/job/256811/definition
Oh wow, I wasn't aware of this feature up to now! Is this chapter linked anywhere in the documentation? I haven't stumbled upon this before, and I think I have read most chapters of the LAVA documentation multiple times. This should definitely get a more prominent place. Thanks, that helped a lot!
I think this was added recently and is supposed to replace the 'recovery mode'. There might be some bugs still. I'm struggling with one related to namespaces. I don't think the feature is described in the docs yet at least not very thoroughly. This is only reference I found: https://master.lavasoftware.org/static/docs/v2/actions-command.html. This instance runs the most recent version from master branch so the released versions might be a bit behind.
milosz
milosz
milosz
>milosz > >> >> > >> >That is what I meant by 'rewrite' the test. Would that work for you? >> > >> >milosz >> > >> >> >> >> >milosz >> >> > >> >> >> >> >> >> >milosz >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Regards, >> >> >> >> >> >> >> >> Tim >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Von: Axel Lebourhis axel.lebourhis@linaro.org >> >> >> >> Gesendet: Dienstag, 27. August 2019 15:34 >> >> >> >> An: Tim Jaacks tim.jaacks@garz-fricke.com >> >> >> >> Cc: lava-users@lists.lavasoftware.org >> >> >> >> Betreff: Re: [Lava-users] Multinode job: how can I assure that all devices are on the same worker? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Hi Tim, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> I think the best solution is to use tags. I do it to force job submissions to a specific worker. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Regards, >> >> >> >> >> >> >> >> Axel >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, 27 Aug 2019 at 15:28, Tim Jaacks tim.jaacks@garz-fricke.com wrote: >> >> >> >> >> >> >> >> Hello again, >> >> >> >> >> >> >> >> I have several test cases where we use LAVA multinode to test hardware and software interfaces externally. E.g. we have an SFTP server running on our DUT. In order to test that, we submit a test using two nodes: >> >> >> >> >> >> >> >> 1. The DUT >> >> >> >> 2. An LXC container >> >> >> >> >> >> >> >> The LXC device connects to the DUT via SFTP and uploads a file. Both sides determine the MD5 sum and the DUT compares them. >> >> >> >> >> >> >> >> This works as long as both the DUT and the LXC device are in the same network (or at least can reach each other via the network). >> >> >> >> >> >> >> >> Now there are more test cases which require additional hardware connections between the worker and the DUT, e.g. a serial interface test. The serial interface on the DUT is connected via an RS232-USB converter to the worker. The LXC can access this converter and send or receive data from the serial interface. >> >> >> >> >> >> >> >> This works as long as the LXC is running on the expected worker the serial interface of the DUT is connected to. >> >> >> >> >> >> >> >> As we are growing our lab, we will add more workers to our setup. There will be LXC devices on all of the workers. >> >> >> >> >> >> >> >> When submitting such a multinode job, which relies on hardware connections between the DUT and the worker, how can I make sure that the LXC part of the job is scheduled on an LXC device on the correct worker? >> >> >> >> >> >> >> >> >> >> >> >> Mit freundlichen Grüßen / Best regards Tim Jaacks >> >> >> >> DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring >> >> >> >> 2 >> >> >> >> 21079 Hamburg >> >> >> >> Direct: +49 40 791 899 - 55 >> >> >> >> Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com >> >> >> >> www.garz-fricke.com WE MAKE IT YOURS! >> >> >> >> >> >> >> >> Sitz der Gesellschaft: D-21079 Hamburg >> >> >> >> Registergericht: Amtsgericht Hamburg, HRB 60514 >> >> >> >> Geschäftsführer: Matthias Fricke, Manfred Garz, >> >> >> >> Marc-Michael Braun >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> Lava-users mailing list >> >> >> >> Lava-users@lists.lavasoftware.org >> >> >> >> https://lists.lavasoftware.org/mailman/listinfo/lava-u >> >> >> >> ser >> >> >> >> s >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> Lava-users mailing list >> >> >> >> Lava-users@lists.lavasoftware.org >> >> >> >> https://lists.lavasoftware.org/mailman/listinfo/lava-u >> >> >> >> ser >> >> >> >> s >> >> >> > >> >> > >> > > _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users