On Mon, 11 Feb 2019 at 15:47, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Neil Williams neil.williams@linaro.org Gesendet: Montag, 11. Februar 2019 16:06 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode pipeline synchronisation / timeouts
On Mon, 11 Feb 2019 at 14:53, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Neil Williams neil.williams@linaro.org Gesendet: Montag, 11. Februar 2019 15:00 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode pipeline synchronisation / timeouts
On Mon, 11 Feb 2019 at 11:46, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Neil Williams neil.williams@linaro.org Gesendet: Montag, 11. Februar 2019 11:38 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode pipeline synchronisation / timeouts
On Mon, 11 Feb 2019 at 10:29, Tim Jaacks tim.jaacks@garz-fricke.com wrote: > > Hello everyone, > > I am having problems with timeouts when using the LAVA multinode protocol. Assume the following minimal pipeline with two nodes (device = DUT, remote = some kind of external hardware interfacing with the DUT): > > - deploy: > role: device > - boot: > role: device > > - deploy: > role: remote > - boot: > role: remote > > - test: > role: remote > > - test: > role: device > > What I would expect: The device is booted first, then the remote is booted.
No. That is a fundamental misunderstanding of MultiNode. One MultiNode test job submission creates multiple LAVA test jobs. Those test jobs are scheduled only when all nodes defined in the target_group are ready to start. When scheduled, all test jobs start immediately.
Once any of the test jobs has got to a Lava Test Definition 1.0 shell, then that shell can use the MultiNode API to send a message declaring that it is ready and then use the same API to wait for whatever other nodes are required.
> Afterwards, the tests run on both nodes, being able to interact with each other. > The pipeline model seems to be implemented in a way that each node has its own pipeline. This kind of makes sense, because the tests of course have to run simultaneously. > > However, in my case booting the device takes a lot more time than booting the remote. This makes the 'test' stage on the remote run a lot earlier than the 'test' stage on the device. > > My problem: How do I define a useful timeout value for the 'test' stage on the remote?
This has nothing to do with timeouts. This is about the MultiNode API. https://master.lavasoftware.org/static/docs/v2/multinodeapi.html #mul tinode-api
I am already using the MultiNode API and my test cases are working in general. I also understood that a MultiNode job submission creates sub-jobs for each node.
Are you using lava-send and lava-wait? Have you ensured that the message sent by one node is different to the message sent by any other node?
Messages are cached by the coordinator, so both nodes can lava-send and then lava-wait. The timeout for the test action containing the lava-wait on the fastest node must allow for all the time required for the slower node to get to the point of sending it's message, as well as however long it will take to execute it's own test action.
Maybe I have not made clear enough what my problem is. A 'test' stage has to have a timeout value, right? And this timeout value is defined in the test job description, so it has to be known before the job is submitted.
The test action timeout needs to allow for all the operations within that test action to occur, this includes any time spent waiting.
Let's assume a useful timeout value for my test is 1 minute. That means: if both test stages (on the device and on the remote) start at the same time, the test should be completed in less than 1 minute, otherwise some kind of failure can be assumed.
BUT: in reality, the two test stages do NOT start at the same time. There is no way to predict or influence the point of time a stage starts.
Separate the synchronisation test shell from the other actions - as recommended in the docs. The test shell which waits for the other node to be ready uses the lava-multinode protocol timeout to determine how long to wait.
The MultiNode API needs to be used to enforce that the two test stages DO start at the same time.
Whichever node is fastest sends a signal that it's ready with lava-send and then uses lava-wait to halt further execution until a similar message is sent by the other node. When the second node sends it's signal, the first node continues operation and all subsequent test actions now run as if both nodes had started at the same time.
Then the total timeout for all test definitions within the test action needs to cover the time required for the waiting and for the action itself. This can be calculated from existing single-node test jobs in some cases or by trial and error.
If my deploy and boot stages on the device take 3 minutes, while they take only 1 minute on my remote, the test stage on the remote times out before the test stage on the device has even started.
The test stage timeout needs to include the expected time required for the synchronisation calls.
I created a simple test job as an example: https://pastebin.com/Gtk1xZ6N There are two nodes. One of them sends a signal to the other one via the MultiNode API. But before, the first node runs an additional test stage containing a sleep. If this stage is removed, the job terminates successfully. If it is left in there, the test stage on node2 times out, even though I did not change any of the timeout values.
Of course it will timeout, the action has been extended beyond it's timeout. 180 seconds > 2 minutes.
- test: role: node1 timeout: minutes: 2 definitions:
- repository: metadata: format: Lava-Test Test Definition 1.0 name: sleep
You must change the timeout values if you materially change the amount of time that the test will normally take to execute. By adding 3 minutes to the execution time, you must increase the timeout for that same action by at least 3 minutes.
Okay, that was a quick-and-dirty mistake. I wanted to make the sleep 90 seconds, so that it is within the action timeout.
Do I miss something here? What is the supposed way to handle this?
MultiNode is complex. You must consider the synchronisation to be part of the test action - it must be included in the total timeout. LAVA cannot tell if the test action is just sleeping or working.
> Obviously I have to take the boot time difference between the two nodes into account. This seems counter-intuitive to me, since the timeout value should affect the actual test only.
> What happens if I use an image on the device which takes even a lot more time to boot?
You extend the lava-multinode protocol timeout and you extend the timeout of the test action of the faster role to allow for the time that will be consumed waiting for the other role.
> Or if I insert more testcases on the device which do not need the remote before? In both cases I would have to adjust the timeout value for the remote 'test' stage.
Yes. That is how the synchronisation works. One side waits and one side sends the message. The time required for the wait must be included in the timeout of the action doing the waiting. The timeout set in the protocol is how long each wait can be. Test actions may wait several times for different services and the test action timeout must take this into account. The timeout applies to everything done by the test action, not just any specific test definition.
Thanks, that is what I thought. And I indeed find this counter-intuitive and it seems like a design compromise to me. An action timeout should be affected only by the action for which it is defined.
I don't follow.
You've defined an operation within the action which takes longer than you have defined for that action, so yes, the action timeout is determined by the time taken for the action to complete all the operations that you have defined.
I am sorry, this was a mistake, as I noted above. I wanted to make the sleep operation 90 seconds. Please see this corrected job description:
Semantically I have two tests here:
- Some "long" test running on node1 only (sleep 90 seconds)
- Some "quick" multi-node test running on node1 and node2
The "quick" test should actually run within less than a second, since it only sends a message from one node to the other. The 1-minute timeout should therefore be sufficient for it.
You have fallen foul of a subtle problem with Lava Test Definition 1.0 support. Two consecutive test actions (in this case, for the same role) will be run as a single test action. It's made worse by over-use of inline definitions.
inline/sleep.yaml is set for role: node1 inline/send-signal.yaml is set for role: node1
You should simply concatenate those two into a single test action because this is how the tests actually get executed:
- test: role: node1 timeout: minutes: 2 definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: sleep description: "Sleep 90 seconds" run: steps: - sleep 90 from: inline name: sleep path: inline/sleep.yaml - repository: metadata: format: Lava-Test Test Definition 1.0 name: send-signal description: "Send signal to node2" run: steps: - lava-send somesignal from: inline name: send-signal path: inline/send-signal.yaml
Do not use test: test: - it does not do what you expect. When using Lava Test Definition 1.0, use a single test action for each role and list multiple repositories. Only if you do an explicit reboot can you use a second test action for Test Definition 1.0.
However, the "long" test runs on node1 before. And this has to be taken into account when defining the timeout for the node2 action, which semantically belongs to a different test.
Because these are not actually different test actions. You only get a discrete timeout here if you have a *different* action in between. It comes down to how the V1 overlay has to be created. If you have a single overlay, you only get a single test-runner and therefore only a single timeout.
This is counter-intuitive to me. In an ideal world (and in my opinion), the 1-minute-timeout should always work for this quick test, no matter what other tests run before (or how long the deploy/boot process is).
Lava Test Definition 1.0 is not capable of doing this - the one part of LAVA which did not get changed in the V2 migration was the Lava Test Definition 1.0. Some subtle issues remain because of this.
It is a subtle problem because as the code develops and we add different test actions (like monitor & interactive), those do not suffer from this problem. However, this behaviour is fundamental to how Lava Test Definition 1.0 needs to operate and a lot of other test jobs depend on this. We are planning a Lava Test Definition 2.0 which will drop a lot of the remaining inheritance. No timescale yet.
We construct test jobs from templates, so we cannot always predict how many testcases a job contains and in which order they are. Both of this actually affects timeout values, though.
Actually, you can. Linaro's QA team does this a lot. What you need is data. You need to build and run the jobs (manually in some cases, without using templates) and then build and run more jobs using the templates. Start off with long timeouts and use the data set to reduce the timeouts. This is how the documentation describes using Timeouts too. You are being too aggressive with the timeouts here. You need to give the jobs more breathing space.
A single test action can include multiple test definitions. The test action timeout must allow for all those definitions to complete.
definitions:
- repository: foo # takes 3 minutes
- repository: bar # takes 25 minutes
- repository: baz # take 30 seconds
The initial recommended timeout for the test action would therefore be 30 minutes. Any less than that and you're likely to get intermittent timeout failures.
Timeouts do NOT exist to artificially speed up test jobs. There needs to be a balance between a timeout which is too aggressive and a timeout which leaves the devices blocked on a dead job for too long.
That is exactly what I want to achieve. In my case above I would have to increase the timeout for the "quick" test to 2 minutes, even though it actually runs in less than a second, just because there is a longer test running before. And I would have to increase it even more, if I run more long tests before. This does not sound reasonable to me. I want to have a timeout which fits the test case and is independent of what happens before.
You are thinking of test actions which don't actually exist. Your example only has two test actions - one which will run inline/sleep.yaml immediately followed by inline/send-signal.yaml under a single timeout and one which will run inline/receive-signal.yaml in a completely separate test job on the other node, under a different timeout, using a different overlay and a different lava-test-runner.
You can test this by running a singlenode test job - if you do test: test: then only one test action will execute. This was deliberate - in V1, having test: test: would cause an invisible reboot between the two. In V2, that reboot must be explicit. You don't need a reboot here, so you get only a single test action and a single test timeout.
Also, on average, a complete set of actions takes 27 minutes and 14 seconds, a timeout of 30 minutes is reasonable if the dispersion is fairly tight. If however there is a large variation between values within the grouping behind that average, then the actual timeout for all test jobs in that group could easily need to be 1 hour.
If that second test definition above, bar, is actually a MultiNode synchronisation, then that still comes under the same rule. The MultiNode protocol timeout is entirely separate. It might need to be 15 or 20 minutes for test jobs with a large disparity between the boot sequences of different nodes. It still gets taken into account in the test action timeout. Equally, the protocol timeout may only be 5 minutes but the test definition does many wait and synchronisation operations, starting up various services.
Timeouts cannot necessarily be predicted in advance, especially with Multinode there is a period of optimisation involving trial and error to get the best timeouts. It may require running a hundred test jobs or more to get the right balance.
However, I just found out that I can simply remove the action timeout on the remote node completely (I thought it was mandatory, but obviously it is not). This at least serves as a workaround for my specific case.
No. That just puts in the default timeout of 30 seconds.
In my case above it leads to a succeeding job, so it cannot be 30 seconds. Is it 30 minutes maybe?
Depends on other defaults, e.g. in the device templates. If no override exists, any timeout defaults to 30 seconds.
What you're doing with that specific job by removing the second timeout value is allowing YAML to use the preceding timeout value.
> Is this a design compromise? Or are there any possibilities of synchronizing pipeline stages on different nodes? I am thinking of some mechanism like "do not start 'test' stage on remote before 'boot' stage on device has finished".
You are thinking of the LAVA MultiNode API.
> > > Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT > ENGINEER Garz & Fricke GmbH Tempowerkring 2 > 21079 Hamburg > Direct: +49 40 791 899 - 55 > Fax: +49 40 791899 - 39 > tim.jaacks@garz-fricke.com > www.garz-fricke.com > WE MAKE IT YOURS! > > Sitz der Gesellschaft: D-21079 Hamburg > Registergericht: Amtsgericht Hamburg, HRB 60514 > Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael > Braun > > > > > _______________________________________________ > Lava-users mailing list > Lava-users@lists.lavasoftware.org > https://lists.lavasoftware.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun