Hi folks,
We at Fairphone have developed a variant of the Tradefed-runner in LAVA test-definitions that is meant to run complete Tradefed test suites on multiple devices by making use of the shards feature in Tradefed. The runner is currently in “staging” state. We still want to share now what we are using and developing to see if there are more people with interest in it. Feedback on the general approach taken would also be much appreciated.
On the higher level, our setup works as follows: • Use MultiNode to allocate multiple devices for one test submission. • One “master” runs the Tradefed shell, similarly as in the existing runner. • The master connects to the workers’ DUTs via adb TCP/IP. These DUTs are transparently available to Tradefed just in the same way as USB-attached devices. • Workers ensure that their respective DUTs remain accessible to the master, especially in case of WLAN disconnects, reboots, crashes, etc.
Major features of our runner: • Support for Android CTS, GTS and STS. • Test run split into “shards” in Tradefed to run tests in parallel on multiple devices. This allows for a major speedup when running large test suites. • Tradefed retry: Rerun test suites until the failure count stabilizes. • No adb root required. • Based on the original Tradefed runner, having at least parts of the common code moved to python libraries.
Current limitations: • Test executions are not always stable. This needs further investigation. • Test executions produce more false positives than local test runs. This needs further investigation but is at least partially due to using adb TCP/IP instead of a local USB connection. • Android VTS not implemented (would require only minor changes)
Our current changes have been pushed to the tradefed_shards_with_retry topic on Gerrit[1]. Besides the two major changes to add MultiNode adb support and then Tradefed support on top of that, a couple of smaller changes that could be useful on their own have also been pushed.
We are looking forward to your feedback and to joint efforts in automating and speeding up Tradefed test executions!
Best regards, Karsten for the Fairphone Software Team
[1] https://review.linaro.org/q/topic:%22tradefed_shards_with_retry%22+(status:o...)