Main changes in 2018.11
===================
0eb0d3ed7 Set requirements for requests to avoid CVE - as announced
previously python3-requests needs to be updated to protect against a
CVE in that package.
If installing 2018.11 on a Stretch system, ensure you have updated
python3-requests to the version from stretch-backports.
f7bbf6a8 Visibility of worker status effects on device views-
(improved in subsequent commits in this release). Indicate the status
of the worker when displaying information about the device so that it
is clear why the device is still Idle if there is a problem with the
worker.
ac0097c4 Fix 500 when rendering device dict page - if the device
dictionary is invalid.
0f7a7ec8 Prevent index out of range error and 97132e91 Extend 0f7a7ec8
to other command_output comparisons and 06203dfe Add a run_cmd helper
to Action class - the Action run_command support had become complex
and problematic. A new parsed_command has been added and a simpler
run_cmd. In time, existing use of run_command will be phased out in
preference for one of the two new commands. This should make it easier
to spot why certain dispatcher commands failed. This was triggered by
intermittent problems running simg2img and img2simg for AOSP.
4736b01d How to test and recover bootloaders in LAVA -
https://master.lavasoftware.org/static/docs/v2/bootloaders.html
ff37c802 Add docs on criteria for LAVA on other distributions -
https://master.lavasoftware.org/static/docs/v2/debian.html
Issues closed in 2018.11
===================
https://git.lavasoftware.org/lava/lava/issues?scope=all&utf8=%E2%9C%93&stat…
Full changelog
===========
ff37c802c Add docs on criteria for LAVA on other distributions
4736b01dc How to test and recover bootloaders in LAVA
2cae2b413 Fix lava-master crash when device yaml is invalid
570fb324d Allow sdist to be passed to setup.py
51b3fe6f6 Fix missing part of aarch64 stretch deployment
f9a3671c1 docker: check method
563fac8da Enable aarch64/pkg-debian-9 in the build tasks
7e71a56ea flasher: fix substitution when cmd contain whitespaces
5bb71d4a1 Package the requires.py script
d77c8d28f Add CI package script for stretch on aarch64
24eface95 docker: use the new Action.run_cmd helper
ddd986e8e Extend the base poweroff timeout.
9d49da1d4 Remove build from .gitignore
24022feb4 Fix missing return value in run step
b5c51dfa8 Force Juno to use NFS vers=3 and extend power off timeout
7ddae776c run_cmd: make spaces explicit
87f1e01e1 Capture the simg2img and img2simg output and log it
f40f0f2c5 Ensure apt is updated before trying to install
06203dfe2 Add a run_cmd helper to Action class
97132e916 Extend 0f7a7ec8 to other command_output comparisons
f0ebf8025 Do not send "\n" twice
ac0097c42 Fix 500 when rendering device dict page
57d80c2e2 Add deployment builds to snapshot directories.
9a1a6bfb1 Apply black to more files in lava_scheduler_app
c6bfbd5f8 Apply black to lava_dispatcher unit tests
0b615023d version: allow user to specify the branch to use
f7bbf6a8b Visibility of worker status effects on device views
f2d9b658e Apply black to lava_scheduler_app/api and tests.
7980d84c3 docker tag: move branch name to suffix, not prefix
0f7a7ec8f Prevent index out of range error
984164f68 Fix omission in package deployment script
d4b942b71 CI use pytest-3 instead of py.test-3
d77c7080c Fix regression in fb4a88388
504115553 Update docs on local dev builds
fb4a88388 Standardise on dots in the version string
4a1910049 Use the rollback support in requires.py
cb86cc09b Improve error message after 4c71c2ebd1
15d7a905f Include package artifacts into repositories
4c71c2ebd Provide more information in bootloader errors
abff27c8b CI: skip deploy when lavafed is running latest version
f691bc059 CI fix typo for "only.refs"
024b27694 Fix errors in docker-admin page
a937a29e4 CI: add missing requirement to pytest
08ea617a4 vland: fix crash when finalizing
987280afd Fix errors in docker-admin page
89f96b1d9 CI fix deployment script
c5de132b1 Deploy lavafed-master when scheduled
08eab0bcb Fix issues with unit test calculations
7e8a6f444 CI use the new arch specific images
7d2956e43 Sort the --names output of requires.py
0eb0d3ed7 Set requirements for requests to avoid CVE
8cd80dcb2 Build packages and docker images for AArch64
1ab6aaa9c Allow unit tests to run without /sys/class/misc/kvm
83649d77b Port 0e598e63 to the xmlrcp api
6335f73d8 Prevent crash if environment requested for non-POSIX
14b347c51 lava_results_app: convert Decimal objects to string
storing as YAML
26bf0af63 lava_results_app: add test case for YAML Decimal
object conversion
1744ea74c Change ownership of health checks and device-types
32a2a6051 Ignore gitlab-ci files when creating a release tarball
bd224fcd1 Update publish for changes in pkg/docker
2e5e60c9f Drop obsolete gitreview file
2018.11-1 has been uploaded to Debian unstable. This version should
migrate into buster, at which point an upload to stretch-backports
will be made too.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
To avoid CVE-2018-18074, LAVA has bumped the dependency on
python3-requests to 2.20.0
https://git.lavasoftware.org/lava/lava/commit/0eb0d3ed7fa122b08c7ada7d27cfa…
If you are running LAVA on Stretch, you should take advantage of the
security fix by installing python3-requests from stretch-backports.
If you are tracking the daily packages for a staging instance, those
packages now depend on 2.20.0. Installing those packages on Stretch
will require installing python3-requests from stretch-backports. This
may cause automatic upgrades to be unable to upgrade the LAVA
packages.
$ sudo apt -t stretch-backports install python3-requests
If you are running on buster, the new version is already available in
buster and will be upgraded automatically.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
The 2018.10 release has been uploaded to Debian:
https://tracker.debian.org/news/998932/accepted-lava-201810-1-source-all-am…
In due course, that is expected to migrate into buster. 2018.10 might
be the last release to be uploaded to stretch-backports in the Debian
archive because the Debian release process for Buster will be starting
in January.
We have been making a lot of changes to extend and enhance the CI
within the LAVA software upstream processes as part of the move to the
Community Project. This now allows us to support 2018.10 and
subsequent releases to make it easier for labs who don't want to
migrate immediately to the new releases.
2018.10 is also available from the LAVA repositories.
New LAVA repositories
=================
https://master.lavasoftware.org/static/docs/v2/installing_on_debian.html#la…
deb https://apt.lavasoftware.org/release stretch-backports main
deb https://apt.lavasoftware.org/release buster main
The production-repo within Linaro has also been updated at
images.validation.linaro.org - this is the same location as previous
releases.
Snapshots
========
>From 2018.10 onwards, LAVA software will be retaining snapshots of
daily builds of LAVA at https://apt.lavasoftware.org/snapshot/
organised by suite, year, month and date.
Official Docker images
=================
For the first time, LAVA is also releasing official Docker images for
amd64. These are available from lavasoftware.org and from the
lavasoftware organisation on hub.docker.comhttps://master.lavasoftware.org/static/docs/v2/docker-admin.html#administer…
lavasoftware
=========
hub.lavasoftware.org/lava/lava/lava-server:2018.10 2c9b2e36f 155.64 MiB
hub.lavasoftware.org/lava/lava/lava-dispatcher:2018.10 fab86e936 236.85 MiB
hub.docker.com
============
docker pull lavasoftware/lava-dispatcher:2018.10
https://hub.docker.com/r/lavasoftware/lava-dispatcher/tags/
docker pull lavasoftware/lava-server:2018.10
https://hub.docker.com/r/lavasoftware/lava-server/tags/
Changes in this release
==================
I've skipped changes to do with the CI process on lavasoftware.org.
24f6bc296 Support gdb/openocd for disco_l475_iot1
1d1cf4380 Fix resource cleanup when using prefix
7f9796ee1 Append the defined prefix to the udev rule files
1e60552f1 Allow a user-specified kernel failure message
8445444bb device-types: minnowboard: add initrd command to nfs_commands
7c006f458 Ensure build dir exists
0e598e63f Update package version support
0c75c37b9 Distinguish between builds on different suites
135269b50 XMLRPC: use django permissions to manage accesses
dcf5e1358 Fix inconsistent header in docker admin
21f2ddf12 Do not depend on network access for test suites
e74e35b93 docs: Improve wording in various places for new users
579b86bb3 Add docs for IoT boot methods and monitor test action
fab08cc2e docker: allow to use local images
0e0f4d352 Comment out people.l.o and snapshots.l.o urls
f429369bd Support for skipping test shell timeout
32e02b925 Add device-type for qcom QDF2400
368f13217 Help for admins using docker.
829658237 Drop sections on V1 removal
05fcc7f9a docker: allow to use local images
d24c2e756 Fix build errors in docs
d4cfad706 Import the packaging support into upstream
caf3df11e Fix pyocd requirement listing
ae66959a1 Callback add a default timeout to 5 seconds
8e556cbab Compute coverage only on master or for tags
8d928d483 Update documentation for changes in share/requires.py
536b3bab9 Advise use of allow_none in XMLRPC clients
2dc976d28 Update Debian installation instructions for new repo
ddc980704 Allow switching to a binary=any build
59ba58b9c download: Raise an Infrastructure Error on timeout
2e3338ebc Limit excessive exception messages
1f10f2996 Fix undefined variables found by prospector
93529bc64 LAVA-1231 Support multiple boots in QEMU
23f25ee7a Ensure Device.is_valid is checked in submissions
6ca13ea7a Update jobs.list for two bugs found in Linaro
5f36ec668 XMLRPC: allow_none should always be True
f077b620d base-uboot: nbd: handle uboot_set_mac
1d2f38615 Fix arch build instructions in build script.
a6e61d0f9 Update pipeline-admin section on device dicts
5d045edec Isolate dependencies for unit tests
97e55bcf4 Advise against using patterns if POSIX is available
cbdc3ca47 Migrate support links to lists.lavasoftware.org
b34022569 Do not dump python object in description.yaml
42bfd72d3 Update the shebang to python3
ede13f753 Add index entry for persistence
b037a8660 Callbacks replace urllib.request with requests
a5bfdd71e Disable sorting Jobs tables by state
59add5e70 Drop release-queue.py
ecfc5c405 Set InfrastructureError for fastboot-flash-action
8d5679606 Fix imports of JobError from the wrong file
38dc6cace Ensure base-uboot quotes fastboot and adb serial
22f64cb62 Use raw string for regexp
dd5532ee3 Apply black formatting to linaro_django_xmlrpc
819986bd9 Apply black to remaining files in share
9556abc5c Add NBD support for iPXE based DUTs
563b88a00 Keep tags with uppercase letters
74b1d8ca2 Make version string lexicographically sortable
485bca7a6 Improve 359f8278 to support all -misc variants
13fe9a5ec Fix crash when calling lava-server without args
c96946ee9 Fix BootloaderSecondaryMedia.validate() with arbitrary boot commands
9debcf425 Fix warnings during package build
6a9ad82b8 add pytest support to lava-dev
1e0de18dc Fix missing change in the doc after 9595d66e
ee97069da Update Authentication Token page to use lavacli
b52d52b5e Add lava-server manage device-tags
9ae06152c Initial blackening support.
8051f1ddf Silence SAST warnings about mark_safe for internal data.
146e143d4 Apply yaml.safe_load to unit tests
01a3b0c0b Silence SAST noise from unit test
9ac2edac8 Silence security noise from unit test use of assert
c4ad0130a Explicitly set autoescape to False for YAML.
3c359acf3 Point README at new site
b9169d75e Fix typo in the variable names for since
53e21ef20 Update jobs.py
386b9efb7 Add more output to scheduler.jobs.list
a7b7939dd Updates for the contribution guide
fdd55ba63 Improve indexing and linking for device dicts
a956159da Support building for specific suites
9a5aa7e9b Minor doc updates
33f3abf62 Update copyright of docs.
8c6761baa Add gitlab ci configuration
eefc21edc Fix build errors in manpages
5b9ef16d7 Port zmq-client example script to Python3
8cd8cee2a Fix copy/paste error in 65ba876cb5
ce3f6120d Use generator expressions when available
e6f4ef1de Use isinstance() instead of type()
4440cd746 doc: fix most broken links
60928847d Support LXC with IoT
0cc3653f9 Allow LXC for FRDM-K64F IoT device
3bb0819fb Allow the suite to be specified for requirements
dff387150 Fix call to requires.py
7ad9790a8 Move requirements away from pypi tooling
75c928fe1 Remove python2 leftover
359f82786 Add support for qemu-system-misc
65ba876cb Let LXC protocol calls use lists as in base
79312ea8a Upgrade doc examples to Stretch
6a2638fb6 Handle cancelled test jobs in get_recent_jobs
c56ab4c66 Fix logic in pending_jobs_by_device_type
af211db77 LAVA-1398 fix group handling for logrotate support.
cdf4aaec3 Support branch for requires.py
6ff5b43b4 Clarify some of the lavacli doc items
f4948855e Check for download files ending with /
0b25c71bc Port URLs to lavasoftware.org
9871e777c Fix link to api/help location.
571ff39e9 Support ZMQ dependency on lava-common
a7acfc0ae Document architecture restrictions.
393e52fda Fix vland interface code so it works with python3 too
6fc476481 Certificate directory needs to be available in common
e159a369b sample_jobs: fix bbb-initrd-nbd NBD usage
98dfbb377 Add unit test for minnowboard-turbot-E3826 GRUB template
7c01f67a7 device-types: add x86 Minnowboard Max and Turbot using GRUB
2d373d36a documentation: add note about no kernel type attribute for Depthcharge
cc432d33b device-types: define block commands in base-grub.jinja2
5e101d817 run_command: print the output when failing
f019af70d device-types: sunxi-common: support booti
8961645f0 LAVA-1014 add a prefix to tmp directories
b022cde9d Fix indent error
b885dd112 Improve documentation about dispatcher configuration
d9196acd2 LAVA-1397 - fix crash in callback data
55c52bb6d LAVA-1156 When running under pytest, fake requests
c6d61c0e4 Clarify about Debian point releases of Stretch
b051ff1b5 Add missing show_fail after e3cdf814
c4f578e46 Use syslog for lava_lxc_device_add
5f7a22074 Portablility support without using 'os'
e3cdf8140 LAVA-1384 - Add cancel button to definition pages
5693b68b1 Fix sphinx syntax in git describe example
f2f1af5d8 Update filtering of TestJob fields in export
927a28820 Fix error in imx8m change
d484031f7 Use git describe to get the version
b76fe5a1d LAVA-1295 iMX8M evk device integration.
5f3d1769e Allow user to specify the build directory
09d6a90dd LAVA-1380 Drop 'os' from functional test jobs
3af493d42 Update requirement for django-tables
8df4482f0 LAVA-1383 - allow overrides of posix constants
ba035ca72 docker: allow to bind devices
2c507744f Allow for '@' in device name
407db2b45 Revert "Raise a JobError when a cmd output is empty"
f2d4490fd admin: improve listing and update/add page
953df9325 Raise a JobError when a cmd output is empty
baf61b465 Check downloaded file size against expected
ca74a1939 Use OSError when applicable
fc083df5c Use contextlib.suppress when applicable
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Another step in the LAVA Software Community Project - the lava-users
and lava-announce mailing lists have moved from lists.linaro.org to
lists.lavasoftware.org so that these can be more easily managed within
the LAVA Software Community, outside Linaro.
The configuration is the same but subscribers may need to change their
local filtering rules.
In addition, a new mailing list has been created,
lava-devel(a)lists.lavasoftware.org which is intended for discussions on
design and development issues within LAVA. If you are working on a
merge request for LAVA, this mailing list can be used for a wider
discussion of any issues which don't fit neatly into comments on the
merge request. If you are working on your own device but have not
prepared or created a merge request, please use
lava-users(a)lists.lavasoftware.org as before.
Message headers like List-Id, List-Unsubscribe, List-Archive,
List-Post, List-Help and List-Subscribe simply switch domain from
lists.linaro.org to lists.lavasoftware.org
>From this point onwards, please only send list emails to lists.lavasoftware.orghttps://lists.lavasoftware.org/pipermail/lava-announce/https://lists.lavasoftware.org/pipermail/lava-devel/https://lists.lavasoftware.org/pipermail/lava-users/
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
(There will be a few more messages than usual on announce this month.
Apologies, but I think it's easier to avoid putting everything into one
enormous message.)
As part of the LAVA Software Community Project, there is a new IRC channel
on Freenode aimed at any discussions around LAVA which do not relate to the
use of LAVA within Linaro. The #linaro-lava channel remains open and will
remain focused on discussions of problems and issues relating to LAVA
instances operated by Linaro and other Linaro-centric topics.
Community members are invited to use the new channel to discuss anyting
related to LAVA but not involving Linaro. Questions about your own
instances, problems with new devices and general LAVA development
questions. The new channel should make it is easier to respond to questions
and discuss issues. The existing IRC guidelines apply, as does the LAVA
Software Community Project Code of Conduct.
In addition, when asking questions on #lavasoftware, please include
information on your LAVA setup - include information on the version of LAVA
you are running and, if the instance is publicly-accessible, include the
URL.
As always, complex questions about device integrations will typically
require information like the device-type template, device dictionary and
test job submission definition - you are much better to ask those questions
by subscribing to and posting to the lava-users mailing list.
https://master.lavasoftware.org/static/docs/v2/support.html#irchttps://master.lavasoftware.org/static/docs/v2/code-of-conduct.html#code-of…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
As part of the move to the LAVA Software Community Project, we will be
moving the hosting of the lava-users and lava-announce mailing lists to the
new server in two weeks - the 16th October.
This will involve changing the URLs of the lists to:
lava-users(a)lists.lavasoftware.org
lava-announce(a)lists.lavasoftware.org
A new list will also appear: lava-devel(a)lists.linaro.org which will support
development of LAVA within the Community Project. Development discussions
relating specifically to Linaro LAVA infrastructure and devices can remain
on linaro-dev(a)lists.linaro.org
If, for any reason, you do not wish to be on the new list, feel free to
unsubscribe between now and the 16th October 2018. All subscribers and
configuration will be migrated to the new location using the configuration
of the lists at lists.linaro.org on that date.
If you decide to stay on the list(s), any filters will need to be updated
on or after the 16th October.
On the day of the migration, there may be a short time when it will not be
possible to post to the current lists. There will be an email from
lava-announce(a)lists.lavasoftware.org when the migration is complete.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
Opening up LAVA development
============================
The LAVA software project has been developed in the open from its very
beginning: Free Software licensing, public discussion spread across IRC and
email, and open access to the software development repositories. The
roadmap and planning behind LAVA development has also become increasingly
open over time, with the use of publicly archived mailing lists and
anonymous access to LAVA issue tracking.
A community of users and contributors has steadily grown up around LAVA.
With the completion of the migration to the V2 codebase, the Linaro LAVA
software team is now moving to a more open development model. We invite new
and existing contributors to be directly involved in planning future
development of the LAVA software project. To this end, Linaro has created
the new LAVA Software Community Project (LSCP) to host and foster LAVA
development with more input and control available to the community. This
removes the need for contributors to register with Linaro, and provides
much more flexibility for LAVA Software development.
The existing Linaro lab in Harston, UK, will remain entirely within Linaro,
providing test infrastructure for Linaro and its members. The sole focus of
the new community project will be LAVA software itself. This will allow the
project to focus on priorities beyond just what Linaro needs, for example
easier integration of devices and new/different use cases. The Linaro
engineers working directly on LAVA software have migrated into a
newly-formed Linaro LAVA Software Team and will act as the founding
maintainers and contributors for the upstream LAVA Software Community
Project. The Linaro LAVA Software Team will still retain access to the
Linaro lab staging instance (https://staging.validation.linaro.org/) for
functional testing using devices available within the Linaro lab.
A new home - https://www.lavasoftware.org
The LAVA Software Community Project (LSCP) has a new home at
https://www.lavasoftware.org.
We have set up initial services there including:
https://www.lavasoftware.org/ - the main website
https://git.lavasoftware.org/ - a GitLab instance to which we have moved
upstream LAVA development
https://master.lavasoftware.org/ - a LAVA master service, available for
development use
Existing LAVA contributors will need to reconfigure their git checkouts
accordingly. The new GitLab instance does not share credentials with the
existing git.linaro.org service; new accounts will be needed - see the
introductory contribution guide
https://master.lavasoftware.org/static/docs/v2/contribution-intro.html and
the full https://master.lavasoftware.org/static/docs/v2/contribution.html
for more details. The former location on git.linaro.org will be archived in
due course.
New repositories exist for the nightly LAVA builds at
https://apt.lavasoftware.org/debian/ (using our existing apt signing key)
and the documentation is being updated to match. The primary location for
test images used in LAVA software functional testing will now be
https://files.lavasoftware.org/; a process to add more files to extend the
functional tests will follow soon. URLs using
http://images.validation.linaro.org/ will continue to be available for a
transitional period.
New mailing lists will also be available soon on the lavasoftware.org site.
There will be a separate announcement about this shortly; subscriptions are
not yet open on the new site.
Contributing to the LSCP
========================
The LAVA Software Community Project has published a Contribution Guide
https://master.lavasoftware.org/static/docs/v2/contribution.html; all
contributors will be expected to adhere to this when submitting issues or
merge requests. These guidelines are designed to clarify the requirements
for contributions, to make contributing more efficient for all involved.
The contribution guide also includes details of how to switch from the old
development process to the new.
We have also adopted a Code of Conduct for the LSCP (
https://www.lavasoftware.org/code-of-conduct.html). All participants in the
LAVA Software Community Project will be expected to communicate and behave
appropriately when working with us.
We request that contributors with currently unmerged Gerrit reviews on
https://review.linaro.org/#/q/status:open+lava/lava should move these
across into merge requests in the new system at
https://git.lavasoftware.org/lava/lava. We will offer help here on request,
but will not automatically migrate them ourselves.
All the outstanding LAVA JIRA stories have been converted into GitLab
issues at https://git.lavasoftware.org/lava/lava/boards. We intend to use
the GitLab workflow for easy integration between CI, issues and merge
requests. At some point in the near future, Linaro will disable the LAVA
projects on Gerrit (https://review.linaro.org/) and JIRA (
https://projects.linaro.org/).
A new way of contributing to LAVA is also under development. A federated
functional testing system for LAVA will allow multiple labs to contribute
to distributed functional testing of LAVA Software, by running specially
designed test jobs on a much wider range of hardware. More details of this
will be announced in due course.
Please ensure that you abide by the Code of Conduct and please read the
contribution guide
https://master.lavasoftware.org/static/docs/v2/contribution.html alongside
this announcement. Discussion of our new project structure is very welcome:
please post to the existing lava-users mailing list at
lava-users(a)lists.linaro.org. Please note that some details of the LSCP are
not yet finalised, so not every question may be answered at this time.
LSCP governance
================
Options for the future governance of the LAVA Software Community Project
are being prepared - more details will follow.
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
To improve the functional test framework, we're looking at a mechanism
based on docker and this has had a knock-on effect on how we use git tags.
Previously, tags were made against the release branch. To get this new
support working, we've moved the 2018.7 tag from the release branch to the
master branch.
This does have one effect when using the developer build scripts:
dch: fatal error at line 1092:
New version specified (2018.4.post1-382-gf2f1af5d8-1) is less than
the current version number (2018.5.post1-2)! Use -b to force.
This is due to the tag issue. To fix, use git pull --tag:
neil@sylvester:lava (master)$ git pull --tag
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (1/1), done.
>From gitlinaro:lava/lava
t [tag update] 2018.7 -> 2018.7
Already up to date.
Developer builds will then proceed.
Note that we've changed the version syntax slightly too.
lava_2018.7-17-gf2f1af5d8-1
See
https://staging.validation.linaro.org/static/docs/v2/debian.html#local-vers…
--
Neil Williams
=============
neil.williams(a)linaro.org
http://www.linux.codehelp.co.uk/
