lava-announce

lava-announce@lists.lavasoftware.org

135 discussions

Re: [Lava-announce] [Lava-users] Merging the LAVA code source

by Neil Williams

On 24 April 2018 at 08:21, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote: > > Not true. The master produces the configuration which is fed to the > dispatcher - the two are very directly tied > > at the level of what devices can support and how that support is used by > test jobs. Changes and fixes in the > > device-type templates will change how the dispatcher uses the code > support to control the device. > > With all due respect, Sir, I would not agree/would disagree to by > your/Linaro's currently presented architecture. Sorry, but this was only a notification of a decision already taken and implemented by the development team. The change itself was not and is not up for discussion on this list. > On Mon, Apr 23, 2018 at 4:06 PM, Neil Williams <neil.williams(a)linaro.org> > wrote: > > On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com> > wrote: > >> > >> > lava-server and lava-dispatcher have been merged into a single source > >> > repository - lava > >> > >> If you ask me, this is the (very) questionable decision. Indeed!? > >> > >> lava-server is the front-end manager of the whole Lava test > >> environment, while lava-dispatcher is the back-end. Connected/splitted > >> by ZMQ protocol. And... They should be separately maintained > > > > > > Not true. The master produces the configuration which is fed to the > > dispatcher - the two are very directly tied at the level of what devices > can > > support and how that support is used by test jobs. Changes and fixes in > the > > device-type templates will change how the dispatcher uses the code > support > > to control the device. The two have always been maintained by the same > team > > and in a tightly integrated manner. > > > >> > >> , since > >> they represent (very) different things. Essentially, they do (very) > >> different tasks. They are, after all, apples and oranges. > > > > > > The bulk of the expected work from here on is Device Integrations. Each > new > > device integration typically involves changes to the dispatcher to > support > > new deployment / boot methods as well as new templates for the device > > configuration. Combining the codebase allows those changes to be more > easily > > verified because the output of the device configuration templates can be > fed > > directly to the lava-dispatcher code changes in the dispatcher unit > tests. > > > > Previously, this has resulted in static device configuration files in the > > dispatcher unit tests and that has caused problems. > > > >> > >> > >> In other words, if you update/buy advanced/more expensive apples, you > >> also force testers to update for nuthin'/buy for the loss the same > >> oranges, they had before?! > > > > > > Not true. the packages are built from a single source package. That does > not > > mean that the binary packages need to be updated on the actual instances. > > > > Up until now, lava-server has always depended on the latest version of > > lava-dispatcher. That specific dependency is being removed as part of > this > > change. > > > > Nevertheless, the principle remains that the lava-master and lava-slave > > should be updated together so that the device configuration is in line > with > > the dispatcher support. See compatibility settings in the documentation. > > > > https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5 > > > > > > > >> > >> > >> Is it the wise decision??? > >> > >> Two cents worth lamenting/analysis, > >> Zoran > >> _______ > >> > >> On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams < > neil.williams(a)linaro.org> > >> wrote: > >> > This is for particular note for developers as it changes the way that > >> > reviews happen. > >> > > >> > If you've noticed a few reviews being abandoned, this is why. > >> > > >> > lava-server and lava-dispatcher have been merged into a single source > >> > repository - lava > >> > > >> > lava-coordinator will follow in time, to ease the update of > >> > lava-coordinator > >> > to Python3. > >> > > >> > This will, in future, allow easier testing of device integrations by > >> > allowing the lava_scheduler_app unit tests to be linked to the > >> > lava_dispatcher unit tests and have a single review which adds both > >> > sides of > >> > the device support. > >> > > >> > There will be a lot of testing, as normal, so staging will be moving > to > >> > packages built from the new source repository tree. > >> > > >> > The old lava-server.git and lava-dispatcher,git repositories will > become > >> > read-only and will get no further code changes. All changes will be > done > >> > in > >> > lava.git > >> > > >> > https://git.linaro.org/lava/lava.git/ > >> > > >> > The documentation will be updated over the next few days. > >> > > >> > -- > >> > > >> > Neil Williams > >> > ============= > >> > neil.williams(a)linaro.org > >> > http://www.linux.codehelp.co.uk/ > >> > > >> > _______________________________________________ > >> > Lava-announce mailing list > >> > Lava-announce(a)lists.linaro.org > >> > https://lists.linaro.org/mailman/listinfo/lava-announce > >> > > >> _______________________________________________ > >> Lava-users mailing list > >> Lava-users(a)lists.linaro.org > >> https://lists.linaro.org/mailman/listinfo/lava-users > > > > > > > > > > -- > > > > Neil Williams > > ============= > > neil.williams(a)linaro.org > > http://www.linux.codehelp.co.uk/ > -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 4 months

Re: [Lava-announce] [Lava-users] Merging the LAVA code source

by Neil Williams

On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote: > > lava-server and lava-dispatcher have been merged into a single source > repository - lava > > If you ask me, this is the (very) questionable decision. Indeed!? > > lava-server is the front-end manager of the whole Lava test > environment, while lava-dispatcher is the back-end. Connected/splitted > by ZMQ protocol. And... They should be separately maintained Not true. The master produces the configuration which is fed to the dispatcher - the two are very directly tied at the level of what devices can support and how that support is used by test jobs. Changes and fixes in the device-type templates will change how the dispatcher uses the code support to control the device. The two have always been maintained by the same team and in a tightly integrated manner. > , since > they represent (very) different things. Essentially, they do (very) > different tasks. They are, after all, apples and oranges. > The bulk of the expected work from here on is Device Integrations. Each new device integration typically involves changes to the dispatcher to support new deployment / boot methods as well as new templates for the device configuration. Combining the codebase allows those changes to be more easily verified because the output of the device configuration templates can be fed directly to the lava-dispatcher code changes in the dispatcher unit tests. Previously, this has resulted in static device configuration files in the dispatcher unit tests and that has caused problems. > > In other words, if you update/buy advanced/more expensive apples, you > also force testers to update for nuthin'/buy for the loss the same > oranges, they had before?! > Not true. the packages are built from a single source package. That does not mean that the binary packages need to be updated on the actual instances. Up until now, lava-server has always depended on the latest version of lava-dispatcher. That specific dependency is being removed as part of this change. Nevertheless, the principle remains that the lava-master and lava-slave should be updated together so that the device configuration is in line with the dispatcher support. See compatibility settings in the documentation. https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5 > > Is it the wise decision??? > > Two cents worth lamenting/analysis, > Zoran > _______ > > On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams <neil.williams(a)linaro.org> > wrote: > > This is for particular note for developers as it changes the way that > > reviews happen. > > > > If you've noticed a few reviews being abandoned, this is why. > > > > lava-server and lava-dispatcher have been merged into a single source > > repository - lava > > > > lava-coordinator will follow in time, to ease the update of > lava-coordinator > > to Python3. > > > > This will, in future, allow easier testing of device integrations by > > allowing the lava_scheduler_app unit tests to be linked to the > > lava_dispatcher unit tests and have a single review which adds both > sides of > > the device support. > > > > There will be a lot of testing, as normal, so staging will be moving to > > packages built from the new source repository tree. > > > > The old lava-server.git and lava-dispatcher,git repositories will become > > read-only and will get no further code changes. All changes will be done > in > > lava.git > > > > https://git.linaro.org/lava/lava.git/ > > > > The documentation will be updated over the next few days. > > > > -- > > > > Neil Williams > > ============= > > neil.williams(a)linaro.org > > http://www.linux.codehelp.co.uk/ > > > > _______________________________________________ > > Lava-announce mailing list > > Lava-announce(a)lists.linaro.org > > https://lists.linaro.org/mailman/listinfo/lava-announce > > > _______________________________________________ > Lava-users mailing list > Lava-users(a)lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lava-users > -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 4 months

Merging the LAVA code source

by Neil Williams

This is for particular note for developers as it changes the way that reviews happen. If you've noticed a few reviews being abandoned, this is why. lava-server and lava-dispatcher have been merged into a single source repository - lava lava-coordinator will follow in time, to ease the update of lava-coordinator to Python3. This will, in future, allow easier testing of device integrations by allowing the lava_scheduler_app unit tests to be linked to the lava_dispatcher unit tests and have a single review which adds both sides of the device support. There will be a lot of testing, as normal, so staging will be moving to packages built from the new source repository tree. The old lava-server.git and lava-dispatcher,git repositories will become read-only and will get no further code changes. All changes will be done in lava.git https://git.linaro.org/lava/lava.git/ The documentation will be updated over the next few days. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

2018.4 production release

by Neil Williams

The 2018.4 release has been uploaded to Debian unstable. It should also be possible to upload to Stretch-backports in about a week from now, depending on internal processes in Debian. The release is also available from the LAVA repositories - production-repo Dropping Python2 ================ 2018.4 is the first release exclusively using Python3. Upgrading from releases prior to 2018.2 will bring in a range of Python3 packages. After installation of 2018.4, a number of Python2 packages are available for removal, if administrators desire. See also: https://lists.linaro.org/pipermail/lava-announce/ 2018-April/000050.html lava-tool still uses Python2 - we are working on lavacli as a Python3 replacement. lavacli has also been updated to 0.8-1 in Debian. lava-server =========== f60a5328 callback: fix crash in python3 e627f8ec Allow skipping one unit test if kvm not enabled 0208402f job: make the link to the bottom more visible dd26e4fb Fix default BRANDING_BUG_URL b2cc847b Adjust bootm addresses for panda f25548b5 Make snow u-boot prompt more precise 6a209317 Misc beaglebone black fixes 1481ac98 Add note for bug in LXC templates on Stretch 4ed6a2f5 api: fix a crash when a test case is not unique 53c57de9 Fix crash when test case metadata is empty 99c32f4d Fix crash when action_metadata is None b286b6bd Fix missing import b708b038 Test user_command and flasher templates c316e516 Update more scripts to use Python3 4be0dd49 fix sphinx warning in docs 458f470e Improve "lava-server manage check" coverage 30efadeb LAVA-1274 fix javascript exception 31764a52 job: only update if failure_comment isn't empty 24fe81e6 LAVA-1276 - Search functionality broken on staging job list page 938a90d8 Update documentation for Python3 change d4f64157 Fix deprecation warning from Jinja2 with Python3 1a5c55e8 Add hi6220-hikey-r2 device-type template 13758edc Update development notes for Python3 8f0d5a94 Pass LXC/fastboot parameters to all jobs in split_multinode_yaml f3d7a8ac Fix API breakage from 2d37b3 2658935c Fix Python2 syntax when creating secret key. 65a30238 Fix Device.CONFIG_PATH for individual unit tests c5fbfa51 Use suppress from contextlib 60b50af3 Remove unused shebang c9ff6e9f Fix shebang for python3 494ae3af Fix wrong import 33c5f72f LAVA-1269 - Store checksum of job.original_definition in metadata f8124a8c Import infrastructure_error from the right module. 60ecef88 Add failure comment and error type to test job notifications. 4e313b45 Fix logic for template check warning ab65a859 Fix the logic in e1f44097 b0c549bb Fix unicode error in migrations ec01c5bf Replace "!!python/unicode" in job definitions. f4386c08 LAVA-1147 Support more than 3 priorities c169c287 Remove exclusive flag from device dict d7cba2d1 LAVA-1250 Allow admin to finish a TestJob 3b54f8d2 Preserve comments in job definition display 66cd46bd LAVA-1167 document the 'history' feature 8d835d78 Python3 changes needed for lava-server 73fa8d2e device-types: add sharkl2 48f63bf9 Fix missing conditions variable in custom query manager. bdb07a21 Fix pep8 errors 70614729 Fix crash when device dict is invalid b00fde96 Remove jessie specific code fa0418b7 Improve the Admin query view 7d8d94de Skip refresh of archived queries 926ba5af Use the TestCase job error message as failure comment e1f44097 Fix invalid_template check ea3d99ca LAVA-1263 separate vexpress bootloader_prompts 0ef60b0e Add admin link to device pages for superuser bba64496 Cleanup some doc error messages 91c03b30 Fix multinode sub job definition display. 48bb281e Job resubmission shows Internal Server Error on incorrect resubmission. 7b2201b5 Cancel subjobs from admin cancel action aac25876 Fix a grub error message so python can compile it into an RE 3555256f device-types: add sun8i-h2-plus-orangepi-r1 1ae6dda0 device-types: add sun8i-h2-plus-orangepi-zero f5e95c3a Save the error when the job fails to start 1fbd68fa allow load_devicetype_template to return raw yaml 2667881c dt: alert when the template is invalid or missing 75cf05ee Move infrastructure_error into test.utils 5819a2cd Add banana pi zero device-type config d8f605c0 Add another possible failure message to grub e5aca37a LAVA-1129 save job full configuration before start f54c2acf Make some after installation debug doc more visible. 4bb83a63 Fix 500 when device-type template does not exist e1371fef juno: allow override of bootloader_prompt b33a3165 Make the code python3 compatible after 0d1e0a07 d73687c6 LAVA-1253 - CSV export headers are incorrect 14aa6477 Force device hc for ConfigurationError and LAVABug 02011867 Use base-fastboot jinja2 template for mediatek-8173 device type. 606ad8b0 LAVA-1240 - LAVA: add test job name to email notification subject 414edc4d Add the overlay deploy method to all uboot devices f1c8d9c7 Add the flasher configuration to all uboot devices fec9dfac settings: simplify and allow setting any variables f2f6844d LAVA-1247 - Results API make_custom_query not working correctly 1a27b586 Remove unused variables from instance.conf de0834d6 settings: remove most variables from settings.conf 24646594 settings: remove appname 3f69dd81 Fix lava-server manage addldapuser --username USER 5aebdfce admin: fix view_siet link when using MOUNT_POINT f784db06 admin: use dynamic urls instead of static ones 0d1e0a07 Fix test for extra results metadata in callback data 356ba428 Move standard ARMMP tests to stretch 227caa82 LAVA-1249 address issues with large pexpect buffers 4bdb5d17 Allow time to drain capacitance 717a3387 Up power off timeout for juno 7a1260d4 Fix bootloader prompt for juno uboot ca334c5c job.validate does not accept any parameter anymore fa0855de Remove v1 leftover 5287d13d Remove unused output_dir e9fc34f0 device-types: rk3399-puma-haikou: align booti_kernel_addr 33c8579b Add imx7s-warp device-type 416fb3e5 Do not crash when re-adding the same device (or dt) 18318d27 Add admin actions to update worker health f57d683d master|logs: automatically reload certificates 390c8e2b lava-master: fix return values 2d37b341 Remove is_pipeline 45d5767a lava-master: record error message as failure_comment 74713edb Use STATIC_URL and get_absolute_url when needed 60283360 Remove TestJob._results_link f9a36ee1 LAVA-1180 Refactor device-type page 3456b9ea LAVA-1128 Use bulk_create to create TestCases cfcc0fc8 Move postgresql timeout to wsgi 4c61a59c LAVA-777 lava-slave is now restartable 806ac08b job api: allow filtering by state/health fb0b700e device-types api: allow get/set health-checks 6eadfc62 List Infrastructure, Bug and Configuration errors lava-dispatcher =============== 4dd73b76a Fix wrong infra error when compressing with xz 50a652a54 Add missing import and fix prospector warnings 817fe007d LAVA-1234 - Man page for lava-lxc-mocker 691041a1 compress: check that the tool does exists 84df8c5d LAVA-1264 improve handling of cancellation 02590e0b LAVA-1282 Only kill lava-slave 1cbebc39 fix typo in manpage 67a19da1 Add template testing for user-command and flasher 40a9a97b Do not crash when command output is not utf-8 73cd3623 LAVA-1266 - LAVA doesn't fail jobs that fail to flash partitions 7340ad00 BootloaderCommandOverlay fix validate logic 99526367 Log str representation of exception objects 957cc171 LAVA-1278 use external decompression for .xz ee0921ab LAVA-648 Slave needs to check DNS 896936c1 Update modprobe.d support (Closes: #888681) 67f49224 Extend installer error prompt list 43a29e4c Ensure QEMU iso method uses configured TFTP dir e88d3cb4 Python3 update for QEMU installer support a05f307d Decode returned log string to utf-8. 52babb39 LAVA-1274 do not log a boolean 15ba29b9 Fix shebang for python3 4c960c49 test: fix file descriptor leaks a6fdafd9 Remove python2 specific code 3e7891d1 Drop Python2 support 8a7d655a Revert "shell: wait for prompt between each command" d5e15af2 Fix lava-os-build output 303b3871 Remove deprecated lava-test-run-attach usage 8cb5f120 Remove leftover from v1 in android helpers 0e304335 shell: wait for prompt between each command b67fb420 Fix raise format 351176ab Remove stale udev rules 8d6c5ec7 Python3 fixes e8a757df LAVA-1167 allow removal of the .git directory 54ad1fbe Remove jessie specific code f4784431 Remove v1 signal handler bb910afd Fixup connection.prompt_str assumption 26897c46 LAVA-1257 do not crash when STARTRUN is missing bd6c2526 LAVA-1256 raise a JobError when url is missing 0e255c3b Raise an Infra Error when an exe is not usable 294a2322 Fix URLs referred in unittests to point to permanent location. a1474089 LAVA-1252 do not crash when device conf is missing 5179d1de Device: factorize code c5645bc6 Remove power_on command as it's never used bf57d6fe Fastboot boot method should have a sequence associated with it. eab74bae Catch errors with lava_test_results_dir early b4ac1dbe Use the permanent location for WaRP7 unit test a41f73ee Remove empty class functions 30b84f20 Fix format string when raising 1854a1dd Do not print pipeline description to stdout 0512e6ce Remove unused output_dir e5043cb3 run_command: split the output at the newline char cadd2123 Add a new deploy method called "flasher" 00437106 Use job tmp directory for the compress overlay 1158c5e7 Add a deploy method that only create the overlay eeac3e67 docker: only mount the overlay when available 6916519d Make the helpers faster d8b205ee Specify the exception to raise when timing out ad106451 LAVA-1146 lava-slave removes stale resources 67b18778 LAVA-777 make lava-slave restartable b61a58e5 Refactor lava-slave code cdd2ef54 Adding deployment action for MPS devices b8e7dc13 xnbd: fix killing of old xnbd-server process -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

Python3

by Neil Williams

The master branches of lava-server and lava-dispatcher have now moved to Python3 exclusively. This means that all developer builds will only support Python3 and that Debian Jessie is no longer supported for LAVA software. It is recommended to use the staging-repo as there are a lot of dependency changes to add Python3 dependencies on machines which have not used Python3 before. apt will make the job much easier than trying to install developer packages and then fixing the dependencies manually. Even if you have 2018.2 installed and have Python2 and Python3 support available, there are still some new dependencies to move the runtime support to Python3. The following NEW packages will be installed: gunicorn3 python3-dateutil python3-gunicorn The documentation has been updated: https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h… Once an instance has been upgraded, apt will typically warn about a range of Python2 packages which are no longer required. These can be removed using the command: $ sudo apt-get --purge autoremove Check the list of packages carefully! One example list could be: Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: python-django-auth-ldap* python-django-restricted-resource* python-django-tables2* python-ldap* python-lzma* python-markdown* python-netifaces* python-pyudev* python-simplejson* python-voluptuous* 0 upgraded, 0 newly installed, 10 to remove and 1 not upgraded. After this operation, 1,855 kB disk space will be freed. Do you want to continue? [Y/n] apt is cautious about listing packages for autoremoval, so you may well find Python2 packages still installed. This is normal. This starts a new round of functional testing in preparation for the next LAVA software release. That release will be Python3 only and it will not be built for Jessie. Debian uploads from here on will only be to unstable and stretch-backports. LAVA repositories will only carry updated packages for stretch-backports. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

Turning off builds for jessie-backports

by Neil Williams

See https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html The time has come to drop regular builds for jessie-backports. If you rely on these, you will have found that the builds of the master branch broke for jessie-backports some time ago (whilst the team were at a conference). So, in the absence of requests to fix the build and in the light of the existing plan to drop Jessie support in Q2 2018 (i.e. the next production release) there seems to be no reason to retain the build. This also means that building from the master branch for jessie is also broken - this is not going to be fixed. This opens the door for testing of lava-server using Python3 exclusively, dropping Python2 support. Depending on how that testing proceeds, the next production release could be Python3 only. Summary ======= * jessie-backports support in the staging-repo has ended * developer builds of the master branch no longer need to build on Jessie (and are currently broken) * there will be no further uploads of lava-server to jessie-backports or jessie-backports-sloppy in Debian. * support for Stretch continues unaffected * when the next production release is made, a backport to stretch-backports in Debian is likely to be uploaded. * There will be no further builds for Jessie for the production-repo or jessie-backports in Debian. * testing of full Python3 support can now begin -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

LAVA 2018.2 production release

by Neil Williams

Python3 ======= 2018.2 is built with Python2 and Python3 support but only Python2 remains is available for runtime (until Jessie support is removed). This results in the built packages for Debian unstable and stretch-backports now bringing in Python3 versions of the existing Python2 packages like python3-django. The unit tests are routinely run using Python2 and Python3. The packages available from the LAVA repositories reflect the same: jessie-backports 2018.2-1+jessie - Python2 only stretch-backports 2018.2-1+stretch - Python2 and Python3, using Python2 at runtime. Bootloader support changes ========================== 25859894 Better detect errors in the bootloader - this adds support to distinguish between a bootloader failure and a kernel failure to detect problems when the bootloader tries to start the kernel. This has an important effect on how some test jobs run - see Quiet Kernels below. The parallel change (7a2b3a68 Change the flow of bootloader commands so they are executed individually) supports detecting failures to download artifacts as distinct from failures to execute once downloaded. Bootloader action optimisations =============================== To support the better error detection, several bootloader actions have been optimised. This means that different actions may be used, changing the names you may be using for timeouts. e.g. https://staging.validation.linaro.org/scheduler/job/209814/definition#defli… The timeout name was u-boot-interrupt - now it is bootloader-interrupt. The UI shows you which actions have been assembled into the pipeline. e.g.: bootloader-interrupt: Wait for prompt Hit any key to stop autoboot (timeout 00:02:00) Quiet Kernels ============= The bootloader support changes wait for an indication that the bootloader has completed and that the kernel has started, by watching for a kernel_start_message. If your kernel is configured to be quiet, then each test job using that kernel *must* clear the kernel_start_message: context: kernel_start_message: '' In most cases, the test job should not use 'quiet' as this hides important debugging information from the kernel boot process. LXC mocker ========== lava-dispatcher now builds a second binary package which is intended to support the use of Docker without needing to change the test job submission which did not use Docker. e.g. submissions to LKFT use LXC and there are issues trying to use LXC inside Docker. To make it simpler to reproduce test jobs when debugging regressions, the lava-lxc-mocker can be installed in Docker instead of lxc. Commands which would then have been run in the LXC will instead be executed inside Docker directly. CAUTION: do not install lava-lxc-mocker *except* inside other containerisation like Docker. When using lava-lxc-mocker, create a dedicated Docker image which handles only one DUT so that tools like fastboot and adb which are constrained inside the LXC do not interfere with other test jobs and devices. If you have been using the developer build scripts and you are using 'sudo debi' then you will need to change to using 'sudo dpkg -i' and listing the lava-dispatcher package instead of the .changes file. Release cadence =============== Now that the roadmap to remove V1 is complete, the need for monthly releases has diminished. As part of a range of other plans aimed at making LAVA more flexible, the monthly release cadence has been dropped. We are looking at a range of options, including wider functional testing of the master branch, to support releases on a rough 3 month cadence but without the added pressure of expected release dates. This will become a lot easier once support for Jessie is dropped and the build matrix becomes simpler. Changes in this release: lava-server =========== 9ef41db7 Fix missing failure comment 206e5c65 manage jobs: allow to force fail a job 42a3bc31 Save TestCases even when metadata is too large 5efb92c7 lava-logs: look for lava.job results in all cases 19937074 Use the named urls instead of static urls e60afcbc Ensure uboot_extra_error_message is usable 843a49ed Use reverse() instead of a static url 7583bb09 device-types: add meson-gxbb-p200 40cbffc8 breadcrumb: keep the last element clickable 2d5f34d8 Update docs examples metadata 5a5548ad device-types: fix broken conditional in base-depthcharge d663a9dc device-types: add rk3399-gru-kevin 21ac9eb2 LAVA-1230 include lava-logs in encryption docs d57e9131 device-types: rpi3 32-bit: allow custom kernel cmdline 32312dc7 device-types: rpi3 32-bit: allow extra_kernel_args append afa28d5a device-types: rpi-3 32-bit: fixup DT load address 0c9e227b Add missing change to the X15 template 968a00dd Fix fdt load command wrapping. 156f1e7b fix quoting of serial numbers 57610c06 lava-logs: simplify log format 5301ef66 breadcrumb: the last element should not be a link 08547de9 Fix api pages titles and breadcrumb fc03f62b LAVA-1083 - Bug links can be managed only on list pages 35146e4c Remove distinct ON because it can't be used correctly with order by. 85f6bd67 Test shell helper support with base-fastboot b86981ad master: do not send START to offline workers b6cf83f8 admin: fix group cancelation 8890d270 Drop auto-login from hikey 620 template 5642e8cd Allow cancelation in Scheduling and Scheduled too. bbf6c5bb LAVA-1220 - Update API calls and docs for scheduler changes. 742433b0 Drop invalid boot_message for hikey 6220 441919d7 Skip kernel-start-message for dynamic connections 8f4ecdcb Add a parameter to u-boot to override interrupting 8c093e7f Reorganise fastboot for multiple OS support abb9fa5d Adding device-type config for MPS devices 240ffeda Add rk3399-puma-haikou device type and health check bd0c7da7 TestCase: fix crash when accessing old test cases 4e22927e Set Unknown as the default option for changing health 078b6499 api: return the right http error code e6b3ffdd Fix doc Unexpected indentation error. b523e20f Fix the filename in header to the actual filename. acf537fb Improve device tables a0a87385 LAVA-1225 catch errors when creating health checks ea8d5a72 Do not show Idle state when device is unavailable dd4f25bd Allow custom tftp commands for loadfdt in U-Boot a074403a LAVA-1219 allow unicode for job.original metadata 58d23f23 Python3 branch now merged into master 717c177c fix missing quote mark in docs dcb8bd5a LAVA-1224 support for Xilinx-ZCU102 ZYNQ fec11287 Improve python2 and 3 support 756d2e6a migrate-job-output: fix inverted logic for dry-run fbeaa57f Add NotificationRecipient in the admin site 135654d6 Fix online devices listing 7007fda8 Fix change in 70cb792c7 to show extra_source 025bab80 LAVA-1221 change mouse pointer whenever needed b28e4c7b LAVA-1222 use verbose names instead of constants a281a6a7 Extend timeout to cover when dispatcher is busy 3985a0e1 Update development docs for Python3 plans 71374e3a migrate-job-output add --slow and --dry-run 9ce62f2d LAVA-1213 - move developer builds to Python3 81fbb4c4 Charts x-axis attribute should refer to job metadata 7af37d79 LAVA-1164 relicence some lava-server files 14604fb5 LAVA-1211 Document appending to kernel command line e8eb6ec0 Add admin actions to update device health b16e197b LAVA-1107 fix "device-types add" documentation 18be0e5b Set device health to unknown after an infra error 54ae8263 LAVA-950 use a ROUTER socket to connect to master 54ed49c2 Fix crash in lava.scheduler.jobs.logs 7a2b3a68 Change the flow of bootloader commands so they are executed individually lava-dispatcher =============== 7b9c3e84 Allow some deployments to not use deployment_data d5dc6418 ssh: fix comparison ebbd717b Remove spurious conditional in autologin b3e4d794 LAVA-1232 - LAVA LXC Mocker 4d445841 Fixes for X15 fastboot_uboot fd6dbec5 actions/deploy/lxc: lxc-apt-install action add non-interactive mode 1621ea5b LAVA-1202 - multiple boot + test section using transfer_overlay fails 84d78a08 LAVA-1229 - read-feedback: failed on DUTs without telnet or ssh 0ae6815e Ensure RetryAction has protocol support. 6b96c961 LAVA-1227 - Handle cancellation within LXC ce1421a3 Update ZCU102 device file for bootloader changes d53e452c Fix login support for second UART a0104eb9 Remove unused pipeline reference file. a9f22a9c Allow for empty values in boot methods 00addc73 Fix lookup for grubmenuselector interrupt prompt 782e69d4 Allow skipping of kernel_start_message 52ac66fa Make waiting for a bootloader prompt optional 41f7d58d Not all boot methods have parameters 8b0d03cd Remove deprecated actions in boot sequence. a3fb0095 Reorganize for hikey 960 AOSP support 7d78da6e Fix usage of FastbootReboot 161c1660 LAVA-1167 - Improve disk usage of test overlay tarball creation 7306a5c6 download: record the http error code when failing fd1f7013 Pass USB devices to LXC via ID_FS_LABEL 8bae0785 raise an infra error when docker is not installed 414861e1 LAVA-1138 - panic not recorded by auto-login-action 548a6bb1 LAVA-1224 - support for Xilinx ZCU102 abb9869f Unit tests for 960 AOSP support 0a2f84ce Fix man pages b15eb55e LAVA-950 use a ROUTER socket to connect to master 7a936509 Fix logrotation by using WatchedFileHandler 8f23430b Fix prospector warnings 25859894 Better detect errors in the bootloader 324615d7 LAVA-1165 - skip reading feedback on closed connection -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 7 months

2018.1-1 Production release

by Neil Williams

Roadmap reminder ================ https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html Installing 2018.1 will apply the changes from 2017.12 and earlier to **permanently erase V1 test data** if this has not been done already. (i.e. if you skipped installing 2017.11 or 2017.12, remember that 2018.1 includes the changes of 2017.11 and 2017.12.) Removal of V1 test jobs ======================= A few provisos with this release - there are a lot more changes to the lava-server package than to the dispatcher because 2018.1 drops the old lava_scheduler_daemon app and moves all scheduling into lava-master. The new scheduler state machine involves some database migrations: Applying lava_scheduler_app.0032_add_state_health_to_testjob_and_device... OK Applying lava_scheduler_app.0033_remove_old_status_fields... OK Applying lava_scheduler_app.0034_worker_last_ping... OK If your instance has a lot of V1 test jobs still, the migration could timeout, so as the final part of the removal of V1, follow the guidance in the documentation to remove V1 test jobs, if you have not already. (By "a lot", we're thinking more than a few thousand.) https://validation.linaro.org/static/docs/v2/v1removal.html#removing-v1-fil… If you have millions of V1 test jobs (as validation.linaro.org had), you will need to stagger the removal of the V1 test jobs using the --older-than support. Talk to us if you have issues, there is an updated script which we used on validation.linaro.org but that would need to be locally cherry-picked from 2018.1 onto the system running 2017.12. The removal script can be interrupted with Ctrl-C and restarted if required. Depending on the performance of the host machine, it may take a long time to run the removal, up to several hours in some cases. Finally on the removal of V1 test jobs, the support removes not just the database object but also the log file directory, so your master should find a lot more usable fileystem space has appeared, once complete. In many cases, the log file directory can be missing for a V1 test job but the warning is just for information. The database object will have been removed. Removal of unwanted dependencies ================================ $ sudo apt-get --purge autoremove Do check the list of packages before proceeding. If there are packages which you need for local support, make sure those packages are marked as "manually installed" in apt by using apt-get install <package_name> (this doesn't reinstall an existing package, it simply marks it as not a candidate for autoremove). It is worth running autoremove after this release has been deployed, extra dependencies can still be a security problem and an admin burden. Stretch ======= 2018.1 *deprecates* Debian Jessie and moves the recommendation on to Debian Stretch as the supported OS for LAVA. https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html Note that when the instance is upgraded to Stretch, this will bring in a new major version of PostgresQL and your LAVA database will need to be *manually* migrated. Neither PostgresQL nor Debian will do this for you. https://validation.linaro.org/static/docs/v2/debian.html#migrating-postgres… When following this guide, remember that your versions will differ from the guide - typically for the upgrade to Stretch, you will be migrating from 9.4 to 9.6. Take care with the commands to ensure you drop and upgrade the correct cluster. Make sure you have a valid backup. The instance will need to be in a *maintenance window* for the migration to occur. Repositories ============ 2018.1 has been uploaded to stretch-backports and jessie-backports for the production-repo on images.validation.linaro.org and jessie-backports will continue to receive updates until we get closer to the end of security support for Jessie. See https://lists.linaro.org/pipermail/lava-announce/2018-January/000046.html lava-server =========== d87ccc73 Restore Health and TestJob Reports calculations 6aa325e0 Add a reason when updating device and worker health aaa1a959 Notifications: use email signature conventions 3f6770f4 Fix notifications and log the exception fbe57d5f Only use get_change_message on Django > 1.10 832edcf8 fix pep8 error ff1ef1c4 Disable high limits on ODROID-XU3 7ae61e90 Remove old sample job 4be662da LAVA-1206 - Improve docs for API chunking 6c717e49 LAVA-1207 - Add xmlrpc API call for test suite list. d8d7ffdd Fix pep8 - E127 continuation line over-indented for visual indent. 81a2a6d1 Return state and health information as strings not numbers. d2451a1d LAVA-1163 - Unify json library across lava-server. d9ea9343 LAVA-1192 document use of static_info 824d9580 Deprecate Jessie in favour of Stretch 0791d8be LAVA-1198 - Make sure API covers retrieving the set of test suites 489b83da Add optional support for python3 unit tests 91782fac A sequence is required before excluding a field in table. 6be77f09 AUTHENTICATION_BACKENDS list is available in distro settings only. 938352ff Add documentation for the depthcharge boot method f6172120 Ensure added blocks are wrapped in conditionals 7ca1a522 LogEntry: improve SQL queries 38c8c8e3 Improve LogEntry tables e6bb9cf3 LAVA-1199 - Implement incremental result exports in XMLRPC 2945e5f7 Simplify TestJob and Device state display 2e6460bd LAVA-1197 - Document REST API chunking on test suite level. 2f7bc9ab Fix pep8 error 4f896471 LAVA-1197 - Implement incremental result exports in REST API 7a0854ac LAVA-1103 document user and group creation 4b7b353a lava-master: fix string serialization 1e9c1aa4 LAVA-1119 - users and groups for devices 4963ffa9 LAVA-1102 - add group support to manage commands bf9dd4ec Add missing sub-command parser for device update 9a480bc0 Fix missing import for lava-master 39faeb5f fix crash when canceling a non-scheduled job 4f201dd6 master: use yaml.dump to dump a dictionary 045ed9ec schema: validate multinode 2d2963d9 Allow browsing LogEntry in the admin interface ba37046c fix pep8 error 8c6b3c35 LAVA-1186 - Fix python3 unit test support be8b1e5d Worker: record the last ping received 1f4157c1 lava-logs: fix issue when unbinding under python3 d64514fb Port linaro_django_xmlrpc to python3 4b04cc30 lava-server manage: make the sub command required f49a1bd7 Port scheduler.jobs api to python3 320cb918 jobs: split the database requests in small chunks babab5ce Prevent noise from apache2 for DocumentRoot 79cfc375 Add FIT parameters to base-depthcharge and rk3288-veyron-jaq 62dd2464 python3: port timing page b52c84c6 Port BreadCrumb to python3 c4432f55 lava-master: sleep a bit when db connection drop 6b71a7e0 Port lava-master to python3 7f0a2ae5 Improve support for python3 e2182b3c Port lava-logs to python3 cc05ce30 All images that accepts compression will also accept archive. 2bb99530 Port lava daemons common functions to python3 1e84fdc9 Remove deprecated settings c530a82e remove restrictions on worker health transitions f66b7f6b workers.show now returns the list of devices 36239079 Print LogEntry messages on the admin main page 02f83e49 Test the Device and TestJob state machine 2d427a84 Remove unused job json functions. 70cb792c LAVA-963 - Support python3 -Wall unit tests 34d52727 Add base-depthcharge and rk3288-veyron-jaq device types 60f85e3e Derive protocol elements from sample jobs. 012034ef Update app_index.html from last django version 849aaa6c manage jobs rm: try to be nice by sleeping a bit 57a2356d Allow for instant canceling bc754b3f Fix udecode crash when receiving unicode strings 6aad013a Generate an event when a job is submitted ed9546fc master: improve scheduling regularity 698817ae Fix fastboot boot sequence for hi960-hikey device type. 107cea4b Fix 500: remove_broken_string has been removed 7c95d706 Extend base-fastboot template for fastboot device types. d779a194 Python 3 support for cStringIO. 65e769d7 Update example test job with pre command 2467411b LAVA-1170 - Unicode literals port 3bf73425 Provide and Document the storage_info support. 270c90a4 Document the state machine and scheduler 87af6a56 LAVA-1169 - Support xmlrpc.client imports in python3 e24865f3 Update lxc fedora example 14792136 No need to save the object returned by get_or_create 6114d0f9 Rewrite from scratch the job scheduler 9da8d83f Fix broken line detection af632f12 Add a test character delay to rpi3 a5163e61 LAVA-962 - Port lava-server to python3 exclusively - urllib b7e7ec2a LAVA-1085 - First time 'run query' click does not enable 'Results' button a47c170e LAVA-1084 - Clicking on 'live query' tooltip will submit request e50c2d03 LAVA-1082 - Results action link points to the same page 6705ec87 Refresh the setting files e4b39c38 Store the lava_server static file in the right dir lava-dispatcher =============== b0de4b84 Enabling energy probe data to be passed to LAVA test shell dd349a77 Use BaseException only when needed c1a0cd99 lava-slave: when canceling, wait for the job f00a7315 Port multinode to python3 f937a861 docker deploy: fix python3 port 11e3571d Generate the FIT image for Depthcharge d46f10c4 Update unit test sample file for images.v.l.o mirror. b4536e09 Include child devices of matched pyudev devices eb04f6fd Remove V1-only test shell helper scripts 9f96a0c5 utils/udev.py: lxc_udev_rule only use logging-url arg when is not None 781312b5 devices/db410c-01.yaml: Add cdt after aboot in flash order 9e809927 LAVA-1172 add support for lava-target-storage 53cfc7db Add support for the depthcharge boot method -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 8 months

Stretch and Python3

by Neil Williams

As announced in June 2017, Python2 is end of life. https://lists.linaro.org/pipermail/lava-announce/2017-June/000032.html However, the outline of the timeframe needs a few changes as Debian Jessie will lose security support around June this year and all LAVA instances will need to be running Stretch before that happens. It has proved to be impractical to support LAVA using Python3 on Debian Jessie and the completion of the migration has meant that lava-server is ready for Python3 ahead of schedule. Therefore, the pending 2018.1 release will **deprecate Jessie** and move the recommended installation of LAVA to Debian Stretch. The 2018.2 release of LAVA will add Python3 support to support developers and contributors using Python3. This means that the 2018.2 LAVA packages will contain python code in both the existing Python2 directories: /usr/lib/python2.7/dist-packages/ as well as the Python3 directories: /usr/lib/python3/dist-packages/ and will therefore bring in both Python2 dependencies *and* Python3 builds of those same dependencies. Installing 2018.2 will therefore bring in python3-django and a bunch of other Python3 packages, enough to run all the unit tests successfully on Debian Stretch using Python3, using stretch-backports. These are not yet used for runtime operations in LAVA, only for development and testing. A future release of LAVA will switch to using Python3 at runtime, dropping support for Python2 at the same time which will also drop support for Debian Jessie. This will happen *before* Jessie loses security support. As Stretch will be the minimum requirement for installation, this version would also move to requiring django 1.11 instead of 1.8 from Jessie. This creates a window for all LAVA instances to upgrade to Stretch using Python2 ahead of the separate move to Python3. To update the summary proposed in June 2017: Summary ======= June 17th, 2017 - Debian 9 Stretch release November 2017 - removal of V1 codebase January 2018 - migration of Cambridge lab to Debian Stretch instead of Jessie January 2018 - testing with django 1.11 LTS 2018.1 - testing with python3 2018.2 - add python3 support Q2 2018 - Release with python3 support, end of support for jessie-backports and drop python2.7 support. Move to minimum django version of 1.11 April 2019 - release of django 2.2LTS Actions ======= *ALL* LAVA instances should migrate to Debian Stretch support as soon as practical but certainly in the next three months. All development related to LAVA must support Python3 as of the release of 2018.1. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 8 months

2017.12.post1 hot fix release

by Neil Williams

This release has been uploaded to Debian and the LAVA repositories. The release fixes a security issue with private test jobs. 2017.12 added support for the REST API to access test job definitions. This change caused a problem where test jobs in which visibility had been set to personal or group were correctly displayed in job tables but the job definitions were accessible via direct URLs. This hotfix resolves this problem in lava-server. There was also a problem with the command line support for adding workers, this has also been fixed in this hotfix release. lava-server =========== 2e00544f LAVA-1168 - LAVA job visibility bug 341f5dfa LAVA-1171 - Show 403 instead of 404 31b7b2a9 Fix "lava-server manage workers add" -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 9 months

← Newer
1
...
6
7
8
9
10
11
12
13
14
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-announce