On the dispatcher, look for /var/log/lava-dispatcher/lava-slave.log

Hi Remi,

 

The log states “lava-run crashed”, I don’t think there is any lava-master crash because lava-master continues working fine for subsequent jobs. How can I get the lava-slave logs to check for a crash there?

 

Thanks,

Alex

 

From: Remi Duraffort <remi.duraffort@linaro.org>
Sent: Tuesday, January 21, 2020 12:52 AM
To: Alexander Moore <Alexander.Moore@cypress.com>
Cc: lava-users@lists.lavasoftware.org
Subject: Re: [Lava-users] lava-run crash after submitting a job

 

Hello,

 

this is lava-master crashing, not lava-run right ?

If lava-run is crashing, please provide the corresponding lava-slave logs.

 

 

Rgds

 

Le lun. 20 janv. 2020 à 23:34, Alexander Moore <Alexander.Moore@cypress.com> a écrit :

Hello,

 

Our internal LAVA setup has been hitting this crash intermittently (it reproduces about one out of every 30 job submissions). The below log snippet is from the 2019.07 LAVA docker images, but we updated to the 2019.12 images and the crash still occurs with the same error signature.

 

lava-master | 2019-10-22 15:37:17,539 DEBUG |--> [523] scheduling

lava-master | 2019-10-22 15:37:17,854 INFO [523] START => lava-dispatcher (CY8CKIT_062-01)

lava-master | 2019-10-22 15:37:17,969 INFO [523] lava-dispatcher => START_OK

lava-master | 2019-10-22 15:37:22,981 INFO [523] lava-dispatcher => END (lava-run crashed, mark job as INCOMPLETE)

lava-master | 2019-10-22 15:37:23,038 ERROR [523] Unable to dump 'description.yaml'

lava-master | 2019-10-22 15:37:23,038 ERROR [523] Compressed data ended before the end-of-stream marker was reached

lava-master | Traceback (most recent call last):

lava-master | File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 337, in _handle_end

lava-master | description = lzma.decompress(compressed_description)

lava-master | File "/usr/lib/python3.5/lzma.py", line 340, in decompress

lava-master | raise LZMAError("Compressed data ended before the "

lava-master | _lzma.LZMAError: Compressed data ended before the end-of-stream marker was reached

 

Please let me know if I can provide any other info to help debug.

 

Thanks,

Alex

This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.

_______________________________________________
Lava-users mailing list
Lava-users@lists.lavasoftware.org
https://lists.lavasoftware.org/mailman/listinfo/lava-users

 

--

Rémi Duraffort

LAVA Architect

Linaro

This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.