Hi Remi,
The docker-compose logs are the only logs I’m aware of. Is the job log what appears in the web interface for the job, i.e. in this case it would be http://lava-server/scheduler/job/4284 ? There are no logs there in this case, the crash happens before any logs are written to the job log. If the job log is something different, please let me know where I can find it and I will send it to you.
From what I can see, it looks like the description.yaml is sent to the master successfully, but on rare occasions a problem occurs with the master sending it to the slave.
Thanks, Alex
From: Remi Duraffort remi.duraffort@linaro.org Sent: Thursday, January 23, 2020 2:25 AM To: Alexander Moore Alexander.Moore@cypress.com Cc: lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] lava-run crash after submitting a job
Hello,
the relevant logs are:
lava-dispatcher | 2020-01-20 20:03:50,031 INFO [4284] Job END lava-dispatcher | 2020-01-20 20:03:50,031 ERROR [4284] Unable to read 'description.yaml' lava-dispatcher | 2020-01-20 20:03:50,032 ERROR [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/4284/description.yaml' lava-dispatcher | Traceback (most recent call last): lava-dispatcher | File "/usr/bin/lava-slave", line 239, in description lava-dispatcher | data = open(filename, 'r').read() lava-dispatcher | FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/slave/tmp/4284/description.yaml' lava-dispatcher | 2020-01-20 20:03:50,034 ERROR [4284] lava-run crashed
The main problem is that we don't see anything in this log.
Could you provide the job logs?
Le mer. 22 janv. 2020 à 20:14, Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> a écrit : Hi Remi,
For more information, I have followed https://collaborate.linaro.org/pages/viewpage.action?pageId=118293253 which uses a docker-compose setup from https://github.com/Linaro/lite-lava-docker-compose
Below pastebin has all docker-compose logs for recent lava-run crash (job 4284) using 2019.12 images. https://pastebin.com/hEXZ7Nbm
Thanks, Alex
From: Remi Duraffort <remi.duraffort@linaro.orgmailto:remi.duraffort@linaro.org> Sent: Wednesday, January 22, 2020 6:01 AM To: Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> Cc: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] lava-run crash after submitting a job
If you use the lava-dispatcher docker container, then the lava-slve logs are printed on stdout. So "docker logs" would give you the logs.
Rgds
Le mar. 21 janv. 2020 à 22:59, Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> a écrit : In my lava-dispatcher docker container, /var/log/lava-dispatcher/ exists but is empty.
Thanks, Alex
From: Remi Duraffort <remi.duraffort@linaro.orgmailto:remi.duraffort@linaro.org> Sent: Tuesday, January 21, 2020 8:37 AM To: Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> Cc: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] lava-run crash after submitting a job
On the dispatcher, look for /var/log/lava-dispatcher/lava-slave.log
Le mar. 21 janv. 2020 à 16:58, Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> a écrit : Hi Remi,
The log states “lava-run crashed”, I don’t think there is any lava-master crash because lava-master continues working fine for subsequent jobs. How can I get the lava-slave logs to check for a crash there?
Thanks, Alex
From: Remi Duraffort <remi.duraffort@linaro.orgmailto:remi.duraffort@linaro.org> Sent: Tuesday, January 21, 2020 12:52 AM To: Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> Cc: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] lava-run crash after submitting a job
Hello,
this is lava-master crashing, not lava-run right ? If lava-run is crashing, please provide the corresponding lava-slave logs.
Rgds
Le lun. 20 janv. 2020 à 23:34, Alexander Moore <Alexander.Moore@cypress.commailto:Alexander.Moore@cypress.com> a écrit : Hello,
Our internal LAVA setup has been hitting this crash intermittently (it reproduces about one out of every 30 job submissions). The below log snippet is from the 2019.07 LAVA docker images, but we updated to the 2019.12 images and the crash still occurs with the same error signature.
lava-master | 2019-10-22 15:37:17,539 DEBUG |--> [523] scheduling lava-master | 2019-10-22 15:37:17,854 INFO [523] START => lava-dispatcher (CY8CKIT_062-01) lava-master | 2019-10-22 15:37:17,969 INFO [523] lava-dispatcher => START_OK lava-master | 2019-10-22 15:37:22,981 INFO [523] lava-dispatcher => END (lava-run crashed, mark job as INCOMPLETE) lava-master | 2019-10-22 15:37:23,038 ERROR [523] Unable to dump 'description.yaml' lava-master | 2019-10-22 15:37:23,038 ERROR [523] Compressed data ended before the end-of-stream marker was reached lava-master | Traceback (most recent call last): lava-master | File "/usr/lib/python3/dist-packages/lava_server/management/commands/lava-master.py", line 337, in _handle_end lava-master | description = lzma.decompress(compressed_description) lava-master | File "/usr/lib/python3.5/lzma.py", line 340, in decompress lava-master | raise LZMAError("Compressed data ended before the " lava-master | _lzma.LZMAError: Compressed data ended before the end-of-stream marker was reached
Please let me know if I can provide any other info to help debug.
Thanks, Alex
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message. _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
-- Rémi Duraffort LAVA Architect Linaro
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.
-- Rémi Duraffort LAVA Architect Linaro
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.
-- Rémi Duraffort LAVA Architect Linaro
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.
-- Rémi Duraffort LAVA Architect Linaro
This message and any attachments may contain confidential information from Cypress or its subsidiaries. If it has been received in error, please advise the sender and immediately delete this message.