Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
``` $ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for http://larry.shen:xxx@xxx.nxp.com/RPC2: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))>
real 0m39.006s user 0m0.989s sys 0m0.191s ```
I check the gunicorn log, it give me: [2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for http://larry.shen:xxx@xxx.nxp.com/RPC2: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))> real 0m39.006s user 0m0.989s sys 0m0.191sI check the gunicorn log, it give me:
[2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
you can try using API instead https://your.instance.url/api/v0.2/jobs/30686/logs
I just fetched 57MB log and it worked just fine. In the long run we can probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB, it will not fine.
I guess it related to gunicorn timeout, if not able to read all log among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also timeout as it also go the way gunicorn.
I found this:
https://docs.gunicorn.org/en/stable/settings.html#timeout
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/ lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workers
Or any other suggestion you could give for this gunicorn worker timeout, I mean short turn solution...
-----Original Message----- From: Milosz Wasilewski milosz.wasilewski@linaro.org Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen larry.shen@nxp.com Cc: lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254316438&sdata=06j7jwCcUHSkcBHDzKYScsdqjqhyQnpOKiLgDqk2CmY%3D&reserved=0 for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com%2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))> real 0m39.006s user 0m0.989s sys 0m0.191sI check the gunicorn log, it give me:
[2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
you can try using API instead https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta...
I just fetched 57MB log and it worked just fine. In the long run we can probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01% 7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc 2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
On Wed, 8 Jan 2020 at 09:56, Larry Shen larry.shen@nxp.com wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB, it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also timeout as it also go the way gunicorn.
I found this:
https://docs.gunicorn.org/en/stable/settings.html#timeout
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/ lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workers
Or any other suggestion you could give for this gunicorn worker timeout, I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski milosz.wasilewski@linaro.org Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen larry.shen@nxp.com Cc: lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254316438&sdata=06j7jwCcUHSkcBHDzKYScsdqjqhyQnpOKiLgDqk2CmY%3D&reserved=0 for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com%2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))> real 0m39.006s user 0m0.989s sys 0m0.191sI check the gunicorn log, it give me:
[2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
you can try using API instead https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta...
I just fetched 57MB log and it worked just fine. In the long run we can probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01% 7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc 2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
Hello,
I tried locally with a log file of 1.1G and lavacli fails while the rest api works.
Which version of LAVA are you using?
I looked at ways to improve the speed for both xmlrpc and the rest api.
For xmlrpc, the main problem is the fact that the data is loaded in memory and then base64 encoded and then send back to the user. Nothing we can do about this without writing our own xmlrpc library (or improving the official library).
For the rest api, we can use FileResponse instead of HttpResponse to improve performance. On my local setup the download is now 60% faster. I will send a merge request.
Rgds
Le mer. 8 janv. 2020 à 11:15, Milosz Wasilewski < milosz.wasilewski@linaro.org> a écrit :
On Wed, 8 Jan 2020 at 09:56, Larry Shen larry.shen@nxp.com wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB,
it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log
among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also
timeout as it also go the way gunicorn.
I found this:
https://docs.gunicorn.org/en/stable/settings.html#timeout
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/
lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workers
Or any other suggestion you could give for this gunicorn worker timeout,
I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski milosz.wasilewski@linaro.org Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen larry.shen@nxp.com Cc: lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log,
it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101:YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml... for full details.
config = yaml.load(f_conf.read())
Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com%2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))>
real 0m39.006s
user 0m0.989s
sys 0m0.191s
I check the gunicorn log, it give me: [2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506) Any suggestion?you can try using API instead
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta...
I just fetched 57MB log and it worked just fine. In the long run we can
probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01% 7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc 2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Hi, Remi,
Next is the log, you can see it always stop at about 30 seconds, if log smaller, it’s ok. So, have your 1.3G log been fetch in 30 seconds? I think this not just related the log size, also related to the network, pc capability etc. If it could be fetched among 30 seconds, even bigger log is ok. We use 2019.11 version’s master.
$ time wget http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs --2020-01-09 14:01:57-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs Resolving lava-master.sw.nxp.com (lava-master.sw.nxp.com)... 134.27.169.17 Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: /api/v0.1/jobs/30686/logs/ [following] --2020-01-09 14:01:58-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ Reusing existing connection to lava-master.sw.nxp.com:80. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 77%[==========================================================> ] 202.48M 10.2MB/s in 31s
2020-01-09 14:02:29 (6.55 MB/s) - Connection closed at byte 212318286. Retrying.
--2020-01-09 14:02:30-- (try: 2) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 94%[=======================================================================> ] 247.13M 8.35MB/s in 31s
2020-01-09 14:03:03 (7.88 MB/s) - Connection closed at byte 259138631. Retrying.
--2020-01-09 14:03:05-- (try: 3) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 7%[====> ] 19.40M 5.34MB/s eta 53s ^C
real 1m12.372s user 0m0.402s sys 0m3.103s
With curl:
time curl -L http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs > aa % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 24 262M 24 63.5M 0 0 1950k 0 0:02:17 0:00:33 0:01:44 5395k curl: (18) transfer closed with 208899007 bytes remaining to read
real 0m33.397s user 0m0.151s sys 0m0.593s
What I miss?
From: Remi Duraffort remi.duraffort@linaro.org Sent: Wednesday, January 8, 2020 11:56 PM To: Milosz Wasilewski milosz.wasilewski@linaro.org Cc: Larry Shen larry.shen@nxp.com; lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
Caution: EXT Email Hello,
I tried locally with a log file of 1.1G and lavacli fails while the rest api works.
Which version of LAVA are you using?
I looked at ways to improve the speed for both xmlrpc and the rest api.
For xmlrpc, the main problem is the fact that the data is loaded in memory and then base64 encoded and then send back to the user. Nothing we can do about this without writing our own xmlrpc library (or improving the official library).
For the rest api, we can use FileResponse instead of HttpResponse to improve performance. On my local setup the download is now 60% faster. I will send a merge request.
Rgds
Le mer. 8 janv. 2020 à 11:15, Milosz Wasilewski <milosz.wasilewski@linaro.orgmailto:milosz.wasilewski@linaro.org> a écrit : On Wed, 8 Jan 2020 at 09:56, Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB, it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also timeout as it also go the way gunicorn.
I found this:
https://docs.gunicorn.org/en/stable/settings.html#timeouthttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fsettings.html%23timeout&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837537875&sdata=0AcBqsU%2BlIXZzGyPCEF84ov50bi2bEIxxQQTSFrRs7I%3D&reserved=0
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/ lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workershttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fdesign.html%23async-workers&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837547873&sdata=anUMalGVlw2UaLeU95nuJPJtGqv4GsHTO%2Fj8Exhu3kk%3D&reserved=0
Or any other suggestion you could give for this gunicorn worker timeout, I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski <milosz.wasilewski@linaro.orgmailto:milosz.wasilewski@linaro.org> Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> Cc: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254316438&sdata=06j7jwCcUHSkcBHDzKYScsdqjqhyQnpOKiLgDqk2CmY%3D&reserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837547873&sdata=w7extxW%2B%2BVkWl74Eb3yU8eIV2GzBqLGE%2FA9wsExjito%3D&reserved=0> for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40xxx.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837557860&sdata=V2NeT0Ca9KAhbWHgSljRInJTXS%2Bv%2FCCGVnXJSXYGMMk%3D&reserved=0>%2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))> real 0m39.006s user 0m0.989s sys 0m0.191sI check the gunicorn log, it give me:
[2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
you can try using API instead https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta...https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.instance.url%2Fapi%2Fv0.2%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837557860&sdata=aEPIpE3vTW2WeYEAAA9knWHyXlBf8jCbDChK%2BIFrr8o%3D&reserved=0
I just fetched 57MB log and it worked just fine. In the long run we can probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.orghttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fs.lavasoftware.org&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837567852&sdata=ERVSXjAXvUwb9NYU81H%2Bu2mzmn%2BDVxZ8VSjk2o6Lyis%3D&reserved=0%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01% 7Clarry.shen%40nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837567852&sdata=P%2FpuJMVhnIkFIdwQjv8wWxSJELBoC4WYZPKjat57PtI%3D&reserved=0%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc 2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
_______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-usershttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837577850&sdata=GuqybrzICjHEhWBva9DoL4wQFCPJwOQQVLGW5qfxc80%3D&reserved=0
-- Rémi Duraffort LAVA Architect Linaro
Hello,
thanks for the detailed logs.
I believe that this should be fixed by https://git.lavasoftware.org/lava/lava/merge_requests/911
Rgds
Le jeu. 9 janv. 2020 à 07:15, Larry Shen larry.shen@nxp.com a écrit :
Hi, Remi,
Next is the log, you can see it always stop at about 30 seconds, if log smaller, it’s ok.
So, have your 1.3G log been fetch in 30 seconds? I think this not just related the log size, also related to the network, pc capability etc. If it could be fetched among 30 seconds, even bigger log is ok.
We use 2019.11 version’s master.
$ time wget http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs
--2020-01-09 14:01:57-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs
Resolving lava-master.sw.nxp.com (lava-master.sw.nxp.com)... 134.27.169.17
Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /api/v0.1/jobs/30686/logs/ [following]
--2020-01-09 14:01:58-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/
Reusing existing connection to lava-master.sw.nxp.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 77%[==========================================================> ] 202.48M 10.2MB/s in 31s
2020-01-09 14:02:29 (6.55 MB/s) - Connection closed at byte 212318286. Retrying.
--2020-01-09 14:02:30-- (try: 2) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/
Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 94%[=======================================================================> ] 247.13M 8.35MB/s in 31s
2020-01-09 14:03:03 (7.88 MB/s) - Connection closed at byte 259138631. Retrying.
--2020-01-09 14:03:05-- (try: 3) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/
Connecting to lava-master.sw.nxp.com (lava-master.sw.nxp.com)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 7%[====> ] 19.40M 5.34MB/s eta 53s ^C
real 1m12.372s
user 0m0.402s
sys 0m3.103s
With curl:
time curl -L http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs > aa
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent LeftSpeed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
24 262M 24 63.5M 0 0 1950k 0 0:02:17 0:00:33 0:01:44 5395k
curl: (18) transfer closed with 208899007 bytes remaining to read
real 0m33.397s
user 0m0.151s
sys 0m0.593s
What I miss?
*From:* Remi Duraffort remi.duraffort@linaro.org *Sent:* Wednesday, January 8, 2020 11:56 PM *To:* Milosz Wasilewski milosz.wasilewski@linaro.org *Cc:* Larry Shen larry.shen@nxp.com; lava-users@lists.lavasoftware.org *Subject:* Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
*Caution: *EXT Email
Hello,
I tried locally with a log file of 1.1G and lavacli fails while the rest api works.
Which version of LAVA are you using?
I looked at ways to improve the speed for both xmlrpc and the rest api.
For xmlrpc, the main problem is the fact that the data is loaded in memory and then base64 encoded and then send back to the user. Nothing we can do about this without writing our own xmlrpc library (or improving the official library).
For the rest api, we can use FileResponse instead of HttpResponse to improve performance. On my local setup the download is now 60% faster. I will send a merge request.
Rgds
Le mer. 8 janv. 2020 à 11:15, Milosz Wasilewski < milosz.wasilewski@linaro.org> a écrit :
On Wed, 8 Jan 2020 at 09:56, Larry Shen larry.shen@nxp.com wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB,
it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log
among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also
timeout as it also go the way gunicorn.
I found this:
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/
lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workers https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fdesign.html%23async-workers&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837547873&sdata=anUMalGVlw2UaLeU95nuJPJtGqv4GsHTO%2Fj8Exhu3kk%3D&reserved=0
Or any other suggestion you could give for this gunicorn worker timeout,
I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski milosz.wasilewski@linaro.org Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen larry.shen@nxp.com Cc: lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log,
it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101:YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml... https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837547873&sdata=w7extxW%2B%2BVkWl74Eb3yU8eIV2GzBqLGE%2FA9wsExjito%3D&reserved=0 for full details.
config = yaml.load(f_conf.read())
Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40xxx.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837557860&sdata=V2NeT0Ca9KAhbWHgSljRInJTXS%2Bv%2FCCGVnXJSXYGMMk%3D&reserved=0 %2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c
om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))>
real 0m39.006s
user 0m0.989s
sys 0m0.191s
I check the gunicorn log, it give me: [2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506) Any suggestion?you can try using API instead
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta... https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.instance.url%2Fapi%2Fv0.2%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837557860&sdata=aEPIpE3vTW2WeYEAAA9knWHyXlBf8jCbDChK%2BIFrr8o%3D&reserved=0
I just fetched 57MB log and it worked just fine. In the long run we can
probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.org
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fs.lavasoftware.org&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837567852&sdata=ERVSXjAXvUwb9NYU81H%2Bu2mzmn%2BDVxZ8VSjk2o6Lyis%3D&reserved=0 %2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%
7Clarry.shen%40nxp.com
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837567852&sdata=P%2FpuJMVhnIkFIdwQjv8wWxSJELBoC4WYZPKjat57PtI%3D&reserved=0 %7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc
2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%7Clarry.shen%40nxp.com%7C689f50a0150f4c04b6dd08d794534e16%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140957837577850&sdata=GuqybrzICjHEhWBva9DoL4wQFCPJwOQQVLGW5qfxc80%3D&reserved=0
--
Rémi Duraffort
LAVA Architect
Linaro
Hi, Remi,
As 2019.12 release has some folder structure changes, so I just add your modify to “/usr/lib/python3/dist-packages/lava_rest_app/api.py” as my local master do not have base folder. But unfortunately, this didn’t help, as you said use “FileResponse” could increase efficiency, but gunicorn is before Django, I don’t know.
Still, as I said before, modify “/lib/systemd/system/lava-server-gunicorn.service” Change “ExecStart=/usr/bin/gunicorn3 lava_server.wsgi --log-level $LOGLEVEL --log-file $LOGFILE -u lavaserver -g lavaserver --workers $WORKERS $RELOAD $BIND” to “ExecStart=/usr/bin/gunicorn3 lava_server.wsgi --log-level $LOGLEVEL --timeout=600 --log-file $LOGFILE -u lavaserver -g lavaserver --workers $WORKERS $RELOAD $BIND” works for me.
Anyway, maybe my check with your code is wrong. I’m not hurry, I could wait for your final 2020.01 release, then try again, thanks.
From: Remi Duraffort remi.duraffort@linaro.org Sent: Thursday, January 9, 2020 2:48 PM To: Larry Shen larry.shen@nxp.com Cc: Milosz Wasilewski milosz.wasilewski@linaro.org; lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
Caution: EXT Email Hello,
thanks for the detailed logs.
I believe that this should be fixed by https://git.lavasoftware.org/lava/lava/merge_requests/911https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavasoftware.org%2Flava%2Flava%2Fmerge_requests%2F911&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918330791&sdata=waKT7Lk%2BT8gSj08MTA9vlax6XjI7MruIJqZ64WN1omQ%3D&reserved=0
Rgds
Le jeu. 9 janv. 2020 à 07:15, Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> a écrit : Hi, Remi,
Next is the log, you can see it always stop at about 30 seconds, if log smaller, it’s ok. So, have your 1.3G log been fetch in 30 seconds? I think this not just related the log size, also related to the network, pc capability etc. If it could be fetched among 30 seconds, even bigger log is ok. We use 2019.11 version’s master.
$ time wget http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logshttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918340792&sdata=n5zzVm0vyedDUUBCdt3MeaxSCURASrJBugz3FzR%2BbHg%3D&reserved=0 --2020-01-09 14:01:57-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logshttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918350790&sdata=PjVtJnGFUooz42YQ%2FT6eh8awLrvZJwh8Lngfr23Qo6A%3D&reserved=0 Resolving lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918350790&sdata=OSDRynbJssXugpS2fLrEIXOOkNdtwi8tdtSpBWBTtIc%3D&reserved=0 (lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918360784&sdata=6sO7kHEqZaGXjSv6BRiw%2Fr1og9wnPokSD2u4TQ%2FKbkg%3D&reserved=0)... 134.27.169.17 Connecting to lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918360784&sdata=6sO7kHEqZaGXjSv6BRiw%2Fr1og9wnPokSD2u4TQ%2FKbkg%3D&reserved=0 (lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918370777&sdata=FnhAD0A1ZOEYcSsBEmZP0u1illaF3V4cPvROWj5M9nM%3D&reserved=0)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: /api/v0.1/jobs/30686/logs/ [following] --2020-01-09 14:01:58-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918370777&sdata=25L%2BfeE0383ZG17u6ccBlZT2qJSuwTaylKGyjGElEBg%3D&reserved=0 Reusing existing connection to lava-master.sw.nxp.com:80https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%3A80&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918380774&sdata=YvsOQEwwYCKZYpf%2FjoDRi36pYuNsRKV1rrYx8hNc2%2FQ%3D&reserved=0. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 77%[==========================================================> ] 202.48M 10.2MB/s in 31s
2020-01-09 14:02:29 (6.55 MB/s) - Connection closed at byte 212318286. Retrying.
--2020-01-09 14:02:30-- (try: 2) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918390765&sdata=k%2FIKqHl98ZrVw4iqfSrqddC7rGlzmKDSpkEep7wOpTA%3D&reserved=0 Connecting to lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918390765&sdata=vCrZrmo%2FcD6xZldvpnvBfx5xOEh4JeDevSmXHUBlSz0%3D&reserved=0 (lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918400767&sdata=0pROECyS9BQCkLIsDB%2F2UW9%2BEDE9X3YJUJ1cIFgqDkc%3D&reserved=0)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 94%[=======================================================================> ] 247.13M 8.35MB/s in 31s
2020-01-09 14:03:03 (7.88 MB/s) - Connection closed at byte 259138631. Retrying.
--2020-01-09 14:03:05-- (try: 3) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918400767&sdata=WT213fzrdNwW5r%2FBTAHHIHLKqluL7Kl6mT1gLqGhiYo%3D&reserved=0 Connecting to lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918410758&sdata=EHNJOTA94NDTu4BZriABU1iDo5mvdeQ8Qs9iHgLLip4%3D&reserved=0 (lava-master.sw.nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918410758&sdata=EHNJOTA94NDTu4BZriABU1iDo5mvdeQ8Qs9iHgLLip4%3D&reserved=0)|134.27.169.17|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 275583101 (263M) [application/yaml] Saving to: ‘logs’
logs 7%[====> ] 19.40M 5.34MB/s eta 53s ^C
real 1m12.372s user 0m0.402s sys 0m3.103s
With curl:
time curl -L http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logshttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918420748&sdata=8%2BVf30t%2Fpma1CIyWhE9fvwQfkFsEOjWBManLMt9t6JA%3D&reserved=0 > aa % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 24 262M 24 63.5M 0 0 1950k 0 0:02:17 0:00:33 0:01:44 5395k curl: (18) transfer closed with 208899007 bytes remaining to read
real 0m33.397s user 0m0.151s sys 0m0.593s
What I miss?
From: Remi Duraffort <remi.duraffort@linaro.orgmailto:remi.duraffort@linaro.org> Sent: Wednesday, January 8, 2020 11:56 PM To: Milosz Wasilewski <milosz.wasilewski@linaro.orgmailto:milosz.wasilewski@linaro.org> Cc: Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com>; lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
Caution: EXT Email Hello,
I tried locally with a log file of 1.1G and lavacli fails while the rest api works.
Which version of LAVA are you using?
I looked at ways to improve the speed for both xmlrpc and the rest api.
For xmlrpc, the main problem is the fact that the data is loaded in memory and then base64 encoded and then send back to the user. Nothing we can do about this without writing our own xmlrpc library (or improving the official library).
For the rest api, we can use FileResponse instead of HttpResponse to improve performance. On my local setup the download is now 60% faster. I will send a merge request.
Rgds
Le mer. 8 janv. 2020 à 11:15, Milosz Wasilewski <milosz.wasilewski@linaro.orgmailto:milosz.wasilewski@linaro.org> a écrit : On Wed, 8 Jan 2020 at 09:56, Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB, it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also timeout as it also go the way gunicorn.
I found this:
https://docs.gunicorn.org/en/stable/settings.html#timeouthttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fsettings.html%23timeout&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918430749&sdata=jQEaKGV2FAPLT%2BvdewNdEq8LgUtJMCd8DgmOXU4oeos%3D&reserved=0
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/ lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workershttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fdesign.html%23async-workers&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918430749&sdata=oul%2BbnPASWWECLP48y6mFMYAA07Cu5YX0maTKq3Xouo%3D&reserved=0
Or any other suggestion you could give for this gunicorn worker timeout, I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski <milosz.wasilewski@linaro.orgmailto:milosz.wasilewski@linaro.org> Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> Cc: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen <larry.shen@nxp.commailto:larry.shen@nxp.com> wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log, it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254316438&sdata=06j7jwCcUHSkcBHDzKYScsdqjqhyQnpOKiLgDqk2CmY%3D&reserved=0<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918440745&sdata=ZfWs%2FfG2hqWV0NuhygSNTm8EWv53bwPPj0udQgGFY0s%3D&reserved=0> for full details. config = yaml.load(f_conf.read()) Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40xxx.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918450742&sdata=H6XG3OhQoiryqSIkksjBQZ3olUiOdqII9mjx5cFCBQk%3D&reserved=0>%2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))> real 0m39.006s user 0m0.989s sys 0m0.191sI check the gunicorn log, it give me:
[2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506)
Any suggestion?
you can try using API instead https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta...https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.instance.url%2Fapi%2Fv0.2%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918450742&sdata=CqwpDK4wSjNCSqig5Es6u39RYhmDgzWTo13v494hdyI%3D&reserved=0
I just fetched 57MB log and it worked just fine. In the long run we can probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.orghttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fs.lavasoftware.org&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918460739&sdata=pEkiLbmxiagISYJZUVJzkdYfd5reuystU7QGwzDS4DU%3D&reserved=0%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01% 7Clarry.shen%40nxp.comhttps://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918460739&sdata=cQpT132JkOVywNJpzv1nPNgTQVLYGXpFGEjDRtJMMsc%3D&reserved=0%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc 2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
_______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.orgmailto:Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-usershttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918470736&sdata=3uWn7rDGU88QKj5tfmgFFIeK08ZNxzbu%2Fpjn80yB%2B0M%3D&reserved=0
-- Rémi Duraffort LAVA Architect Linaro
-- Rémi Duraffort LAVA Architect Linaro
Hello,
if this optimization (using FileResponse) is not enough for your use case, that's maybe only because the connection is not fast enough between the server and your client.
As you said, the only solution is to increase Gunicorn timeout.
Rgds
Le jeu. 9 janv. 2020 à 08:34, Larry Shen larry.shen@nxp.com a écrit :
Hi, Remi,
As 2019.12 release has some folder structure changes, so I just add your modify to “/usr/lib/python3/dist-packages/lava_rest_app/api.py” as my local master do not have base folder.
But unfortunately, this didn’t help, as you said use “FileResponse” could increase efficiency, but gunicorn is before Django, I don’t know.
Still, as I said before, modify “/lib/systemd/system/lava-server-gunicorn.service”
Change “ExecStart=/usr/bin/gunicorn3 lava_server.wsgi --log-level $LOGLEVEL --log-file $LOGFILE -u lavaserver -g lavaserver --workers $WORKERS $RELOAD $BIND” to
“ExecStart=/usr/bin/gunicorn3 lava_server.wsgi --log-level $LOGLEVEL --timeout=600 --log-file $LOGFILE -u lavaserver -g lavaserver --workers $WORKERS $RELOAD $BIND” works for me.
Anyway, maybe my check with your code is wrong. I’m not hurry, I could wait for your final 2020.01 release, then try again, thanks.
*From:* Remi Duraffort remi.duraffort@linaro.org *Sent:* Thursday, January 9, 2020 2:48 PM *To:* Larry Shen larry.shen@nxp.com *Cc:* Milosz Wasilewski milosz.wasilewski@linaro.org; lava-users@lists.lavasoftware.org *Subject:* Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
*Caution: *EXT Email
Hello,
thanks for the detailed logs.
I believe that this should be fixed by https://git.lavasoftware.org/lava/lava/merge_requests/911 https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.lavasoftware.org%2Flava%2Flava%2Fmerge_requests%2F911&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918330791&sdata=waKT7Lk%2BT8gSj08MTA9vlax6XjI7MruIJqZ64WN1omQ%3D&reserved=0
Rgds
Le jeu. 9 janv. 2020 à 07:15, Larry Shen larry.shen@nxp.com a écrit :
Hi, Remi,
Next is the log, you can see it always stop at about 30 seconds, if log smaller, it’s ok.
So, have your 1.3G log been fetch in 30 seconds? I think this not just related the log size, also related to the network, pc capability etc. If it could be fetched among 30 seconds, even bigger log is ok.
We use 2019.11 version’s master.
$ time wget http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918340792&sdata=n5zzVm0vyedDUUBCdt3MeaxSCURASrJBugz3FzR%2BbHg%3D&reserved=0
--2020-01-09 14:01:57-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918350790&sdata=PjVtJnGFUooz42YQ%2FT6eh8awLrvZJwh8Lngfr23Qo6A%3D&reserved=0
Resolving lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918350790&sdata=OSDRynbJssXugpS2fLrEIXOOkNdtwi8tdtSpBWBTtIc%3D&reserved=0 (lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918360784&sdata=6sO7kHEqZaGXjSv6BRiw%2Fr1og9wnPokSD2u4TQ%2FKbkg%3D&reserved=0)... 134.27.169.17
Connecting to lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918360784&sdata=6sO7kHEqZaGXjSv6BRiw%2Fr1og9wnPokSD2u4TQ%2FKbkg%3D&reserved=0 (lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918370777&sdata=FnhAD0A1ZOEYcSsBEmZP0u1illaF3V4cPvROWj5M9nM%3D&reserved=0)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /api/v0.1/jobs/30686/logs/ [following]
--2020-01-09 14:01:58-- http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918370777&sdata=25L%2BfeE0383ZG17u6ccBlZT2qJSuwTaylKGyjGElEBg%3D&reserved=0
Reusing existing connection to lava-master.sw.nxp.com:80 https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%3A80&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918380774&sdata=YvsOQEwwYCKZYpf%2FjoDRi36pYuNsRKV1rrYx8hNc2%2FQ%3D&reserved=0 .
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 77%[==========================================================> ] 202.48M 10.2MB/s in 31s
2020-01-09 14:02:29 (6.55 MB/s) - Connection closed at byte 212318286. Retrying.
--2020-01-09 14:02:30-- (try: 2) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918390765&sdata=k%2FIKqHl98ZrVw4iqfSrqddC7rGlzmKDSpkEep7wOpTA%3D&reserved=0
Connecting to lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918390765&sdata=vCrZrmo%2FcD6xZldvpnvBfx5xOEh4JeDevSmXHUBlSz0%3D&reserved=0 (lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918400767&sdata=0pROECyS9BQCkLIsDB%2F2UW9%2BEDE9X3YJUJ1cIFgqDkc%3D&reserved=0)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 94%[=======================================================================> ] 247.13M 8.35MB/s in 31s
2020-01-09 14:03:03 (7.88 MB/s) - Connection closed at byte 259138631. Retrying.
--2020-01-09 14:03:05-- (try: 3) http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs/ https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs%2F&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918400767&sdata=WT213fzrdNwW5r%2FBTAHHIHLKqluL7Kl6mT1gLqGhiYo%3D&reserved=0
Connecting to lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918410758&sdata=EHNJOTA94NDTu4BZriABU1iDo5mvdeQ8Qs9iHgLLip4%3D&reserved=0 (lava-master.sw.nxp.com https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918410758&sdata=EHNJOTA94NDTu4BZriABU1iDo5mvdeQ8Qs9iHgLLip4%3D&reserved=0)|134.27.169.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 275583101 (263M) [application/yaml]
Saving to: ‘logs’
logs 7%[====> ] 19.40M 5.34MB/s eta 53s ^C
real 1m12.372s
user 0m0.402s
sys 0m3.103s
With curl:
time curl -L http://lava-master.sw.nxp.com/api/v0.1/jobs/30686/logs https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flava-master.sw.nxp.com%2Fapi%2Fv0.1%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918420748&sdata=8%2BVf30t%2Fpma1CIyWhE9fvwQfkFsEOjWBManLMt9t6JA%3D&reserved=0
aa
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent LeftSpeed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
24 262M 24 63.5M 0 0 1950k 0 0:02:17 0:00:33 0:01:44 5395k
curl: (18) transfer closed with 208899007 bytes remaining to read
real 0m33.397s
user 0m0.151s
sys 0m0.593s
What I miss?
*From:* Remi Duraffort remi.duraffort@linaro.org *Sent:* Wednesday, January 8, 2020 11:56 PM *To:* Milosz Wasilewski milosz.wasilewski@linaro.org *Cc:* Larry Shen larry.shen@nxp.com; lava-users@lists.lavasoftware.org *Subject:* Re: [Lava-users] [EXT] Re: lavacli cannot fetch big logs.
*Caution: *EXT Email
Hello,
I tried locally with a log file of 1.1G and lavacli fails while the rest api works.
Which version of LAVA are you using?
I looked at ways to improve the speed for both xmlrpc and the rest api.
For xmlrpc, the main problem is the fact that the data is loaded in memory and then base64 encoded and then send back to the user. Nothing we can do about this without writing our own xmlrpc library (or improving the official library).
For the rest api, we can use FileResponse instead of HttpResponse to improve performance. On my local setup the download is now 60% faster. I will send a merge request.
Rgds
Le mer. 8 janv. 2020 à 11:15, Milosz Wasilewski < milosz.wasilewski@linaro.org> a écrit :
On Wed, 8 Jan 2020 at 09:56, Larry Shen larry.shen@nxp.com wrote:
Yes, Milosz,
57MB is fine, in fact, our 100M+ also fine, just if bigger, like 270MB,
it will not fine.
ah, ok. I thought there was some other issue with cli.
I guess it related to gunicorn timeout, if not able to read all log
among the timeout, it will fail.
I tried your suggestion, but even I switch to use rest api, it will also
timeout as it also go the way gunicorn.
I found this:
It said:
Default gunicorn was set 30 seconds for worker timeout.
It seems I have to change " /lib/systemd/system/
lava-server-gunicorn.service" to enlarger the timeout. Just wonder in the day which async http server so popular, if possible lava could adopt some such kinds of technology? Like next: https://docs.gunicorn.org/en/stable/design.html#async-workers https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gunicorn.org%2Fen%2Fstable%2Fdesign.html%23async-workers&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918430749&sdata=oul%2BbnPASWWECLP48y6mFMYAA07Cu5YX0maTKq3Xouo%3D&reserved=0
Or any other suggestion you could give for this gunicorn worker timeout,
I mean short turn solution...
I'll defer to Remi. IMHO it would make sense to implement streaming response somewhere. The down side is you don't know the download size in advance but it should not time out.
milosz
-----Original Message----- From: Milosz Wasilewski milosz.wasilewski@linaro.org Sent: Wednesday, January 8, 2020 5:07 PM To: Larry Shen larry.shen@nxp.com Cc: lava-users@lists.lavasoftware.org Subject: [EXT] Re: [Lava-users] lavacli cannot fetch big logs.
Caution: EXT Email
On Wed, 8 Jan 2020 at 08:18, Larry Shen larry.shen@nxp.com wrote:
Hi, there,
We have one job to do some stress test, the final log is about 270MB.
We use `lavacli -i production jobs logs --raw 30686` to fetch the log,
it gives me next:
$ time lavacli -i production jobs logs --raw 30686 /usr/lib/python3/dist-packages/lavacli/__init__.py:101:YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml... https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmsg.pyyaml.org%2Fload&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918440745&sdata=ZfWs%2FfG2hqWV0NuhygSNTm8EWv53bwPPj0udQgGFY0s%3D&reserved=0 for full details.
config = yaml.load(f_conf.read())
Unable to call 'jobs.logs': <ProtocolError for https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flarry .shen%3Axxx%40xxx.nxp.com
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40xxx.nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918450742&sdata=H6XG3OhQoiryqSIkksjBQZ3olUiOdqII9mjx5cFCBQk%3D&reserved=0 %2FRPC2&data=02%7C01%7Clarry.shen%40nxp.c
om%7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc2b4c6fa92cd99c5c3016 35%7C0%7C0%7C637140712254316438&sdata=t2frtOwreMTjSSVMDca1ob2T5M0% 2B8QCpBcaXUL1tArc%3D&reserved=0: 500 ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))>
real 0m39.006s
user 0m0.989s
sys 0m0.191s
I check the gunicorn log, it give me: [2020-01-08 08:11:12 +0000] [188] [CRITICAL] WORKER TIMEOUT (pid:11506) Any suggestion?you can try using API instead
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.insta... https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyour.instance.url%2Fapi%2Fv0.2%2Fjobs%2F30686%2Flogs&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918450742&sdata=CqwpDK4wSjNCSqig5Es6u39RYhmDgzWTo13v494hdyI%3D&reserved=0
I just fetched 57MB log and it worked just fine. In the long run we can
probably switch lavacli to use REST API.
milosz
Lava-users mailing list Lava-users@lists.lavasoftware.org https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist s.lavasoftware.org
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fs.lavasoftware.org&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918460739&sdata=pEkiLbmxiagISYJZUVJzkdYfd5reuystU7QGwzDS4DU%3D&reserved=0 %2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%
7Clarry.shen%40nxp.com
https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F40nxp.com&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918460739&sdata=cQpT132JkOVywNJpzv1nPNgTQVLYGXpFGEjDRtJMMsc%3D&reserved=0 %7Ca778c428e4a947d9b3ab08d7941a20f6%7C686ea1d3bc
2b4c6fa92cd99c5c301635%7C0%7C0%7C637140712254326435&sdata=s2KqYKrm PfCEBOSjF93IpV07I0P6iMp5zQtRRDfYgjU%3D&reserved=0
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lavasoftware.org%2Fmailman%2Flistinfo%2Flava-users&data=02%7C01%7Clarry.shen%40nxp.com%7C6b09d84bdf784baa624408d794cfe3cd%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637141492918470736&sdata=3uWn7rDGU88QKj5tfmgFFIeK08ZNxzbu%2Fpjn80yB%2B0M%3D&reserved=0
--
Rémi Duraffort
LAVA Architect
Linaro
--
Rémi Duraffort
LAVA Architect
Linaro
lava-users@lists.lavasoftware.org
-
Larry Shen -
Milosz Wasilewski -
Remi Duraffort