Hello Milosz,

Sorry I missed giving full boot action in my previous mail:

- boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'login:' auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "***"

Error: auto-login-action timed out after 900 seconds end: 2.2 auto-login-action (duration 00:15:00) [common] case: auto-login-action definition: lava duration: 900.00 extra: ... level: 2.2 namespace: common result: fail boot-image-retry failed: 1 of 1 attempts. 'auto-login-action timed out after 900 seconds' auto-login-action timed out after 900 seconds end: 2 boot-image-retry (duration 00:15:00) [common]

Kindly suggest.

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Friday, October 18, 2024 1:42 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Sweta,

On Thu, Oct 17, 2024 at 4:35 PM sweta.ghosh@nagarro.com wrote:

Hello, We are testing our custom debian image and have added banner messages before login and after login and after the implementation LAVA job is failing on auto-login. Please find example banner messages: |_ _/ | | | | _| | |_____\ Info: Example.....: ABC Example......: ABC IP: etho.......:123 Info:

1. some info

Some Info:

1. ###

Some more info: ####

login :

It looks like due to a huge banner message before the login prompt it is getting confused and due to this, it gets stuck at auto login and timeouts. Please suggest if we can ignore this in LAVA to login and execute the test case

Please find the job details: device_type: qemu job_name: banner timeouts: job: minutes: 30 action: minutes: 25 connection: minutes: 5 priority: medium visibility: public context: arch: amd64 lava_test_results_dir: "/home/lava-%s" actions:

• deploy: timeout: minutes: 15 to: tmpfs images: rootfs: url: file:///home/image image_arg: XXXX

• boot: timeout: minutes: 15 method: qemu media: tmpfs prompts:

  • 'login:'

I think this might be your issue. Did you try to change the prompt to sth else?

auto_login:
  login_prompt: "login:"
  username: abc
  password_prompt: "Password:"
  password: "abc"

Best Regards, Milosz

Sweta,

On Thu, Oct 24, 2024 at 8:12 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

Hello Milosz,

I couldn't understand what you mean by " Did you try to change the prompt to sth else?" . Could you please share an example , how can I do the same.

This part:

...

prompts:
- 'login:'

I think this might be your issue. Did you try to change the prompt to sth else?

Try changing the prompt and see if it helps.

Best Regards, Milosz

Hey Sweta,

thanks for the longer log, this sheds some more light on the issue - but still isn't enough to debug it.

First regarding the promt which Milosz already mentioned: You set it to "login:" - that most probably is wrong. Linux hosts usually have a prompt something like "username@hostname:" e.g. "root@example:". Please set yaml config to the promt your device really uses.

Then regarding your logs:

Does the line "[login-action] Waiting for messages, (timeout 00:0x:xx)" appear in logs? I guess yes.

Does the line "Matched prompt #xx: login:" and "Sending username root" appear? At which line? If yes, the login prompt (or something earlier in your banner message) gets matched - and it's not LAVAs fault.

As we can see in your screenshot, your login prompt isn't just "login:" - it is "do[redacted]t login:". Set your yaml config "login_prompt" to this string, this will make detection of that login prompt less error prone.

Regarding your code point question: Just search the code for "login_prompt" and you will find https://gitlab.com/lava/lava/-/blob/master/lava_dispatcher/actions/boot/__in... As far as I can see there's just a time limit and no line limit for detecting the login prompt.

Best regards

Stefan

On 10/25/2024 8:40 AM, Sweta Ghosh wrote:

Hello Milosz,

Thanks for replying. I tried the changing the prompts by adding strings from welcome messages and login_prompt but it looks like it is not able to reach that point where my login prompt is displayed. My login prompt(circled) is displayed in logs only after auto-login fails. Please find the screenshot attached. Is there is any limit to number of line it try to match the prompt?

Could you please point me to the code that lava uses to detect the prompt. Thanks, Sweta

---

*From:* Milosz Wasilewski milosz.wasilewski@foundries.io *Sent:* Thursday, October 24, 2024 1:07 PM *To:* Sweta Ghosh sweta.ghosh@nagarro.com *Cc:* lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org *Subject:* Re: [lava-users] Re: Auto login fails after adding banner message before login prompt   [Email from a non-Nagarro source: please exercise caution with links and attachments]

Sweta,

On Thu, Oct 24, 2024 at 8:12 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

...

Hello Milosz,

I couldn't understand what you mean by " Did you try to change the prompt to sth else?" . Could you please share an example , how can I do the same.

This part:

...

...

prompts:      - 'login:'

I think this might be your issue. Did you try to change the prompt to sth else?

Try changing the prompt and see if it helps.

Best Regards, Milosz

---

lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

Hello,

I have tried to reduce the welcome message to one line i.e System Info: . But even after the it is not able to match the prompt to auto login . Please find the error screenshot attached and job description below:

In the job logs there was no "matched prompt". Also, I tried login_prompt: "device-mgmt login:" but it didn't work.

- boot: timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "do[redacted]t login:" username: root password_prompt: "Password:" password: "root" prompts: - 'System Info: device-mgmt login:' - 'System Info: device-mgmt login:' - 'System Info:device-mgmt login:' - 'System Info:' - 'device-mgmt login:' - 'root@device-mgmt'

[cid:a9a21c78-b927-41f9-9e38-7568e7884b0f]

It looks like LAVA is not able to filter the prompt due to welcome message. Could you please suggest , how can I get logs to see how these messages are visible to LAVA and maybe I can add the exact value it is expecting.

Should I open a bug for the same? Kindly suggest if there is a workaround or if I am missing something in my job description.

Thanks, Sweta ________________________________ From: Sweta Ghosh sweta.ghosh@nagarro.com Sent: Friday, October 25, 2024 2:54 PM To: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

Hello Stefan,

Thanks for replying.

* Does the line "[login-action] Waiting for messages, (timeout 00:0x:xx)" appear in logs? I guess yes. I can see error "Job error: auto-login-action timed out after 900 seconds" "auto-login-action: Wait for prompt ['Linux version [0-9]'] (timeout 00:15:00)"[cid:1bdaa9d2-2de6-4b9e-92d2-d8c289e454fb] But I couldn't find "[login-action] Waiting for messages, (timeout 00:0x:xx)" in my logs. * Does the line "Matched prompt #xx: login:" and "Sending username root" appear? No

Please find the job details of the logs i shared earlier: - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'Variant.....: XXXXX' - 'HOST login:' - 'root@HOST' auto_login: login_prompt: "HOST login:" username: root password_prompt: "Password:" password: "root"

Also tried - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'Variant.....: XXXXX' - 'HOST login:' - 'root@HOST' auto_login: login_prompt: "do[redacted]t login:" username: root password_prompt: "Password:" password: "root"

Getting logs: wait for prompt timed out end: 2.2 auto-login-action (duration 00:12:03) [common] definition: lavahttps://lava.debian.in/results/226/lava/auto-login-action namespace: commonhttps://lava.debian.in/results/226/lava/auto-login-action case: auto-login-actionhttps://lava.debian.in/results/226/lava/auto-login-action level: 2.2https://lava.debian.in/results/226/lava/auto-login-action duration: 722.85https://lava.debian.in/results/226/lava/auto-login-action result: failhttps://lava.debian.in/results/226/lava/auto-login-action extra: ...https://lava.debian.in/results/226/lava/auto-login-action boot-image-retry failed: 1 of 1 attempts. 'wait for prompt timed out' wait for prompt timed out end: 2 boot-image-retry (duration 00:12:03) [common] definition: lavahttps://lava.debian.in/results/226/lava/boot-image-retry namespace: commonhttps://lava.debian.in/results/226/lava/boot-image-retry case: boot-image-retryhttps://lava.debian.in/results/226/lava/boot-image-retry level: 2https://lava.debian.in/results/226/lava/boot-image-retry duration: 722.86https://lava.debian.in/results/226/lava/boot-image-retry result: failhttps://lava.debian.in/results/226/lava/boot-image-retry extra: ...https://lava.debian.in/results/226/lava/boot-image-retry Cleaning after the job Cleaning up download directory: /var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs start: 4.1 power-off (timeout 00:00:10) [common] end: 4.1 power-off (duration 00:00:00) [common] start: 4.2 read-feedback (timeout 00:05:00) [common] Listened to connection for namespace 'common' for up to 1s Finalising connection for namespace 'common' Sending line: 'poweroff' poweroff Already disconnected device-mgmt login: poweroff Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 173, in expect proc = super().expect(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 343, in expect return self.expect_list(compiled_pattern_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 372, in expect_list return exp.expect_loop(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop return self.timeout(e) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout raise exc pexpect.exceptions.TIMEOUT: Timeout exceeded. <lava_dispatcher.shell.ShellCommand object at 0x7fc3ec2a43d0> command: /usr/bin/qemu-system-x86_64 args: [b'/usr/bin/qemu-system-x86_64', b'-cpu', b'host', b'-enable-kvm', b'-nographic', b'-net', b'nic,model=virtio,macaddr=52:54:00:12:34:58', b'-net', b'user', b'-m', b'1024', b'-monitor', b'none', b'-global', b'ICH9-LPC.disable_s3=1', b'-global', b'isa-fdc.driveA=', b'-drive', b'if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd', b'-drive', b'file=/var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs/XXXXX-qemu_amd64.wic,discard=unmap,if=none,id=disk,format=raw', b'-m', b'8G', b'-monitor', b'none', b'-serial', b'mon:stdio', b'-display', b'none', b'-cpu', b'qemu64', b'-smp', b'4', b'-machine', b'q35,accel=kvm:tcg', b'-global', b'ICH9-LPC.noreboot=off', b'-device', b'ide-hd,drive=disk', b'-nographic', b'-chardev', b'socket,id=chrtpm,path=/tmp/swtpm.sock/qemu-swtpm.sock', b'-tpmdev', b'emulator,id=tpm0,chardev=chrtpm', b'-device', b'tpm-tis,tpmdev=tpm0', b'-drive', b'format=qcow2,file=/var/lib/lava/dispatcher/tmp/226/apply-overlay-guest-_kawe9vb/lava-guest.qcow2,media=disk,if=ide,id=lavatest'] buffer (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' before (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 6170 child_fd: 10 closed: False timeout: 900 delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: <lava_dispatcher.shell.ShellLogger object at 0x7fc3f188d190> logfile_send: <lava_dispatcher.shell.ShellLogger object at 0x7fc3ec48bb10> maxread: 2000 ignorecase: False searchwindowsize: 4000 delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 1.0 searcher: searcher_re: 0: re.compile('Linux version [0-9]') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 301, in wait return self.raw_connection.expect(self.prompt_str, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 178, in expect raise TestError("ShellCommand command timed out.") lava_common.exceptions.TestError: ShellCommand command timed out. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 239, in run self._run() File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 229, in _run self.pipeline.run_actions(self.connection, max_end_time) File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/logical.py", line 62, in run connection = self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/actions/boot/__init__.py", line 328, in run res = self.wait(connection) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 943, in wait return connection.wait(max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 303, in wait raise JobError("wait for prompt timed out") lava_common.exceptions.JobError: wait for prompt timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 23, in _wrap_ptyprocess_err yield File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 328, in close self.ptyproc.close(force=force) File "/usr/lib/python3/dist-packages/ptyprocess/ptyprocess.py", line 406, in close raise PtyProcessError('Could not terminate the child.') ptyprocess.util.PtyProcessError: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 308, in run feedback[1].finalise() File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 246, in finalise self.disconnect("closing") File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 253, in disconnect super().disconnect(reason) File "/usr/lib/python3/dist-packages/lava_dispatcher/connection.py", line 153, in disconnect self.raw_connection.close(force=True) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 325, in close with _wrap_ptyprocess_err(): File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 25, in _wrap_ptyprocess_err raise ExceptionPexpect(*e.args) pexpect.exceptions.ExceptionPexpect: Could not terminate the child. end: 4.2 read-feedback (duration 00:00:02) [common] definition: lavahttps://lava.debian.in/results/226/lava/read-feedback namespace: commonhttps://lava.debian.in/results/226/lava/read-feedback case: read-feedbackhttps://lava.debian.in/results/226/lava/read-feedback level: 4.2https://lava.debian.in/results/226/lava/read-feedback duration: 1.70https://lava.debian.in/results/226/lava/read-feedback result: failhttps://lava.debian.in/results/226/lava/read-feedback extra: ...https://lava.debian.in/results/226/lava/read-feedback Failed to run 'finalize': Could not terminate the child. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 173, in expect proc = super().expect(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 343, in expect return self.expect_list(compiled_pattern_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 372, in expect_list return exp.expect_loop(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop return self.timeout(e) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout raise exc pexpect.exceptions.TIMEOUT: Timeout exceeded. <lava_dispatcher.shell.ShellCommand object at 0x7fc3ec2a43d0> command: /usr/bin/qemu-system-x86_64 args: [b'/usr/bin/qemu-system-x86_64', b'-cpu', b'host', b'-enable-kvm', b'-nographic', b'-net', b'nic,model=virtio,macaddr=52:54:00:12:34:58', b'-net', b'user', b'-m', b'1024', b'-monitor', b'none', b'-global', b'ICH9-LPC.disable_s3=1', b'-global', b'isa-fdc.driveA=', b'-drive', b'if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd', b'-drive', b'file=/var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs/XXXX-qemu_amd64.wic,discard=unmap,if=none,id=disk,format=raw', b'-m', b'8G', b'-monitor', b'none', b'-serial', b'mon:stdio', b'-display', b'none', b'-cpu', b'qemu64', b'-smp', b'4', b'-machine', b'q35,accel=kvm:tcg', b'-global', b'ICH9-LPC.noreboot=off', b'-device', b'ide-hd,drive=disk', b'-nographic', b'-chardev', b'socket,id=chrtpm,path=/tmp/swtpm.sock/qemu-swtpm.sock', b'-tpmdev', b'emulator,id=tpm0,chardev=chrtpm', b'-device', b'tpm-tis,tpmdev=tpm0', b'-drive', b'format=qcow2,file=/var/lib/lava/dispatcher/tmp/226/apply-overlay-guest-_kawe9vb/lava-guest.qcow2,media=disk,if=ide,id=lavatest'] buffer (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' before (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 6170 child_fd: 10 closed: False timeout: 900 delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: <lava_dispatcher.shell.ShellLogger object at 0x7fc3f188d190> logfile_send: <lava_dispatcher.shell.ShellLogger object at 0x7fc3ec48bb10> maxread: 2000 ignorecase: False searchwindowsize: 4000 delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 1.0 searcher: searcher_re: 0: re.compile('Linux version [0-9]') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 301, in wait return self.raw_connection.expect(self.prompt_str, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 178, in expect raise TestError("ShellCommand command timed out.") lava_common.exceptions.TestError: ShellCommand command timed out. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 239, in run self._run() File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 229, in _run self.pipeline.run_actions(self.connection, max_end_time) File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/logical.py", line 62, in run connection = self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/actions/boot/__init__.py", line 328, in run res = self.wait(connection) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 943, in wait return connection.wait(max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 303, in wait raise JobError("wait for prompt timed out") lava_common.exceptions.JobError: wait for prompt timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 23, in _wrap_ptyprocess_err yield File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 328, in close self.ptyproc.close(force=force) File "/usr/lib/python3/dist-packages/ptyprocess/ptyprocess.py", line 406, in close raise PtyProcessError('Could not terminate the child.') ptyprocess.util.PtyProcessError: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 308, in run feedback[1].finalise() File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 246, in finalise self.disconnect("closing") File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 253, in disconnect super().disconnect(reason) File "/usr/lib/python3/dist-packages/lava_dispatcher/connection.py", line 153, in disconnect self.raw_connection.close(force=True) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 325, in close with _wrap_ptyprocess_err(): File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 25, in _wrap_ptyprocess_err raise ExceptionPexpect(*e.args) pexpect.exceptions.ExceptionPexpect: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 341, in run connection = super().run(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 848, in run return self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 278, in run_actions raise LAVABug(str(exc)) lava_common.exceptions.LAVABug: Could not terminate the child. Removing root tmp directory at /var/lib/lava/dispatcher/tmp/226 JobError: Your job cannot terminate cleanly. definition: lavahttps://lava.debian.in/results/226/lava/job case: jobhttps://lava.debian.in/results/226/lava/job result: failhttps://lava.debian.in/results/226/lava/job error_msg: wait for prompt timed outhttps://lava.debian.in/results/226/lava/job error_type: Jobhttps://lava.debian.in/results/226/lava/job

Thanks, Sweta ________________________________ From: Stefan lists.lavasoftware.org_23@green-sparklet.de Sent: Friday, October 25, 2024 1:17 PM To: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Hey Sweta,

thanks for the longer log, this sheds some more light on the issue - but still isn't enough to debug it.

First regarding the promt which Milosz already mentioned: You set it to "login:" - that most probably is wrong. Linux hosts usually have a prompt something like "username@hostname:" e.g. "root@example:". Please set yaml config to the promt your device really uses.

Then regarding your logs:

Does the line "[login-action] Waiting for messages, (timeout 00:0x:xx)" appear in logs? I guess yes.

Does the line "Matched prompt #xx: login:" and "Sending username root" appear? At which line? If yes, the login prompt (or something earlier in your banner message) gets matched - and it's not LAVAs fault.

As we can see in your screenshot, your login prompt isn't just "login:" - it is "do[redacted]t login:". Set your yaml config "login_prompt" to this string, this will make detection of that login prompt less error prone.

Regarding your code point question: Just search the code for "login_prompt" and you will find https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com...https://gitlab.com/lava/lava/-/blob/master/lava_dispatcher/actions/boot/__init__.py#L165 As far as I can see there's just a time limit and no line limit for detecting the login prompt.

Best regards

Stefan

On 10/25/2024 8:40 AM, Sweta Ghosh wrote:

Hello Milosz,

Thanks for replying. I tried the changing the prompts by adding strings from welcome messages and login_prompt but it looks like it is not able to reach that point where my login prompt is displayed. My login prompt(circled) is displayed in logs only after auto-login fails. Please find the screenshot attached. Is there is any limit to number of line it try to match the prompt?

Could you please point me to the code that lava uses to detect the prompt. Thanks, Sweta

---

*From:* Milosz Wasilewski milosz.wasilewski@foundries.io *Sent:* Thursday, October 24, 2024 1:07 PM *To:* Sweta Ghosh sweta.ghosh@nagarro.com *Cc:* lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org *Subject:* Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Sweta,

On Thu, Oct 24, 2024 at 8:12?AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

...

Hello Milosz,

I couldn't understand what you mean by " Did you try to change the prompt to sth else?" . Could you please share an example , how can I do the same.

This part:

...

...

prompts:
- 'login:'

I think this might be your issue. Did you try to change the prompt to sth else?

Try changing the prompt and see if it helps.

Best Regards, Milosz

---

lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

_______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

Hello Milosz,

Thanks for replying.

But when I remove all the prompts I get below error: [cid:9cba866f-2b82-471b-ba15-23d76aa9349c]

Job details: - boot: timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 2:04 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Sweta,

On Tue, Oct 29, 2024 at 8:17 AM Sweta Ghosh <sweta.ghosh@nagarro.commailto:sweta.ghosh@nagarro.com> wrote: Hello,

I have tried to reduce the welcome message to one line i.e System Info: . But even after the it is not able to match the prompt to auto login . Please find the error screenshot attached and job description below:

In the job logs there was no "matched prompt". Also, I tried login_prompt: "device-mgmt login:" but it didn't work.

- boot: timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "do[redacted]t login:" username: root password_prompt: "Password:" password: "root" prompts: - 'System Info: device-mgmt login:' - 'System Info: device-mgmt login:' - 'System Info:device-mgmt login:' - 'System Info:'

This is matched as a prompt. Unfortunately it happens before the login prompt so your job times out. Please remove _all_ prompts and try to run the job with only auto_login section. If this works, you can add a prompt that matches your actual prompt after login. I would also reduce the login_prompt to "login:". This works well for me, example: https://lava.infra.foundries.io/scheduler/job/54807/definition#defline92 https://lava.infra.foundries.io/scheduler/job/54807#L1299

Best Regards, Milosz

- 'device-mgmt login:' - 'root@device-mgmt'

[cid:ii_192d765a804cb971f162]

It looks like LAVA is not able to filter the prompt due to welcome message. Could you please suggest , how can I get logs to see how these messages are visible to LAVA and maybe I can add the exact value it is expecting.

Should I open a bug for the same? Kindly suggest if there is a workaround or if I am missing something in my job description.

Thanks, Sweta ________________________________ From: Sweta Ghosh <sweta.ghosh@nagarro.commailto:sweta.ghosh@nagarro.com> Sent: Friday, October 25, 2024 2:54 PM To: Stefan <lists.lavasoftware.org_23@green-sparklet.demailto:lists.lavasoftware.org_23@green-sparklet.de>; lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org <lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org> Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

Hello Stefan,

Thanks for replying.

* Does the line "[login-action] Waiting for messages, (timeout 00:0x:xx)" appear in logs? I guess yes. I can see error "Job error: auto-login-action timed out after 900 seconds" "auto-login-action: Wait for prompt ['Linux version [0-9]'] (timeout 00:15:00)"[cid:ii_192d765a803cb971f161] But I couldn't find "[login-action] Waiting for messages, (timeout 00:0x:xx)" in my logs. * Does the line "Matched prompt #xx: login:" and "Sending username root" appear? No

Please find the job details of the logs i shared earlier: - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'Variant.....: XXXXX' - 'HOST login:' - 'root@HOST' auto_login: login_prompt: "HOST login:" username: root password_prompt: "Password:" password: "root"

Also tried - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'Variant.....: XXXXX' - 'HOST login:' - 'root@HOST' auto_login: login_prompt: "do[redacted]t login:" username: root password_prompt: "Password:" password: "root"

Getting logs: wait for prompt timed out end: 2.2 auto-login-action (duration 00:12:03) [common] definition: lavahttps://lava.debian.in/results/226/lava/auto-login-action namespace: commonhttps://lava.debian.in/results/226/lava/auto-login-action case: auto-login-actionhttps://lava.debian.in/results/226/lava/auto-login-action level: 2.2https://lava.debian.in/results/226/lava/auto-login-action duration: 722.85https://lava.debian.in/results/226/lava/auto-login-action result: failhttps://lava.debian.in/results/226/lava/auto-login-action extra: ...https://lava.debian.in/results/226/lava/auto-login-action boot-image-retry failed: 1 of 1 attempts. 'wait for prompt timed out' wait for prompt timed out end: 2 boot-image-retry (duration 00:12:03) [common] definition: lavahttps://lava.debian.in/results/226/lava/boot-image-retry namespace: commonhttps://lava.debian.in/results/226/lava/boot-image-retry case: boot-image-retryhttps://lava.debian.in/results/226/lava/boot-image-retry level: 2https://lava.debian.in/results/226/lava/boot-image-retry duration: 722.86https://lava.debian.in/results/226/lava/boot-image-retry result: failhttps://lava.debian.in/results/226/lava/boot-image-retry extra: ...https://lava.debian.in/results/226/lava/boot-image-retry Cleaning after the job Cleaning up download directory: /var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs start: 4.1 power-off (timeout 00:00:10) [common] end: 4.1 power-off (duration 00:00:00) [common] start: 4.2 read-feedback (timeout 00:05:00) [common] Listened to connection for namespace 'common' for up to 1s Finalising connection for namespace 'common' Sending line: 'poweroff' poweroff Already disconnected device-mgmt login: poweroff Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 173, in expect proc = super().expect(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 343, in expect return self.expect_list(compiled_pattern_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 372, in expect_list return exp.expect_loop(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop return self.timeout(e) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout raise exc pexpect.exceptions.TIMEOUT: Timeout exceeded. <lava_dispatcher.shell.ShellCommand object at 0x7fc3ec2a43d0> command: /usr/bin/qemu-system-x86_64 args: [b'/usr/bin/qemu-system-x86_64', b'-cpu', b'host', b'-enable-kvm', b'-nographic', b'-net', b'nic,model=virtio,macaddr=52:54:00:12:34:58', b'-net', b'user', b'-m', b'1024', b'-monitor', b'none', b'-global', b'ICH9-LPC.disable_s3=1', b'-global', b'isa-fdc.driveA=', b'-drive', b'if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd', b'-drive', b'file=/var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs/XXXXX-qemu_amd64.wic,discard=unmap,if=none,id=disk,format=raw', b'-m', b'8G', b'-monitor', b'none', b'-serial', b'mon:stdio', b'-display', b'none', b'-cpu', b'qemu64', b'-smp', b'4', b'-machine', b'q35,accel=kvm:tcg', b'-global', b'ICH9-LPC.noreboot=off', b'-device', b'ide-hd,drive=disk', b'-nographic', b'-chardev', b'socket,id=chrtpm,path=/tmp/swtpm.sock/qemu-swtpm.sock', b'-tpmdev', b'emulator,id=tpm0,chardev=chrtpm', b'-device', b'tpm-tis,tpmdev=tpm0', b'-drive', b'format=qcow2,file=/var/lib/lava/dispatcher/tmp/226/apply-overlay-guest-_kawe9vb/lava-guest.qcow2,media=disk,if=ide,id=lavatest'] buffer (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' before (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 6170 child_fd: 10 closed: False timeout: 900 delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: <lava_dispatcher.shell.ShellLogger object at 0x7fc3f188d190> logfile_send: <lava_dispatcher.shell.ShellLogger object at 0x7fc3ec48bb10> maxread: 2000 ignorecase: False searchwindowsize: 4000 delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 1.0 searcher: searcher_re: 0: re.compile('Linux version [0-9]') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 301, in wait return self.raw_connection.expect(self.prompt_str, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 178, in expect raise TestError("ShellCommand command timed out.") lava_common.exceptions.TestError: ShellCommand command timed out. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 239, in run self._run() File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 229, in _run self.pipeline.run_actions(self.connection, max_end_time) File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/logical.py", line 62, in run connection = self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/actions/boot/__init__.py", line 328, in run res = self.wait(connection) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 943, in wait return connection.wait(max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 303, in wait raise JobError("wait for prompt timed out") lava_common.exceptions.JobError: wait for prompt timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 23, in _wrap_ptyprocess_err yield File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 328, in close self.ptyproc.close(force=force) File "/usr/lib/python3/dist-packages/ptyprocess/ptyprocess.py", line 406, in close raise PtyProcessError('Could not terminate the child.') ptyprocess.util.PtyProcessError: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 308, in run feedback[1].finalise() File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 246, in finalise self.disconnect("closing") File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 253, in disconnect super().disconnect(reason) File "/usr/lib/python3/dist-packages/lava_dispatcher/connection.py", line 153, in disconnect self.raw_connection.close(force=True) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 325, in close with _wrap_ptyprocess_err(): File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 25, in _wrap_ptyprocess_err raise ExceptionPexpect(*e.args) pexpect.exceptions.ExceptionPexpect: Could not terminate the child. end: 4.2 read-feedback (duration 00:00:02) [common] definition: lavahttps://lava.debian.in/results/226/lava/read-feedback namespace: commonhttps://lava.debian.in/results/226/lava/read-feedback case: read-feedbackhttps://lava.debian.in/results/226/lava/read-feedback level: 4.2https://lava.debian.in/results/226/lava/read-feedback duration: 1.70https://lava.debian.in/results/226/lava/read-feedback result: failhttps://lava.debian.in/results/226/lava/read-feedback extra: ...https://lava.debian.in/results/226/lava/read-feedback Failed to run 'finalize': Could not terminate the child. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 173, in expect proc = super().expect(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 343, in expect return self.expect_list(compiled_pattern_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 372, in expect_list return exp.expect_loop(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop return self.timeout(e) ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout raise exc pexpect.exceptions.TIMEOUT: Timeout exceeded. <lava_dispatcher.shell.ShellCommand object at 0x7fc3ec2a43d0> command: /usr/bin/qemu-system-x86_64 args: [b'/usr/bin/qemu-system-x86_64', b'-cpu', b'host', b'-enable-kvm', b'-nographic', b'-net', b'nic,model=virtio,macaddr=52:54:00:12:34:58', b'-net', b'user', b'-m', b'1024', b'-monitor', b'none', b'-global', b'ICH9-LPC.disable_s3=1', b'-global', b'isa-fdc.driveA=', b'-drive', b'if=pflash,format=raw,unit=0,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4M.fd', b'-drive', b'file=/var/lib/lava/dispatcher/tmp/226/deployimages-sfxemxh6/rootfs/XXXX-qemu_amd64.wic,discard=unmap,if=none,id=disk,format=raw', b'-m', b'8G', b'-monitor', b'none', b'-serial', b'mon:stdio', b'-display', b'none', b'-cpu', b'qemu64', b'-smp', b'4', b'-machine', b'q35,accel=kvm:tcg', b'-global', b'ICH9-LPC.noreboot=off', b'-device', b'ide-hd,drive=disk', b'-nographic', b'-chardev', b'socket,id=chrtpm,path=/tmp/swtpm.sock/qemu-swtpm.sock', b'-tpmdev', b'emulator,id=tpm0,chardev=chrtpm', b'-device', b'tpm-tis,tpmdev=tpm0', b'-drive', b'format=qcow2,file=/var/lib/lava/dispatcher/tmp/226/apply-overlay-guest-_kawe9vb/lava-guest.qcow2,media=disk,if=ide,id=lavatest'] buffer (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' before (last 100 chars): 'and type "http://%22%5Cx1b%5B0m%5Cr%5Cr%5Cn%5Cr%5Cr%5CnYou are currently connected via: ttyS0 (115200)\r\r\n\r\r\nHOST login: ' after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 6170 child_fd: 10 closed: False timeout: 900 delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: <lava_dispatcher.shell.ShellLogger object at 0x7fc3f188d190> logfile_send: <lava_dispatcher.shell.ShellLogger object at 0x7fc3ec48bb10> maxread: 2000 ignorecase: False searchwindowsize: 4000 delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 1.0 searcher: searcher_re: 0: re.compile('Linux version [0-9]') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 301, in wait return self.raw_connection.expect(self.prompt_str, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 178, in expect raise TestError("ShellCommand command timed out.") lava_common.exceptions.TestError: ShellCommand command timed out. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 239, in run self._run() File "/usr/lib/python3/dist-packages/lava_dispatcher/job.py", line 229, in _run self.pipeline.run_actions(self.connection, max_end_time) File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/logical.py", line 62, in run connection = self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/actions/boot/__init__.py", line 328, in run res = self.wait(connection) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 943, in wait return connection.wait(max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/shell.py", line 303, in wait raise JobError("wait for prompt timed out") lava_common.exceptions.JobError: wait for prompt timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 23, in _wrap_ptyprocess_err yield File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 328, in close self.ptyproc.close(force=force) File "/usr/lib/python3/dist-packages/ptyprocess/ptyprocess.py", line 406, in close raise PtyProcessError('Could not terminate the child.') ptyprocess.util.PtyProcessError: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 249, in run_actions new_connection = action.run(connection, action_max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 308, in run feedback[1].finalise() File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 246, in finalise self.disconnect("closing") File "/usr/lib/python3/dist-packages/lava_dispatcher/connections/serial.py", line 253, in disconnect super().disconnect(reason) File "/usr/lib/python3/dist-packages/lava_dispatcher/connection.py", line 153, in disconnect self.raw_connection.close(force=True) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 325, in close with _wrap_ptyprocess_err(): File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 25, in _wrap_ptyprocess_err raise ExceptionPexpect(*e.args) pexpect.exceptions.ExceptionPexpect: Could not terminate the child. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/lava_dispatcher/power.py", line 341, in run connection = super().run(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 848, in run return self.pipeline.run_actions(connection, max_end_time) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lava_dispatcher/action.py", line 278, in run_actions raise LAVABug(str(exc)) lava_common.exceptions.LAVABug: Could not terminate the child. Removing root tmp directory at /var/lib/lava/dispatcher/tmp/226 JobError: Your job cannot terminate cleanly. definition: lavahttps://lava.debian.in/results/226/lava/job case: jobhttps://lava.debian.in/results/226/lava/job result: failhttps://lava.debian.in/results/226/lava/job error_msg: wait for prompt timed outhttps://lava.debian.in/results/226/lava/job error_type: Jobhttps://lava.debian.in/results/226/lava/job

Thanks, Sweta ________________________________ From: Stefan <lists.lavasoftware.org_23@green-sparklet.demailto:lists.lavasoftware.org_23@green-sparklet.de> Sent: Friday, October 25, 2024 1:17 PM To: lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org <lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org> Subject: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Hey Sweta,

thanks for the longer log, this sheds some more light on the issue - but still isn't enough to debug it.

First regarding the promt which Milosz already mentioned: You set it to "login:" - that most probably is wrong. Linux hosts usually have a prompt something like "username@hostname:" e.g. "root@example:". Please set yaml config to the promt your device really uses.

Then regarding your logs:

Does the line "[login-action] Waiting for messages, (timeout 00:0x:xx)" appear in logs? I guess yes.

Does the line "Matched prompt #xx: login:" and "Sending username root" appear? At which line? If yes, the login prompt (or something earlier in your banner message) gets matched - and it's not LAVAs fault.

As we can see in your screenshot, your login prompt isn't just "login:" - it is "do[redacted]t login:". Set your yaml config "login_prompt" to this string, this will make detection of that login prompt less error prone.

Regarding your code point question: Just search the code for "login_prompt" and you will find https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com...https://gitlab.com/lava/lava/-/blob/master/lava_dispatcher/actions/boot/__init__.py#L165 As far as I can see there's just a time limit and no line limit for detecting the login prompt.

Best regards

Stefan

On 10/25/2024 8:40 AM, Sweta Ghosh wrote:

Hello Milosz,

Thanks for replying. I tried the changing the prompts by adding strings from welcome messages and login_prompt but it looks like it is not able to reach that point where my login prompt is displayed. My login prompt(circled) is displayed in logs only after auto-login fails. Please find the screenshot attached. Is there is any limit to number of line it try to match the prompt?

Could you please point me to the code that lava uses to detect the prompt. Thanks, Sweta

---

*From:* Milosz Wasilewski <milosz.wasilewski@foundries.iomailto:milosz.wasilewski@foundries.io> *Sent:* Thursday, October 24, 2024 1:07 PM *To:* Sweta Ghosh <sweta.ghosh@nagarro.commailto:sweta.ghosh@nagarro.com> *Cc:* lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org <lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org> *Subject:* Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

Sweta,

On Thu, Oct 24, 2024 at 8:12 AM Sweta Ghosh <sweta.ghosh@nagarro.commailto:sweta.ghosh@nagarro.com> wrote:

...

Hello Milosz,

I couldn't understand what you mean by " Did you try to change the prompt to sth else?" . Could you please share an example , how can I do the same.

This part:

...

...

prompts:
- 'login:'

I think this might be your issue. Did you try to change the prompt to sth else?

Try changing the prompt and see if it helps.

Best Regards, Milosz

---

lava-users mailing list -- lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.orgmailto:lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

_______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.orgmailto:lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s _______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.orgmailto:lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.orgmailto:lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

Hello Milosz, I tried as suggested please find the error message and job details:

- boot: prompts: - 'abcd:' timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

[cid:37525a33-11e1-4af7-839a-312cb7edf222]

I would like to highlight that LAVA job was working fine with my image before welcome message (/etc/issue) was added in the image and it broke the job. As welcome messages are necessary in my case so it would be nice if LAVA could either ignore those welcome messages and auto-login and execute the test like before. Please find the logs of job using image without welcome message

[cid:59723380-c85a-4f42-85ae-b3eafd541199]

- boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'device-mgmt login:' - 'root@device-mgmt' auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 2:51 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 8:43 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

Hello Milosz,

Thanks for replying.

But when I remove all the prompts I get below error:

Job details:

• boot: timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

ok, we're getting somewhere. Now you can add a prompt that doesn't match anything in your job. This should at least allow you to exercise auto_login section.

Best Regards, Milosz

The whole welcome is of 27 lines but for trouble shooting i have reduced it to only 1 line in /etc/issue in test image with which I am currently testing: System Info:

The actual welcome messages will look like this: |_ _ / __ \ / ____| | | | | | | (___ | | | | | |___ \ _| | | |__| |____) | |_____ ____/|_____/

System Info: Version.....: Debian OS Type......: Linux

Interfaces: eth0.......:192.168.1.10

Instruction: 1. welcome to console 2. login with user and password

Alternative : 1. Make sure network interface is up of eth0 network 2. Open a browser and type "http://192.168.1.10"

You are currently connected via:ttyS0 (115200)

device-mgmt login:

Output of /etc/issue

_____ ____ ______ |_ _ / __ \ / ____| | | | | | | (___ | | | | | |___ \ _| | | |__| |____) | |_____ ____/|_____/ System : Version.....: \S{VALUE} Type......: \S{NAME} Interfaces: abc : \e{green}\4{interface}\e{reset} Instruction: 1. welcome to console 2. login with user and password

\e{darkgray}Alternative : 1. Make sure network interface is up of \4{interface}'s network 2. Open a browser and type "http://%5C4%7Binterface%7D%22%5Ce%7Breset%7D

You are currently connected via: \l (\b)

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 3:31 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 9:34 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

Hello Milosz, I tried as suggested please find the error message and job details:

• boot: prompts:
  • 'abcd:'
  timeout: minutes: 6 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

I would like to highlight that LAVA job was working fine with my image before welcome message (/etc/issue) was added in the image and it broke the job. As welcome messages are necessary in my case so it would be nice if LAVA could either ignore those welcome messages and auto-login and execute the test like before. Please find the logs of job using image without welcome message

Could you paste the contents of the welcome message? LAVA is matching some string there and stops the auto_login procedure. I think this is the line just after "System info" but it's not displayed in the screenshots you're posting.

Best Regards, Milosz

I can see auto-login-action:Wait for prompt ['Linux version [0-9]'] (timeout 00:06:00)

[cid:a322069a-386e-4981-895c-e9e39568100f]

* Then you should also see the list of prompts that LAVA is looking for: No I dont see any logs for prompts

Sharing the full logs after execute-qemu in a file attached.

Thanks, Sweta

________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 4:23 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 10:41 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

The whole welcome is of 27 lines but for trouble shooting i have reduced it to only 1 line in /etc/issue in test image with which I am currently testing: System Info:

The actual welcome messages will look like this: |_ _ / __ \ / ____| | | | | | | (___ | | | | | |___ \ _| | | |__| |____) | |_____ ____/|_____/

System Info: Version.....: Debian OS Type......: Linux

Interfaces: eth0.......:192.168.1.10

Instruction:

1. welcome to console
2. login with user and password

Alternative :

1. Make sure network interface is up of eth0 network
2. Open a browser and type "https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.1.1...http://192.168.1.10/"

You are currently connected via:ttyS0 (115200)

device-mgmt login:

Output of /etc/issue

---

|_ _ / __ \ / ____| | | | | | | (___ | | | | | |___ \ _| | | |__| |____) | |_____ ____/|_____/ System : Version.....: \S{VALUE} Type......: \S{NAME} Interfaces: abc : \e{green}\4{interface}\e{reset} Instruction:

1. welcome to console
2. login with user and password

\e{darkgray}Alternative :

1. Make sure network interface is up of \4{interface}'s network
2. Open a browser and type "[http://%5C4%7Binterface%7D%5Dhttp://%5C4%7Binterface%7D%22%5Ce%7Breset%7D

You are currently connected via: \l (\b)

The only suspect there is "login", but I don't think this is an issue. You mentioned that you reduced it to one line just for testing and it still fails. This is interesting. Let's see if the problem doesn't start earlier than that. First "prompt" LAVA is looking for is loading kernel. Example here: https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.infra...https://lava.infra.foundries.io/scheduler/job/54785#L565

auto-login-action: Wait for prompt ['Linux version [0-9]'] (timeout 00:08:00)

Do you have this in your log?

Then you should also see the list of prompts that LAVA is looking for: https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.infra...https://lava.infra.foundries.io/scheduler/job/54785#L713

['-\[ cut here \]', 'Unhandled fault', 'BUG: KCSAN:', 'BUG: KASAN:', 'BUG: KFENCE:', 'Oops(?: -|:)', 'WARNING:', '(kernel BUG at|BUG:)', 'invalid opcode:', 'Kernel panic - not syncing', 'fio@am62xx-evm', 'Password:', 'root@am62xx-evm', 'login:', 'Login incorrect']

Do you have these in your log?

Best Regards, Milosz

Yes, after changing the kernel_start_message to Starting systemd-udevd version \d+(.\d+)*(-\S+)* it progressed. But now even after matching the prompt it is failing. Please find the job description: - boot: prompts: - 'root@device-mgmt' - 'device-mgmt login:' timeout: minutes: 10 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 5:01 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 11:10 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

I can see auto-login-action:Wait for prompt ['Linux version [0-9]'] (timeout 00:06:00)

Then you should also see the list of prompts that LAVA is looking for: No I dont see any logs for prompts

Sharing the full logs after execute-qemu in a file attached.

It looks like the "Linux version" string doesn't show in your log. This is the real issue. I don't know why including the welcome message changes that, but this is what you need to fix. You need this line in your log

https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.infra...https://lava.infra.foundries.io/scheduler/job/54818#L748 [ 0.000000] Linux version 6.1.57 ...

If it's not there, auto_login action will not execute. Your job is waiting for the "Linux version" and doesn't find it. That's why it times out.

The default regex for this piece is: "Linux version [0-9]" You can overwrite this regex by setting "kernel_start_message" in the device dictionary:

{% set kernel_start_message="your regex goes here" %}

Best Regards, Milosz

Hello Milosz,

I updated the job as suggested. But even though it is matching the prompt successfully the "kernel-messages" as passed. It stays in hung state and fails after timeout.

- boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'root@device-mgmt' auto_login: login_prompt: "device-mgmt login:" username: root password_prompt: "Password:" password: "root"

Please find the logs attched.

Thanks, Sweta

________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Tuesday, October 29, 2024 6:55 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 1:02 PM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

Yes, after changing the kernel_start_message to Starting systemd-udevd version \d+(.\d+)*(-\S+)* it progressed. But now even after matching the prompt it is failing. Please find the job description:

• boot: prompts:
  • 'root@device-mgmt'
  • 'device-mgmt login:'
  timeout: minutes: 10 method: qemu media: tmpfs auto_login: login_prompt: "login:" username: root password_prompt: "Password:" password: "root"

The log looks a bit odd. In some places the list of patterns is empty. In your device's rendered dictionary you can look for default-shell-prompt. This might explain some issues.

One thing I might suggest is to remove "device-mgmt login:" from prompts. You can set it as login_prompt. LAVA is matching this string and ignores auto_login section. Anyway, at this point it's an exercise in selecting proper regexes for your job.

Best Regards, Milosz

Hello Milosz,

Thanks for helping me so far. I did the following to rule out the device dictionary issue. I re-enabled the linux version logs in my image and now using full welcome message. Now I am getting kernel parsing messages but my prompt is not getting matched [cid:37b9b0af-3cca-4198-8160-2ed235042130] [cid:1337b08c-99f3-4132-8549-95be50c44391]

Job description: - boot: timeout: minutes: 15 method: qemu media: tmpfs prompts: - 'device-mgmt login:' - 'root@device-mgmt' - 'You are currently connected via: ttyS0 (115200)' auto_login: login_prompt: "device-mgmt login:" username: root password_prompt: "Password:" password: "root"

Attaching the job logs. I still think my welcome messages are doing the trick. Hope you can guide more .

Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Wednesday, October 30, 2024 2:18 AM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: Stefan lists.lavasoftware.org_23@green-sparklet.de; lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Re: Auto login fails after adding banner message before login prompt

[Email from a non-Nagarro source: please exercise caution with links and attachments]

On Tue, Oct 29, 2024 at 3:55 PM Sweta Ghosh sweta.ghosh@nagarro.com wrote:

Hello Milosz,

I updated the job as suggested. But even though it is matching the prompt successfully the "kernel-messages" as passed. It stays in hung state and fails after timeout.

• boot: timeout: minutes: 15 method: qemu media: tmpfs prompts:
  • 'root@device-mgmt'
  auto_login: login_prompt: "device-mgmt login:" username: root password_prompt: "Password:" password: "root"

Please find the logs attched.

I don't think I can help here. There is something wrong in the device dict, a job definition or both. The log shows this after matching the login prompt:

Setting prompt string to ['\r']

This is incorrect. It should have a list of prompts from the boot action. My example below:

https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flava.infra...https://lava.infra.foundries.io/scheduler/job/54821#L1274 Setting prompt string to ['fio@am62xx-evm', 'Password:', 'root@am62xx-evm']

It looks like you have a new line character somewhere and it's causing issues.

Best Regards, Milosz