Hi Salil,
On 31/10/2022 12:40, Salil Mehta wrote:
From: James Morse [mailto:james.morse@arm.com] Sent: Friday, October 28, 2022 5:17 PM To: Salil Mehta salil.mehta@huawei.com; joyce.qi@linaro.org Cc: linaro-open-discussions@op-lists.linaro.org; lorenzo.pieralisi@linaro.org; ilkka@os.amperecomputing.com; Jean-Philippe Brucker jean-philippe.brucker@arm.com; salil.mehta@opnsrc.net Subject: Re: [Linaro-open-discussions] Re: Invitation: Linaro Open Discussions monthly meeting @ Wed 5 Oct 2022 19:00 - 20:00 (HKT) (linaro-open-discussions@op-lists.linaro.org)
On the qemu side: Using KVM and your Qemu tree, PSCI returns success, when it should return denied for CPUs that are forbidden due to firmware policy: [ 0.027597] smp: Bringing up secondary CPUs ... [ 5.108204] CPU1: failed to come online [ 5.109198] CPU1: failed in unknown state : 0x0 [ 5.110384] smp: Brought up 1 node, 1 CPU
Did you try to use below branch(with some suggested fixes) which I shared with you?
[1] James Approach with online-capable and present==possible (some fixes) https://github.com/salil-mehta/linux.git virt-cpuhp-arm64/rfc-v2/jmorse-pres-eq-poss-cpu
No, its the qemu changes I'm after.
Returning "success" to CPU_ON, but not actually bringing the CPU online is an abomination. I can't imagine what the maintainer will throw at you if you suggest it.
This is mostly harmless for linux, but using PSCI_DENIED squashes the error, and avoids the timeout waiting for the secondary to turn up. Mostly harmless as this may cause 'cpus stuck in kernel' to get set, which would prevent kexec.
This probably means you're not using the HVC_TO_USER support for KVM that gets added as part of this series. You can then use the PSCI_TO_USER cap to disable the kernel's PSCI handler, which lets the VMM manage this directly.
Yes, that is true . I intentionally removed the PSCI-to-userspace patches as I was wondering if the CPU devices are not available in the guest kernel for the disabled CPUs how can they be made online?
... and if your guest kernel is malicious?
PSCI has to be able to return an error if you try to online a disabled CPU, (as without enforcement the feature is pointless). So there is no need to track which CPUs are disabled in the MADT - just online the lot, and handle the error gracefully.
Doing this then lines up nicely with the ACPI changes, where if firmware isn't doing any enforcement, then the CPUs get registered and a nasty firmware-bug message is printed. This is what makes those changes safe on x86 where the _STA enabled bit may be wrong on mass-produced laptops. If their 'firmware' isn't enforcing any policy, then the _STA bit gets ignored.
Still using KVM, firmware should clear the enabled bit from _STA for any call after the eject-request. I see this is still 0xF before _EJx is called.
This is how the existing handshake protocol works. Firmware intimates the OS about ejection request which confirms its status about ejection-in-progress. Firmware/VMM then sends the ACPI event to remove the CPU from the kernel(which effectively is try-to-offline process and making not-present), Until this point _STA.ENABLED=1 (which is a correct behavior since firmware/VMM cannot disable the CPU before it has been made offline by the OS - can we snatch the physical resources before completing the housekeeping at OS?). Once offline'ing/removal is done, OS then informs the firmware/VMM about this completion using _EJ0 method (Which is effective way of giving green signal to the firmware/VMM to go ahead and remove the physical resources).
Okay, the scope is narrower than after the eject-request... The problem is apci_processor's remove method calls _STA, and finds "no change", so it does nothing. But it appears this is what Qemu does on x86 too. Once _EJ0 has been called, I'd expect _STA to reflect the conclusion of the eject-request.
Linux needs to see _STA change, as otherwise it can't know which bits in _STA have changed.
Firmware then removes the device and any further evaluation of _STA would show _STA.ENABLED=0 and ideally should show is not present(but this is something which I have not changed in the QEMU - maybe we need to do or does it even matter at QEMU level?).
No, it looks like linux is calling these in the wrong order. The ACPI processor driver needs to have its remove call made once the CPU is offline and _EJ0 has been called,...
(I thought this is what qemu was doing on x86, but now that I test it again...!)
[..]
It looks like a post_eject callback on the scan handler is the right way to do this.
This means the cpu remains registered, and user-space doesn't get notified that the CPU is no longer available. User-space can still try to enable the CPU, which will fail.
That is true and to fix this we could use the online-capable Bit instead of using the _STA.ENABLED Bit status as discussed in the 5th Oct LOD meeting. I have done those changes in the above branch [1].
But the MADT online-capable bit is static, it only ever has one value. We need something that can be changed by firmware, (and the OS _knows_ is being changed by firmware). This is what _STA is for.
You can't guess from an eject-request that the CPU is being disabled, what happens when we need to add support for physical hot-remove?
Thanks,
James IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.