darp6: keyboard insensitivity at boot leads to difficulty unlocking encrypted hard drive.

[bryanpaget]

I've recently noticed that when I try to unlock my encrypted hard drive at boot the keyboard is reluctant to listen to me! I THINK this started after the most recent darp6 firmware update but I could be mistaken. At the very least it started around the same time.

I have to press each key multiple times before the * appears in the password field. My workaround for this is to use my external keyboard (which works perfectly) but if I were out travelling then it would be very difficult to unlock my hard drive. 😱

Much love. ❤️

[bryanpaget]

The issue is resolved if I reboot and try again but it sometimes takes a few reboots to get the keyboard to work properly.

[leviport]

I'm also seeing this occasionally happen on oryp6. Would you happen to be running open EC on your darp6?

[bryanpaget]

I have applied all firmware patches so I think I am running open EC. Is there a way for me to confirm?

[leviport]

Sorry to leave you hanging, I got sidetracked on something else yesterday. You can check the ec version with the ectool script:

git clone https://github.com/system76/ec
cd ec
./ectool.sh info

[bryanpaget]

✔ ~/ec [master|✔] 
14:54 $ ./ectool.sh info
    Finished release [optimized] target(s) in 0.00s
board: system76/darp6
version: 2021-01-19_d6d37c0

[ids1024]

I've noticed this as well.

Perhaps evtest could help to show what is different between the internal and external keyboard here. (Assuming it's in some way related to input events.)

[jackpot51]

For any of you seeing the issue, does pressing up, which will show the prompt using a terminal instead of the graphical one, make the keyboard work without issues?

[ids1024]

I tried rebooting a couple times, but I didn't see this. I don't think it's a consistent problem... Wonder what makes it happen.

Haven't tried unlocking in a terminal, but it always works fine with an external keyboard (as stated previously). And it's possible with the built-in keyboard when the issue occurs, just tediously slow.

[leviport]

For any of you seeing the issue, does pressing up, which will show the prompt using a terminal instead of the graphical one, make the keyboard work without issues?

It does not, however the lagginess seems to go away after 10 or 15 seconds. This happens pretty regularly on oryp7, so I can definitely help test.

[bryanpaget]

For any of you seeing the issue, does pressing up, which will show the prompt using a terminal instead of the graphical one, make the keyboard work without issues?

I can't test this. I rolled back to 20.04 recently and when I did I chose not to encrypt the hard drive (due to this bug).

I can say that the "slow" keyboard problem persists after a fresh OS install at the GDM login screen. As @leviport mentioned it does go away after ~15 seconds.

[jhorsm]

Hi. I just got done working through a s76 support ticket on this issue (59582) on a darp6 where I uploaded logs from the OPEN-EC Firmware and the Proprietary firmware. And I also included a video on how to recreate the behavior. I was redirected to this open issue.. I do not know if the log files there can be reviewed. Or should I upload them here/needed?

To recreate the issue: Cold boot of the device with nothing plugged in to the laptop with the open-ec firmware. The pauses show in graphical side and also in the console (Hitting left arrow). The problem does not show on warm reboots.

dmesg of the kernel shows me the following error code (All the logs are on the support ticket). This error is not in the 052020 firmware. Might be related.

3.801891] ACPI Error: AE_TIME, Returned by Handler for [EmbeddedControl] (20201113/evregion-294)
[ 3.801895] ACPI Error: Timeout from EC hardware or EC device driver (20201113/evregion-304)

[ 3.801907] No Local Variables are initialized for Method [SKBL]

[ 3.801909] Initialized Arguments for Method [SKBL]: (1 arguments defined for method invocation)
[ 3.801909] Arg0: 00000000dda305af Integer 0000000000000000

[ 3.801916] ACPI Error: Aborting method _SB.S76D.SKBL due to previous error (AE_TIME) (20201113/psparse-529)
[ 3.801922] ACPI Error: Aborting method _SB.S76D.RSET due to previous error (AE_TIME) (20201113/psparse-529)
[ 3.801926] ACPI Error: Aborting method _SB.S76D.INIT due to previous error (AE_TIME) (20201113/psparse-529)
[ 3.801937] System76 ACPI Driver: probe of 17761776:00 failed with error -1
[ 4.476980] input: PNP0C50:00 06CB:CD65 Mouse as /devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-1/i2c-PNP0C50:00/0018:06CB:CD65.0001/input/input8
[ 4.477038] input: PNP0C50:00 06CB:CD65 Touchpad as /devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-1/i2c-PNP0C50:00/0018:06CB:CD65.0001/input/input9
[ 4.477079] hid-generic 0018:06CB:CD65.0001: input,hidraw0: I2C HID v1.00 Mouse [PNP0C50:00 06CB:CD65] on i2c-PNP0C50:00
[ 4.524223] system76: Model System76 darp6 found
[ 4.524226] system76: No known WMI event notification GUID found

Let me know if there is anything else needed..

[crawfxrd]

Does anyone else have the ACPI timeout when this occurs?

[jhorsm]

@crawfxrd Just an update on some more troubleshooting I did on this issue. I was in fedora 34 with no system76 drivers with LUKS Enabled. It has the same problem. When I pulled the dmesg. (No sys76 Driver loaded.) It gave me the following error. At 34 seconds I was able to regain use of the keyboard.

[ 3.227021] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[ 5.923672] psmouse serio1: synaptics: Unable to query device: -5
[ 16.963823] input: PS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input6
[ 17.491683] psmouse serio1: Failed to enable mouse on isa0060/serio1
[ 34.560025] BTRFS: device label fedora_localhost-live devid 1 transid 3273 /dev/dm-0 scanned by systemd-udevd (1031)

So I did an experiment with this error I saw on the touchpad.

I disconnected the touchpad from the darp6 mainboard.

Powered off the device completely and booted it to recreate the problem. The Problem went away. Did this four times or so.

Plugged the touchpad back into the mainboard. The problem returned.

I am in progress of putting POP back on it to see what its kernel tells me. Will follow up shortly.

Edit: Same behavior shows up in POP. Only difference is the timeout messages above.

[jackpot51]

I wonder why it is attempting to use ps/2 when the system supports I2C HID. Can you check for I2C HID messages in dmesg? Something like dmesg | grep -i hid

[jhorsm]

As requested.
@pop-os:~$ sudo dmesg | grep -i hid
[ 1.266198] hid: raw HID events driver (C) Jiri Kosina
[ 1.310330] i2c_hid i2c-PNP0C50:00: supply vdd not found, using dummy regulator
[ 1.310355] i2c_hid i2c-PNP0C50:00: supply vddl not found, using dummy regulator
[ 4.493141] hid-generic 0018:06CB:CD65.0001: input,hidraw0: I2C HID v1.00 Mouse [PNP0C50:00 06CB:CD65] on i2c-PNP0C50:00
[ 72.764719] input: Intel HID events as /devices/platform/INT33D5:00/input/input11
[ 73.042819] hid-multitouch 0018:06CB:CD65.0001: input,hidraw0: I2C HID v1.00 Mouse [PNP0C50:00 06CB:CD65] on i2c-PNP0C50:00

Also I need to apologize. But the logging in POP is showing the error differently. I thought it was the same at first glance. From same log. I am also including the dmesg from this boot.

[ 1.506127] psmouse serio1: Failed to reset mouse on isa0060/serio1: -5
[ 40.134230] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5
[ 41.194232] psmouse serio1: Failed to enable mouse on isa0060/serio1
[ 50.703197] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/input/input6
[ 51.230225] psmouse serio1: Failed to enable mouse on isa0060/serio1

dmesg-openec.txt

[ggobbe]

I've had the same problem since I received my Oryx Pro 6 end of October 2020.
I have it with both PopOS 20.10 and 20.04 and it remains after reinstalling the OS multiple times.
I also had the issue with Fedora 33 when I tried it end of last year.

Here is my dmesg output
dmesg.oryp6.txt

❯ ./scripts/ectool.sh info
    Finished release [optimized] target(s) in 0.01s
board: system76/oryp6
version: 2021-03-16_7c3e89e

[crawfxrd]

I was able to reproduce this on gaze15 with i8042.debug=1 and i2c_hid.debug=1 set:

• Bad boot
• Good boot

i8042 is getting a response from the touchpad for the mouse ID request after setting up the keyboard.

[    0.758391] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
[    0.759461] i8042: [11] d4 -> i8042 (command)
[    0.760592] i8042: [11] f2 -> i8042 (parameter)
[    0.764534] i8042: [12] fa <- i8042 (interrupt, 1, 12)
[    0.765624] i8042: [12] 00 <- i8042 (interrupt, 1, 12)

i8042 then sends a ton of requests to the second port (command 0xD4) until about 22 seconds into the boot before giving up. This happens while the decrypt screen is up and waiting for input.

---

So, presumably, a workaround is to boot with i8042.noaux=1 so that i8042 doesn't attempt to configure the touchpad.

[jhorsm]

Able to get similar output with the same debug flags. Also used the workaround kernel flag i8042.noaux=1 and I did about 10 or so cold boots and tried my best to recreate the issue like I normally would and the issue did not come back. Same time ACPI timeouts are not showing either. So for me the work around works. Will report back if I encounter it not working right with logs.

Here are the dmesgs of both logs with the debug flags enabled. One with the issue. The other with the noaux flag enabled.

This one is the issue.
dmesg-debug-enabled.txt

This one is with the workaround enabled.

i8042.noaux.dmesg.txt

Thank you for the workaround.

[ggobbe]

Where should the i8042.noaux=1 flag be added? As I don't have grub is it in the efi configuration file like this ⬇️ ?

/boot/efi/loader/entries/Pop_OS-current.conf

title Pop!_OS
linux /EFI/Pop_OS-c75c018b-0c0e-40d0-824c-2a08a038b6b5/vmlinuz.efi
initrd /EFI/Pop_OS-c75c018b-0c0e-40d0-824c-2a08a038b6b5/initrd.img
options root=UUID=c75c018b-0c0e-40d0-824c-2a08a038b6b5 ro quiet loglevel=0 systemd.show_status=false splash i8042.noaux=1

[jacobgkau]

@ggobbe If you're on Pop!_OS, you can run sudo kernelstub -a "i8042.noaux=1" and it should be added to both the file that you referenced and the cmdline file in /boot/efi/EFI.

We'll probably end up adding this to the system76-driver package so it's applied automatically if it won't be addressed in the firmware.

[ggobbe]

I ran sudo kernelstub -a "i8042.noaux=1" and it works perfectly on my oryp6 with PopOS 20.04.
No more input lag on encryption password prompt when booting 🎉

Thanks @jacobgkau and everyone!

[crawfxrd]

It seems that after receiving the spurious interrupts, serio attempts to detect every touchpad over PS/2, with all of them failing because the touchpad stops responding.

[jackpot51]

We should really find a way to fix this in firmware.

[jackpot51]

I will look into it on the darp6 I have, since it is pretty heavily affected

[crawfxrd]

Resolved by: system76/ec#201

[crawfxrd]

Fix has been released in 2021-07-20_93c2809.

[jhorsm]

Hey there. I appreciate the release of the fix. When is this going to be released to darp6 devices? I am still seeing a release of 2021-01-19 on my darp6. Thanks.

[crawfxrd]

@jhorsm It is released. You are not seeing it due to pop-os/system76-firmware#66.