Problem with some keys using nRF Desktop with iPad/Apple products

[Glen M]

Hi,

My customer tried using the keyboard we made based on nRF Desktop with NCS version 2.1.0 with an iPad and noticed that most of the keys work, but space bar does not, and caps lock does not.

I suspected something was up with the HID descriptor so I've been digging into that all day, and I've been troubleshooting it using an nRF52840-DK with the excellent Nordic-provided Bluetooth sniffer code on it with Wireshark.

I found that the keyboard is advertising that its report map is at characteristic handle 3a, but when the iPad tries to read the report map from that characteristic it fails. So the iPad not getting the HID report map might explain the strange behavior I'm seeing.

I turned on RTT debugging and I see the error happening there, too: "bt_l2cap: Ignoring data for unknown channel ID 0x003a"

I tried searching for that error text and I found a DevZone post about it where Nordic Support declared that it was no big deal and could be ignored, but it seems too much of a coincidence that this could explain the issues I'm having.  That support ticket is here:

 Bluetooth: Ignoring data for unknown channel ID 0x003a 

That characteristic is under the BT_UUID_HIDS_REPORT_MAP_VAL characteristic (0x2a4b) I found in \nRF_Connect\v2.1.0\zephyr\include\zephyr\bluetooth\uuid.h

but so far I haven't found the place in the Bluetooth code that associates the channel ID of 3a with the report map.

Help! Has this perhaps been solved in a later version of NCS? I've avoided upgrading because a lot of stuff has changed in NCS since I started this project and the last time I tried upgrading the error list I got make it seem like it was going to be a lot of work to upgrade.

I've attached a couple of screenshots from my Wireshark capture showing where the device is advertising its report map on 3A and then throwing an error when the host tries to read it.

Thanks,

Glen

[Glen M]

I just tried it using my second nRF52840-DK with the latest NCS 2.61 and a new instance of nRF Desktop. The ID has changed slightly, but the error still exists.

[Sigurd]

Hi!

Glen M said:

I just tried it using my second nRF52840-DK with the latest NCS 2.61 and a new instance of nRF Desktop.

This was with out-of-the-box nrf_desktop application?

ith an iPad and noticed that most of the keys work, but space bar does not, and caps lock does not.

Is it only on the iPad /iOS this is an issue? Does it work as expected on Windows and Android ?

BR,

Sigurd

[Glen M]

Any thoughts on this from Nordic? This issue is urgent.

[Sigurd]

Hi Glen!

The "bt_l2cap: unknown channel ID 0x003a" is a red herring, it's standard that iOS devices sends this.

What iPadOS version and iPad generation are you testing with?

Note that connecting a Bluetooth accessory to your iPad requires iPadOS 13.4 or later. 

I will try to get an iPad today, and see if I can recreate the issue here.

[Glen M]

Thanks Sigurd - I'm using a 10th gen iPad with iPadOS version 16.3.1

[Glen M]

I've been digging back through every change I made to the original nRF Desktop app and I think this change I made to the mouse HID report might be a part of the problem -I'm working to confirm this. I needed to include a horizontal scroll in addition to vertical scroll, and this was my understanding of how to put that information in the report, but maybe I put it in the wrong place or did it in a nonstandard way?

EDIT: This does not seem to be the problem.

[Glen M]

I've been trying to methodically work from an unmodified keyboard build on my nRF52840 devkit to one that I managed to break by putting in things from my real project's prj.conf. 

I will attach my modified files (otherwise it's a normal nRF Desktop build for the nRF52840-DK using SDK 2.1.0 using the keyboard configuration)

I changed the LEDs and buttons on the devkit so I can see if the caps lock is working. I use button 4 to kick off the printing of "nordic" over and over and then use button 2 to toggle caps lock, and one of the LEDs indicates the caps lock state. I've also noticed that when things are working, when I connect to the iPad it turns the border of the iPad blue briefly, and when I go to the Notes application to test typing, if it's going to work then the border of that app turns yellow. When it's broken, that screen highlighting doesn't work and the caps lock doesn't work. It just always prints nordic in lower case.

nrf52840dk_nrf52840.zip

[Glen M]

Any updates from Nordic? I've seen "an engineer has been assigned" a couple times on this but no updates.

I'd like to be able to give my customer a status report. It's OK if it's not solved yet, but I'd like to at least hear "we're working on it, here's what we're thinking", or if I'm on my own on this I'll get to work trying to solve it - but I don't want to duplicate effort if you are getting somewhere with it. 

Thanks, Glen

[Sigurd]

Hi!

Glen M said:

Any updates from Nordic? I've seen "an engineer has been assigned" a couple times on this but no updates.

Sorry for the delay. Many public holidays in Norway in the month of May.

I was able to get hold of a iPad, and have started testing this.

Using nRF Desktop build for the nRF52840-DK using SDK 2.1.0,using the keyboard configuration, with your modifications applied.

Here is the hex file:

5125.merged.zip

I have tested it a few times, the behavior I see, is that button 2 does nothing. button 4 starts the "nordic" printing, and for caps lock, I need to hold button 3 down for it to be uppercase letters. If button 3 is not pressed, it's lowercase letters. Is that what you are seeing as well?

[Glen M]

Hi Sigurd - thanks for getting back to me.

I just tested my board with both my Windows 10 PC and with an iPad.

Here's the behavior with the Windows PC:

• Button 2 and LED 3 are for caps lock. This works on the PC.
• Button 1 prints 'a'
• Button 3 is left shift, so it makes it do caps as long as you're pressing it. I did this to confirm that caps are possible on the iPad.
• As usual, hold down button 1 while powering up to turn on advertising
• Button 4 starts and stops the "n o r d i c" printing. I have it throwing in a caps lock key before each word to see if it would toggle caps, but that doesn't work on iPad OR on my PC, so maybe I did something wrong there.

On the iPad I notice the following things:

• Button 1 and 3 work as above
• Button 2 seems to do nothing and LED3 never comes on. If I could figure out how to turn on caps lock on the iPad I'd like to see if the report that turns on LED3 is working - it may well be since it's clearly not getting the caps lock key command
• On the PC it prints the spaces I included between the "nordic" letters, on the iPad there are no spaces unless I'm holding down button 3, forcing the letters to be capital. Then it includes the spaces for some reason. Maybe I'm sending the wrong keycode for the space bar?

Anyway, the caps lock not working is the thing the customer noticed and it's the most obvious thing that's broken so that's what I've been paying attention to most.

My boss suggested that if a combined keyboard+mouse HID report isn't working that we could try sending two separate reports, but either way I think that requires getting deep into the HID report code, which is intimidating.

Thanks,

Glen

[Sigurd]

Hi!

I see the same behavior as you describe. On Windows and Android it works, but not on iPad or iPhone. So seems like something with the keyboard AND the mouse reports confuses the iOS device.

I have been looking into what's happening on-air

On Android when I press and release button 2(Caps-lock button):

we see that the android responds back with ATT Write Command Packet,  basically confirming back to the nRF52 that caps-lock is on.

On iPad, we don't get this ATT Write Command Packet back.

But interestingly, I'm seeing 2 packets per key press/release. 2 Packet for button-press, and then 2 for button-release. So that's seem like a bug, not sure yet if it's relevant for the issue or not. Will continue debugging this.

[Glen M]

Hi Sigurd,

I've been experimenting more with this. I found that CONFIG_DESKTOP_HID_REPORT_KEYBOARD_SUPPORT is not included in the project by default, so I added it. Now, the keyboard stuff is working - but the mouse stuff is broken on both my Windows PC and the iPad.

Does that mean that including both things in one report is just broken and perhaps the problem with the iPad just stems from the keyboard stuff not being included in the descriptor prior to me adding in that define?

EDITED: Ok I found that my mouse wasn't working because I had turned off the enables for that in my main project while testing previously. Turning the mouse stuff back on gets be right back where I started. So I guess I'm back to the hypothesis that the combined report just doesn't work for some reason.