HID Finger Swipe Functionality (iPhone)

[matt_wilson]

I’m very familiar with the HID keyboard and mouse profile and have built products using it but need direction on how to replicate the functionality of a up/down/left/right finger swipe on a iPhone. Any guidance will be valuable.  Thanks 

[matt_wilson]

What I'm trying to accomplish is exactly what this remote is doing but not sure what type of Bluetooth HID profile it's working with. What it's doing is performing up/down/left/right scroll as if it was a finger swipe.  I think it is acting like a digitizer but it shows up as a keyboard profile and not a mouse.  There is another thread that talks about HID digitizer and has an example project to go with it but its base is on top of mouse. So I'm little confused on what HID profile to focus on.  I do have the packet logs captured from this remote.  Is there a way to capture the report profile from a device?

www.aliexpress.com/.../4000061114207.html

[ovrebekk]

Hi Matthew

If you have access to the device in question you should be able to connect to it from the nRF Connect mobile app, and read out the HID report descriptor from there. 

Then you can try to replicate that HID descriptor in your own peripheral implementation. 

Best regards
Torbjørn

[ovrebekk]

Hi

I agree with Jefferson's comment. When the NRF_ERROR_RESOURCES error occurs you should wait for the next BLE_GATTS_EVT_HVN_TX_COMPLETE event, and then you can continue uploading packets until the error occurs again (and then rinse and repeat). 

In order to try sending more packets pr connection event you can try increasing the NRF_SDH_BLE_GAP_EVENT_LENGTH define in sdk_config.h, but as the phone also has it's own limit there is no guarantee that increasing the GAP event length will increase the total number of packets. 

Best regards
Torbjørn

[matt_wilson]

Torbjørn & Jefferson,

I was able to successfully get my test code to work.  It's not pretty but the concept has been proven.  I was able to reverse engineer the remote and combine it with the keyboard report.  The solution was to queue up the packets and send them sequentially after each success TX response.  If I had time I would use the nrf_queue or nrf_ringbuffer library for this solution.

The final functionality is when you press one of the 4 buttons it will send a stylus/pen swipe sequence.  You can see this in action if you are in a drawing app.

I want to thank you Torbjørn for taking the time to answer my questions that lead me to the answers I was seeking.  I learned a great deal from this exercise.  I have attached the final code for others to reference if desired.

All that is left is to break down the packet structure and build a data structure for.  If I get that working I will post the final structure.  All I have to go on is what I wrote in a few post back.

0167.main.c

[Noon]

Hi guys,

I was facing the same issue than matt_wilson: emulate swipe movement from a custom HID Device.

From this HID Report (https://devzone.nordicsemi.com/f/nordic-q-a/38587/is-multi-touch-hid-possible-with-nrf51822-chip), i'm able to "customise" my swipe movements. It's a quick fix but it actually works. 

I start a timer when buttons are pressed, and send the report as this:

static void swipe_timeout_handler(void * p_context)
{

  switch (swipe_type)
    {
        case LEFT:
            y = 15000;
            x = (30000) - cpt*2500;
            cpt++;
            digitizer_send3(0, 1, x, y, true);
            if(cpt == 5) {
                digitizer_send3(0, 1, x, y, false);
                timers_stop();
                cpt = 0;
            }
            break;

        case RIGHT:
            y = 15000;
            x = cpt*2500;
            cpt++;
            digitizer_send3(0, 1, x, y, true);
            if(cpt == 5) {
                digitizer_send3(0, 1, x, y, false);
                timers_stop();
                cpt = 0;
            }
            break;

        case UP:
            y = (30000) - cpt*2500;
            x = 0;
            cpt++;
            digitizer_send3(0, 1, x, y, true);
            if(cpt == 5) {
                digitizer_send3(0, 1, x, y, false);
                timers_stop();
                cpt = 0;
            }
            break;

        case DOWN:
            y = cpt*2500;
            x = 0;
            cpt++;
            digitizer_send3(0, 1, x, y, true);
            if(cpt == 10) {
                digitizer_send3(0, 1, x, y, false);
                timers_stop();
                cpt = 0;
            }
            break;

        default:
            // No implementation needed.
            break;
    }
   
}

By doing this way, it is possible to emulate differents kinds of swipe by changing timer, cpt, x, y.

While writing, i've see the matt solution with queuing. Will try this later...

[ovrebekk]

Hi 

Good to hear you found a working solution Matthew, and thanks for sharing the code

Same to the others chiming in as well, clearly this is something many are trying to do.

Once you share your final code you can mark you reply as the answer, making it easy for other to find also.  

Best regards
Torbjørn

[matt_wilson]

Torbjorn,

Thanks for all your help.  I have been in the process of migrating my work to my project code base.  I have been using the Adafruit Bluefruit52 library because it has simplified my development time.  I would rather do it in Segger Studio but right now I prefer to use this 3rd party library because it handles most of the basic/core functionality of an application.  Of course I ran into issues and if you have time please review this other ticket.  I'm so close to getting this working but reaching out for some final help because I hit a hard wall...

https://devzone.nordicsemi.com/f/nordic-q-a/58116/hid-digitizer-stylus-integration---adafruit-bluefruit52

[ovrebekk]

Hi Matthew

The case was given to me already, I will take a look next week. 

I have to warn you that we can't really support software libraries not released by us. Bluefruit52 is an Arduino API I assume and would have to be supported by them, but I will try to help if it's not too time consuming. 

Best regards
Torbjørn

[matt_wilson]

Trobjorn,

I understand and thanks for the feedback.  Besides, I just figured it out.  I solved all the hard problems but left a small hidden setting unchecked.

Does Nordic recommend any other 3rd party libraries like Adafruit that are not Arduino based?

[ovrebekk]

Hi Matthew

Good to hear you figured out the issue

Traditionally one of the largest third party software platforms was the mBed platfrom from ARM:
https://www.mbed.com/en/

Their API's are very user friendly and "Arduino like", but I am not sure how well ARM is supporting it anymore. They also use an online IDE/compiler solution which is great for hobby or prototyping projects, but not ideal for more complex projects. 

There are also some fun projects such as MicroPython or CircuitPython, allowing you to run Python code on certain Nordic devices, but these take you even more into the hobby/prototyping space. 

In the end I would discourage using third party software platforms if you are intending to go to mass production with a product based on a Nordic device. Then you want the software support that we can provide, and you want to leverage code that has been thoroughly tested both by us and by countless Nordic customers. 

If your current project is more of a proof of concept or evaluation project then I see no problem using other software platforms. 

Best regards
Torbjørn

[matt_wilson]

Torbjørn,

So I did all this working and it seems that with iOS 13.4 made some changes that caused this report descriptor to stop working.  I had a product close to going into production in a week and I have no clue what changes were made in iOS.  I'm using the Digitizer-Pen-Stylus report descriptor and think I have to move to something else.  Do you have any recommendations on what I should do?  Do you know of any working examples of the Digitizer that I could pull from to test.

[matt_wilson]

iOS is now ignoring my input.

Mar 20 16:47:55 iPhone backboardd[64] <Error>: ignoring digitizer event for main screen, not from built-in digitizer (<BKIOHIDService: 0x109dc7430; serviceStatus: matched; IOHIDService: IOHIDService name:IOHIDEventDriver id:0x100000dce primaryUsagePage:0x1 primaryUsage:0x6 transport:BluetoothLowEnergy reportInterval:8000 batchInterval:0 events:87 mask:0x800; senderID: 0x100000DCE; displayUUID: 0x0; eventSource: externalKeyboard; primaryUsagePage: 0x1; primaryUsage: 0x6; authenticated: NO; builtIn: NO>):+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Timestamp: 73357602092 Total Latency: 877 us SenderID: 0x0000000100000DCE BuiltIn: 0 ValueType: Absolute EventType: Digitizer Flags: 0x90003 DisplayIntegrated: 1 TransducerType: Stylus TransducerIndex: 0 Identity: 0 EventMask: 2 Events: Touch ButtonMask: 0 Range: 1 Touch: 0 Pressure: 0.000000 AuxiliaryPressure: 0.000000 Twist: 0.000000 GenerationCount: 0 Will