Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-touch touchscreen support (#123) #297

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions examples/MultiTouch/MultiTouch.ino
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*
Copyright (c) 2021 ilufang
See the readme for credit to other people.

Multi-touch touchscreen example
Draw 7 parallel lines across the screen.
Open Microsoft Whiteboard, select a brush and press the button.

See HID Project documentation for more infos
https://github.com/NicoHood/HID/wiki/Gamepad-API
NicoHood marked this conversation as resolved.
Show resolved Hide resolved
*/

#include <HID-Project.h>

const int pinButton = 2;

void setup() {
pinMode(pinButton, INPUT_PULLUP);
MultiTouch.begin();
}

void loop() {
while(digitalRead(pinButton));

int16_t x = 2000, y = 2000;

for (; x <= 8000; x+=10) {
for (int i = 0; i < 7; i++) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said the library supports 10 fingers. Why are you only demonstrating 8 here? Also a few comments about the fact that 10 fingers are supported would be nice.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Hybrid Mode" issue below.

2 fingers will fully exercise the report. Anything more than 2 exercises the Hybrid Mode reporting. I went over an issue with reporting not-multiple-of-2 fingers when unused fingers are not zeroed, so I decided to report 7 here. This will have fully populated reports, multiple reports for hybrid mode and partially populated reports.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And why do you define 10 fingers then? I think I dont understand that correct.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current state, 2 is the number of fingers in the report, defined by the descriptor; 10 is the theoretical maximum number of fingers, defined by the feature report:

  • Each report has 2 fingers
  • If more than 2 fingers is present, we send more reports. Hybrid mode defines how to do it.
  • We can send as many reports as desired, but Windows won't process more than 10 (software restriction I guess)
  • Windows will say 10-finger support in System Properties
  • We don't have a 10-finger report because it will be unnecessarily large for the common case where only 2 fingers are expected. Maybe there are also size limits so we can't pack like 100 touches into a single report, but still able to report them through 20 reports with 5 touches in each.

I'm setting 2/10 because I think that represents the most common scenarios. In fact I'm trying to do something that sends ~8 touches regularly up to max ~36 concurrent touches. I was expecting the user to configure these parameters.

MultiTouch.setFinger(i, x, y+i*1000, (i+1)*15);
}
MultiTouch.send();
delay(10);
}

for (int i = 0; i < 7; i++) {
MultiTouch.releaseFinger(i);
}
MultiTouch.send();
}
118 changes: 118 additions & 0 deletions src/HID-APIs/MultiTouchAPI.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
/*
Copyright (c) 2021 ilufang
See the readme for credit to other people.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

// Include guard
#pragma once

/// Maximum amount of fingers supported
#define HID_MULTITOUCH_MAXFINGERS 10
/// Number of fingers in a single report
#define HID_MULTITOUCH_REPORTFINGERS 2

// A report will always be the same size, even if you report fewer fingers than
// REPORTFINGERS. The unused finger entries will simply be zero. If more fingers
// are present than REPORTFINGERS, multiple reports will be sent to report all
// fingers. This is know as "Hybrid Mode" on MSDN. The number of supported
// fingers identified by Windows will still be MAXFINGERS. More than MAXFINGERS
// contacts may be ignored by Windows even with hybrid mode.

#define _MT_STATE_INACTIVE 0
#define _MT_STATE_CONTACT 1
#define _MT_STATE_RELEASED 2
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about those? What is the difference between those states and are they used by the windows client? If yes, why dont we expose these?

Nevertheless, please put them in an enum.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No they are just for internal tracking. 0 and 1 just says if an entry is in use or not. 2 helps to identify a "release" action because windows expects that a released touch is reported once with tip-switch set off off after releasing, instead of just being silently excluded from the report.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If my suggest API implementation will work, I think this tracking is no longer required.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are moved to hpp as an enum, but I think the current internal tracking cannot be further simplified. See below.


typedef struct ATTRIBUTE_PACKED {
NicoHood marked this conversation as resolved.
Show resolved Hide resolved
uint8_t reportID;
uint8_t count;
struct ATTRIBUTE_PACKED {
uint8_t identifier;
uint8_t touch;
uint8_t pressure;
uint8_t x1, x0;
NicoHood marked this conversation as resolved.
Show resolved Hide resolved
uint8_t y1, y0;
} contacts[HID_MULTITOUCH_REPORTFINGERS];
} HID_MultiTouchReport_Data_t;

typedef struct {
uint8_t status;
int8_t pressure;
int16_t x, y;
NicoHood marked this conversation as resolved.
Show resolved Hide resolved
} _finger_t;

class MultiTouchAPI
{
public:
MultiTouchAPI() {
_fingers_count = 0;
for (int i = 0; i < HID_MULTITOUCH_MAXFINGERS; i++) {
_fingers[i].status = _MT_STATE_INACTIVE;
}
}
NicoHood marked this conversation as resolved.
Show resolved Hide resolved

inline void begin();

/**
* Set contact status for a finger in the internal data structure. You must
* call send manually after setting all fingers to flush them through USB.
*
* @param id Finger id. Must be in the range of 0-MAXFINGERS. Same finger
* must have same id throughout contact. Allocations does not need
* to be continuous.
* @param x, y Coordinates. Range 0-10000. (0,0) is top-left on Windows.
* @param pressure Contact pressure. Range 0-127. When set to 0, the touch
* is reported as hovering (in-range)
* @return 1 if success. 0 if id is out-of-bounds
*/
inline int setFinger(uint8_t id, int16_t x, int16_t y, int8_t pressure=100);

/**
* Release finger in the internal data structure. You must call send
* manually after setting all fingers to flush them through USB.
*
* @param id Finger id. Must be in the range of 0-MAXFINGERS. Same finger
* must have same id throughout contact. Allocations does not need
* to be continuous.
* @return 1 if success. 0 if id is out-of-bounds
*/
inline int releaseFinger(uint8_t id);

/**
* Generates an HID report reflecting the currently recorded touch status
* and send through USB.
*/
inline int send();
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if those API names can't be improved. Checkout the other APIs:
https://github.com/NicoHood/HID/blob/master/src/HID-APIs/KeyboardAPI.h#L48-L56
https://github.com/NicoHood/HID/blob/master/src/HID-APIs/AbsoluteMouseAPI.h#L69-L75

What about having:

// Move and press
press(uint8_t id, int16_t x, int16_t y, int8_t pressure=100);
press(uint8_t id, int8_t pressure=100);

// Move, press and release
tap(uint8_t id, int16_t x, int16_t y, int8_t pressure=100);
tap(uint8_t id, int8_t pressure=100);

// Release (after press)
release(uint8_t id);
releaseAll();
      
// Sending is public in the base class for advanced users.
virtual void SendReport(void* data, int length) = 0;

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not yet very about having logic to perform sequences of operations to implement gestures. I'm sticking with the simplest operations for now. I'll come back to this when I find a chance.


protected:
/// Send the generated report. Needs to be implemented in a lower level
virtual int _sendReport() = 0;

/// Internal records of the current touch status
_finger_t _fingers[HID_MULTITOUCH_MAXFINGERS];
/// Number of active contacts, including just release contacts
uint8_t _fingers_count;

/// HID report to send. Not
NicoHood marked this conversation as resolved.
Show resolved Hide resolved
HID_MultiTouchReport_Data_t _report;
};

// Implementation is inline
#include "MultiTouchAPI.hpp"
105 changes: 105 additions & 0 deletions src/HID-APIs/MultiTouchAPI.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
/*
Copyright (c) 2021 ilufang
See the readme for credit to other people.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

// Include guard
#pragma once

#define _LSB(v) ((v >> 8) & 0xff)
#define _MSB(v) (v & 0xff)
NicoHood marked this conversation as resolved.
Show resolved Hide resolved

void MultiTouchAPI::begin() {
send();
}

int MultiTouchAPI::setFinger(uint8_t id, int16_t x, int16_t y, int8_t pressure) {
if (id >= HID_MULTITOUCH_MAXFINGERS) {
return 0;
}
if (_fingers[id].status == _MT_STATE_INACTIVE) {
_fingers_count++;
}
_fingers[id].status = _MT_STATE_CONTACT;
_fingers[id].pressure = pressure;
_fingers[id].x = x;
_fingers[id].y = y;
return 1;
}

int MultiTouchAPI::releaseFinger(uint8_t id) {
if (id >= HID_MULTITOUCH_MAXFINGERS) {
return 0;
}
_fingers[id].status = _MT_STATE_RELEASED;
return 1;
}

int MultiTouchAPI::send() {
int ret = 0;

// Craft report(s)
_report.count = _fingers_count;

int rptentry=0;
for (int i = 0; i < HID_MULTITOUCH_MAXFINGERS; i++) {
if (_fingers[i].status == _MT_STATE_INACTIVE)
continue;

_report.contacts[rptentry].identifier = i; // valid for first report only

if (_fingers[i].status == _MT_STATE_RELEASED) {
// Released contacts need to be reported once with TipSW=0
_report.contacts[rptentry].touch = {};
_fingers_count--;
_fingers[i].status = _MT_STATE_INACTIVE;
} else {
// Active contacts must be reported even when not moved
_report.contacts[rptentry].touch = _fingers[i].pressure > 0 ? 3 : 1;
_report.contacts[rptentry].x1 = _MSB(_fingers[i].x);
_report.contacts[rptentry].x0 = _LSB(_fingers[i].x);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you declare this x1 and x0 variable as int16, there is no need to to the MSB LSB shifting.

Like here:
https://github.com/NicoHood/HID/blob/master/src/HID-APIs/AbsoluteMouseAPI.h#L47-L48

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

_report.contacts[rptentry].y1 = _MSB(_fingers[i].y);
_report.contacts[rptentry].y0 = _LSB(_fingers[i].y);
_report.contacts[rptentry].pressure = _fingers[i].pressure;
}

rptentry++;
if (rptentry == HID_MULTITOUCH_REPORTFINGERS) {
// Report full. Send now.
// If there are more contacts, they will be sent in subsequent
// reports with contact count set to 0
// See "Hybrid Mode" on MSDN docs
ret += _sendReport();
rptentry = 0;
_report.count = 0;
}
}

if (rptentry != 0) {
// Send remaining touches
for (; rptentry != HID_MULTITOUCH_REPORTFINGERS; rptentry++) {
_report.contacts[rptentry] = {};
}
ret += _sendReport();
}

return ret;
}
1 change: 1 addition & 0 deletions src/HID-Project.h
Original file line number Diff line number Diff line change
Expand Up @@ -52,5 +52,6 @@ THE SOFTWARE.
#include "SingleReport/SingleNKROKeyboard.h"
#include "MultiReport/NKROKeyboard.h"
#include "MultiReport/SurfaceDial.h"
#include "SingleReport/MultiTouch.h"

// Include Teensy HID afterwards to overwrite key definitions if used
Loading