Better multi-touch on iOS

[hlynbech]

Hi everybody,

I found a minor bug in juce_iphone_UIViewComponentPeer.mm, which in effect cancels all touches if you lift one finger. In other words, if you have touched the screen with two or more fingers and then releases just one finger, you get a "touch up" event on all fingers in JUCE.

This is because you have to inspect the iOS event variable when receiving a -(void)touchesEnded call, to find out which touch was ended. The current JUCE code does not do this.

The correction of the JUCE code is in the interpretation of the touches in UIViewComponentPeer::handleTouches, and should look something like this:

Code: Select all

void UIViewComponentPeer::handleTouches (UIEvent* event, const bool isDown, const bool isUp, bool isCancel)
{
    NSArray* touches = [[event touchesForView: view] allObjects];

    for (unsigned int i = 0; i < [touches count]; ++i)
    {
        UITouch* touch = [touches objectAtIndex: i];

        CGPoint p = [touch locationInView: view];
        const Point<int> pos ((int) p.x, (int) p.y);
        juce_lastMousePos = pos + getScreenPosition();

        const int64 time = getMouseTime (event);

        int touchIndex = currentTouches.indexOf (touch);

        if (touchIndex < 0)
        {
            touchIndex = currentTouches.size();
            currentTouches.add (touch);
        }

        if ([touch phase] == UITouchPhaseBegan || [touch phase] == UITouchPhaseStationary || [touch phase] == UITouchPhaseMoved) //was: isDown)
        {
            currentModifiers = currentModifiers.withoutMouseButtons();
            handleMouseEvent (touchIndex, pos, currentModifiers, time);
            currentModifiers = currentModifiers.withoutMouseButtons().withFlags (ModifierKeys::leftButtonModifier);
        }
        else if ([touch phase] == UITouchPhaseEnded || [touch phase] == UITouchPhaseCancelled) //was: (isUp)
        {
            currentModifiers = currentModifiers.withoutMouseButtons();
            currentTouches.remove (touchIndex);
        }

        if (isCancel)
            currentTouches.clear();

        handleMouseEvent (touchIndex, pos, currentModifiers, time);
    }
}

[jules]

Ah, very interesting! Thanks a lot for the bug-hunting, I'll have a look at that right away!

[martinrobinson]

Just for the record... I found a bug in this solution that produced false double-taps and other weird behaviour. But this is one of the threads that bit the dust yesterday with the server crash.

I've no idea what Jules' response was (as I was just connecting as the server crashed!) but it seems fixed now with the 1.53.82 update.

Thanks Jules

[martinrobinson]

It's still not quite right... latest tip 1.53.82 of course...

Basic test component:

Code: Select all

class MainComponent : public Component, public LookAndFeel
{
public:
   MainComponent()
   {
      setLookAndFeel (this);
      slider1.setTextBoxStyle(Slider::NoTextBox, 0, 0, 0);
      slider2.setTextBoxStyle(Slider::NoTextBox, 0, 0, 0);
      addAndMakeVisible (&slider1);
      addAndMakeVisible (&slider2);
   }

   void resized()
   {
      slider1.setBounds (20, 20, getWidth() - 40, 40);
      slider2.setBounds (20, 80, getWidth() - 40, 40);
   }
   
   int getSliderThumbRadius (Slider& slider)
   {
      return 20;
   }
   
private:
   Slider slider1;
   Slider slider2;
};

The sliders work fine when using single touches (one finger at a time). They sometimes work with two fingers, - one finger per slider but often not:

• It seems critical that when the second finger touches the second slider the first finger must be still directly over the first slider's thumb otherwise the first finger seems to get "cancelled".
• When the fingers are released, one of the sliders (usually) remains highlighted (as if mouseExit() or the equivalent hasn't been called) and/or both sliders jump to the same position. This doesn't seem to be dependent on the order of releasing the fingers (so it seems to randomly exhibit one of these behaviours if you do 1-down, 2-down, 1-up, 2-up; or, 1-down, 2-down, 2-up, 1-up).

[martinrobinson]

Added this quick-and-dirty sublclass of Slider to see what's going on:

Code: Select all

class MySlider : public Slider
{
public:
   void mouseEnter (const MouseEvent& e)
   {
      printf("mouseEnter[%d]: { %d, %d } time(%u)\n",
            e.source.getIndex(),
            e.x, e.y,
            (uint32)e.eventTime.toMilliseconds());

      Slider::mouseEnter(e);
   }
   
   void mouseExit (const MouseEvent& e)
   {
      printf("mouseExit[%d]: { %d, %d } time(%u)\n",
            e.source.getIndex(),
            e.x, e.y,
            (uint32)e.eventTime.toMilliseconds());
      
      Slider::mouseExit(e);
   }
   
   void mouseDown (const MouseEvent& e)
   {
      printf("mouseDown[%d]: { %d, %d } down{ %d, %d } time(%u)\n",
            e.source.getIndex(),
            e.x, e.y,
            e.getMouseDownPosition().getX(), e.getMouseDownPosition().getY(),
            (uint32)e.eventTime.toMilliseconds());
      
      Slider::mouseDown(e);
   }
   
   void mouseDrag (const MouseEvent& e)
   {
      printf("mouseDrag[%d]: { %d, %d } down{ %d, %d } time(%u)\n",
            e.source.getIndex(),
            e.x, e.y,
            e.getMouseDownPosition().getX(), e.getMouseDownPosition().getY(),
            (uint32)e.eventTime.toMilliseconds());
      
      Slider::mouseDrag(e);
   }
   
   void mouseUp (const MouseEvent& e)
   {
      printf("mouseUp[%d]: { %d, %d } down{ %d, %d } time(%u)\n",
            e.source.getIndex(),
            e.x, e.y,
            e.getMouseDownPosition().getX(), e.getMouseDownPosition().getY(),
            (uint32)e.eventTime.toMilliseconds());
      
      Slider::mouseUp(e);
   }
};

Then replaced Slider with MySlider in the code in the previous post.

Some tests...

finger 1 down on slider 1, finger 2 down on slider 2 (ensuring finger 1 is still over the thumb for slider 1), release both "simultaneously":

Code: Select all

mouseEnter[0]: { 25, 9 } time(2615904011)
mouseDown[0]: { 25, 9 } down{ 25, 9 } time(2615904011)
mouseDrag[0]: { 25, 9 } down{ 25, 9 } time(2615904011)
mouseDrag[0]: { 25, 6 } down{ 25, 9 } time(2615904459)
mouseDrag[0]: { 25, 5 } down{ 25, 9 } time(2615904555)
mouseDrag[0]: { 25, 4 } down{ 25, 9 } time(2615904699)
mouseDrag[0]: { 25, 3 } down{ 25, 9 } time(2615905259)
mouseDrag[0]: { 25, 2 } down{ 25, 9 } time(2615905548)
mouseDrag[0]: { 25, 1 } down{ 25, 9 } time(2615906572)
mouseDrag[0]: { 25, 5 } down{ 25, 9 } time(2615907580)
mouseDrag[0]: { 25, 3 } down{ 25, 9 } time(2615907596)
mouseDrag[0]: { 25, 2 } down{ 25, 9 } time(2615907612)
mouseUp[0]: { 25, 2 } down{ 25, 9 } time(2615907612)
mouseDown[0]: { 25, 2 } down{ 25, 2 } time(2615907612)
mouseDrag[0]: { 25, 2 } down{ 25, 2 } time(2615907612)
mouseDown[1]: { 16, 24 } down{ 16, 24 } time(2615907612)
mouseDrag[1]: { 16, 24 } down{ 16, 24 } time(2615907612)
mouseDrag[0]: { 25, 1 } down{ 25, 2 } time(2615907644)
mouseDrag[1]: { 15, 25 } down{ 16, 24 } time(2615907644)
mouseDrag[0]: { 25, 0 } down{ 25, 2 } time(2615907660)
mouseDrag[1]: { 14, 25 } down{ 16, 24 } time(2615907676)
mouseDrag[1]: { 15, 26 } down{ 16, 24 } time(2615909179)
mouseDrag[0]: { 26, 0 } down{ 25, 2 } time(2615909403)
mouseDrag[1]: { 16, 26 } down{ 16, 24 } time(2615909435)
mouseDrag[1]: { 17, 26 } down{ 16, 24 } time(2615910059)
mouseDrag[0]: { 27, 0 } down{ 25, 2 } time(2615911082)
mouseDrag[1]: { 18, 26 } down{ 16, 24 } time(2615911098)
mouseDrag[0]: { 27, 10 } down{ 25, 2 } time(2615911146)
mouseDrag[0]: { 18, 86 } down{ 25, 2 } time(2615911146)
mouseUp[0]: { 18, 86 } down{ 25, 2 } time(2615911146)
mouseExit[0]: { 18, 86 } time(2615911146)
mouseEnter[0]: { 18, 26 } time(2615911146)
mouseExit[0]: { 25, -48 } time(2615911161)
mouseEnter[0]: { 25, 12 } time(2615911161)

Notice there's a false lifting and replacing of the first finger as the second on goes down. And there are are some extraneous mouseExits and mouseEnters at the end for source index 0 but none for index 1 (i.e., the second finger). The -48 coordinate seems fishy as it's offscreen (I think).

finger 1 down on slider 1, finger 2 down on slider 2 (ensuring finger 1 is still over the thumb for slider 1), release finger 2 (still holding finger 1) :

Code: Select all

mouseEnter[0]: { 119, 21 } time(2616579970)
mouseDown[0]: { 119, 21 } down{ 119, 21 } time(2616579970)
mouseDrag[0]: { 119, 21 } down{ 119, 21 } time(2616579970)
mouseDrag[0]: { 117, 24 } down{ 119, 21 } time(2616580738)
mouseDown[1]: { 338, 19 } down{ 338, 19 } time(2616581170)
mouseDrag[1]: { 338, 19 } down{ 338, 19 } time(2616581170)
mouseUp[0]: { 117, 24 } down{ 119, 21 } time(2616581170)
mouseDown[0]: { 117, 24 } down{ 117, 24 } time(2616581170)
mouseDrag[0]: { 117, 24 } down{ 117, 24 } time(2616581170)
mouseDrag[1]: { 335, 20 } down{ 338, 19 } time(2616581282)
mouseDrag[0]: { 116, 24 } down{ 117, 24 } time(2616581282)
mouseDrag[1]: { 334, 20 } down{ 338, 19 } time(2616581570)
mouseDrag[0]: { 115, 25 } down{ 117, 24 } time(2616581570)
mouseUp[0]: { 115, 25 } down{ 117, 24 } time(2616581826)

finger 1 down on slider 1, drag finger 1 away from slider 1's thumb, finger 2 down on slider 2:

Code: Select all

mouseEnter[0]: { 104, 26 } time(2616654379)
mouseDown[0]: { 104, 26 } down{ 104, 26 } time(2616654379)
mouseDrag[0]: { 104, 26 } down{ 104, 26 } time(2616654379)
mouseDrag[0]: { 103, 26 } down{ 104, 26 } time(2616654987)
mouseDrag[0]: { 103, 27 } down{ 104, 26 } time(2616655003)
mouseDrag[0]: { 103, 28 } down{ 104, 26 } time(2616655019)
mouseDrag[0]: { 103, 30 } down{ 104, 26 } time(2616655035)
mouseDrag[0]: { 103, 31 } down{ 104, 26 } time(2616655051)
mouseDrag[0]: { 103, 33 } down{ 104, 26 } time(2616655067)
mouseDrag[0]: { 103, 36 } down{ 104, 26 } time(2616655083)
mouseDrag[0]: { 103, 39 } down{ 104, 26 } time(2616655099)
mouseDrag[0]: { 103, 42 } down{ 104, 26 } time(2616655115)
mouseDrag[0]: { 103, 45 } down{ 104, 26 } time(2616655131)
mouseDrag[0]: { 103, 48 } down{ 104, 26 } time(2616655147)
mouseDrag[0]: { 103, 50 } down{ 104, 26 } time(2616655163)
mouseDrag[0]: { 103, 51 } down{ 104, 26 } time(2616655179)
mouseDrag[0]: { 103, 52 } down{ 104, 26 } time(2616655195)
mouseDrag[0]: { 103, 53 } down{ 104, 26 } time(2616655211)
mouseDrag[0]: { 103, 54 } down{ 104, 26 } time(2616655227)
mouseDrag[0]: { 103, 55 } down{ 104, 26 } time(2616655243)
mouseDrag[0]: { 103, 57 } down{ 104, 26 } time(2616655260)
mouseDrag[0]: { 103, 58 } down{ 104, 26 } time(2616655275)
mouseDrag[0]: { 103, 59 } down{ 104, 26 } time(2616655291)
mouseDrag[0]: { 103, 61 } down{ 104, 26 } time(2616655307)
mouseDrag[0]: { 103, 62 } down{ 104, 26 } time(2616655323)
mouseDrag[0]: { 103, 63 } down{ 104, 26 } time(2616655355)
mouseDrag[0]: { 103, 64 } down{ 104, 26 } time(2616655515)
mouseUp[0]: { 103, 64 } down{ 104, 26 } time(2616655740)
mouseExit[0]: { 103, 64 } time(2616655740)
mouseDrag[1]: { 369, 18 } down{ 354, 20 } time(2616655740)
mouseUp[1]: { 369, 18 } down{ 354, 20 } time(2616655740)
mouseDown[1]: { 369, 18 } down{ 369, 18 } time(2616655740)
mouseDrag[1]: { 369, 18 } down{ 369, 18 } time(2616655740)
mouseDrag[1]: { 366, 18 } down{ 369, 18 } time(2616657853)

The false mouseUp/mouseDown in the first test explains why the fingers need to stay over the thumbs on subsequent touches.

[jules]

Sorry, looks like some stupidness in my handling of the touch indexes.. Will check in some less broken code shortly.

[martinrobinson]

'fraid not...

finger 1 down on slider 1, finger 2 down on slider 2, finger 2 up, finger 1 up:

Code: Select all

mouseDown[0]: { 65, 17 } down{ 65, 17 } time(2690246034)
mouseDrag[0]: { 65, 17 } down{ 65, 17 } time(2690246034)
mouseDrag[0]: { 54, 19 } down{ 65, 17 } time(2690246466)
mouseDrag[0]: { 53, 19 } down{ 65, 17 } time(2690246514)
mouseDrag[0]: { 52, 19 } down{ 65, 17 } time(2690246563)
mouseDrag[0]: { 51, 19 } down{ 65, 17 } time(2690246659)
mouseDrag[0]: { 50, 19 } down{ 65, 17 } time(2690246707)
mouseDrag[0]: { 49, 19 } down{ 65, 17 } time(2690246755)
mouseDrag[1]: { 210, 17 } down{ 209, 15 } time(2690247011)
mouseUp[1]: { 210, 17 } down{ 209, 15 } time(2690247011)
mouseDown[1]: { 210, 17 } down{ 210, 17 } time(2690247011)
mouseDrag[1]: { 210, 17 } down{ 210, 17 } time(2690247011)
mouseUp[0]: { 49, 19 } down{ 65, 17 } time(2690247011)
mouseDown[0]: { 49, 19 } down{ 49, 19 } time(2690247011)
mouseDrag[0]: { 49, 19 } down{ 49, 19 } time(2690247011)
mouseDrag[1]: { 209, 17 } down{ 210, 17 } time(2690247155)
mouseDrag[0]: { 49, 20 } down{ 49, 19 } time(2690247155)
mouseDrag[1]: { 208, 17 } down{ 210, 17 } time(2690247619)
mouseDrag[1]: { 209, 17 } down{ 210, 17 } time(2690248147)
mouseDrag[0]: { 49, 21 } down{ 49, 19 } time(2690248259)
mouseDrag[0]: { 50, 21 } down{ 49, 19 } time(2690248418)
mouseDrag[1]: { 210, 17 } down{ 210, 17 } time(2690248594)
mouseUp[0]: { 50, 21 } down{ 49, 19 } time(2690248610)


Notice the drag for finger 2 (touch index 1) BEFORE the down event! This doesn't happen consistently but is a sign it's not quite there yet!
(Also still notice the false up/down for finger 1.) The final "mouse[0]" is actually reported for the lifting of finger 2 and there's no message for the final lifting of finger 1.

I'm trying to see what's going on behind the scenes but I'm just getting my head around the various layers of calls with several different files open tracing what triggers what! This stuff is a nightmare to debug.

[martinrobinson]

On a related note would a fake event to force a mouseExit be a bad idea:

juce_ios_UIViewComponentPeer.mm

Code: Select all

        ...

        if (isCancel)
        {
            currentTouches.clear();
            currentModifiers = currentModifiers.withoutMouseButtons();
        }
           
        handleMouseEvent (touchIndex, pos, currentModifiers, time);
       
        if (isUp || isCancel)
            handleMouseEvent (touchIndex, Point<int> (-1, -1), currentModifiers, time);
    }


[jules]

Goddammit.. Not sure why this is proving so difficult!

Looks like there was still some confusion in there between the global set of modifiers and the touch-specific modifiers - I think this might help (but haven't tried it yet):

Code: Select all

void UIViewComponentPeer::handleTouches (UIEvent* event, const bool isDown, const bool isUp, bool isCancel)
{
    NSArray* touches = [[event touchesForView: view] allObjects];

    for (unsigned int i = 0; i < [touches count]; ++i)
    {
        UITouch* touch = [touches objectAtIndex: i];

        CGPoint p = [touch locationInView: view];
        const Point<int> pos ((int) p.x, (int) p.y);
        juce_lastMousePos = pos + getScreenPosition();

        const int64 time = getMouseTime (event);

        int touchIndex = currentTouches.indexOf (touch);

        if (touchIndex < 0)
        {
            for (touchIndex = 0; touchIndex < currentTouches.size(); ++touchIndex)
                if (currentTouches.getUnchecked (touchIndex) == nil)
                    break;

            currentTouches.set (touchIndex, touch);
        }

        ModifierKeys modsToSend (currentModifiers);

        if (isDown)
        {
            currentModifiers = currentModifiers.withoutMouseButtons().withFlags (ModifierKeys::leftButtonModifier);
            modsToSend = currentModifiers;

            // this forces a mouse-enter/up event, in case for some reason we didn't get a mouse-up before.
            handleMouseEvent (touchIndex, pos, modsToSend.withoutMouseButtons(), time);
        }
        else if (isUp)
        {
            modsToSend = modsToSend.withoutMouseButtons();
            currentTouches.set (touchIndex, nil);

            int totalActiveTouches = 0;
            for (int j = currentTouches.size(); --j >= 0;)
                if (currentTouches.getUnchecked(j) != nil)
                    ++totalActiveTouches;

            if (totalActiveTouches == 0)
                isCancel = true;
        }

        if (isCancel)
        {
            currentTouches.clear();
            currentModifiers = currentModifiers.withoutMouseButtons();
        }

        handleMouseEvent (touchIndex, pos, modsToSend, time);
    }
}

[martinrobinson]

Nope.. still not right:

finger 1 down on slider 1, finger 2 down on slider 2, finger 2 up, (finger 1 still down):

Code: Select all

mouseDown[0]: { 27, 28 } down{ 27, 28 } time(2699662089)
mouseDrag[0]: { 27, 28 } down{ 27, 28 } time(2699662089)
mouseDrag[0]: { 29, 26 } down{ 27, 28 } time(2699662712)
mouseDrag[0]: { 30, 26 } down{ 27, 28 } time(2699662728)
mouseDrag[0]: { 31, 26 } down{ 27, 28 } time(2699662840)
mouseDrag[0]: { 31, 27 } down{ 27, 28 } time(2699663224)
mouseDrag[0]: { 31, 28 } down{ 27, 28 } time(2699664168)
mouseDown[1]: { 247, 12 } down{ 247, 12 } time(2699664473)
mouseDrag[1]: { 247, 12 } down{ 247, 12 } time(2699664473)
mouseUp[0]: { 31, 28 } down{ 27, 28 } time(2699664473)
mouseDown[0]: { 31, 28 } down{ 31, 28 } time(2699664473)
mouseDrag[0]: { 31, 28 } down{ 31, 28 } time(2699664473)
mouseDrag[1]: { 246, 14 } down{ 247, 12 } time(2699664488)
mouseDrag[1]: { 245, 14 } down{ 247, 12 } time(2699664665)
mouseDrag[0]: { 31, 29 } down{ 31, 28 } time(2699664665)
mouseUp[1]: { 245, 14 } down{ 247, 12 } time(2699665272)
mouseUp[0]: { 31, 29 } down{ 31, 28 } time(2699665272)

..but finger 1 is reported as lifted at the same time as finger 2 (and there's still the false touch up/down on finger 1 as finger 2 goes down).

[martinrobinson]

What about this, it seems to work...

Code: Select all

void UIViewComponentPeer::handleTouches (UIEvent* event, const bool isDown, const bool isUp, bool isCancel)
{   
    NSArray* touches = [[event touchesForView: view] allObjects];
   
    for (unsigned int i = 0; i < [touches count]; ++i)
    {
        UITouch* touch = [touches objectAtIndex: i];
       
        if([touch phase] == UITouchPhaseStationary)
            continue;
       
        CGPoint p = [touch locationInView: view];
        const Point<int> pos ((int) p.x, (int) p.y);
        juce_lastMousePos = pos + getScreenPosition();
       
        const int64 time = getMouseTime (event);
       
        int touchIndex = currentTouches.indexOf (touch);
       
        if (touchIndex < 0)
        {
            for (touchIndex = 0; touchIndex < currentTouches.size(); ++touchIndex)
                if (currentTouches.getUnchecked (touchIndex) == nil)
                    break;
           
            currentTouches.set (touchIndex, touch);
        }
       
        ModifierKeys modsToSend (currentModifiers);
       
        if (isDown)
        {
            if ([touch phase] != UITouchPhaseBegan)
                continue;
           
            currentModifiers = currentModifiers.withoutMouseButtons().withFlags (ModifierKeys::leftButtonModifier);
            modsToSend = currentModifiers;
           
            // this forces a mouse-enter/up event, in case for some reason we didn't get a mouse-up before.
            handleMouseEvent (touchIndex, pos, modsToSend.withoutMouseButtons(), time);
        }
        else if (isUp)
        {
            if (! ([touch phase] == UITouchPhaseEnded || [touch phase] == UITouchPhaseCancelled))
                continue;
           
            modsToSend = modsToSend.withoutMouseButtons();
            currentTouches.set (touchIndex, nil);
           
            int totalActiveTouches = 0;
            for (int j = currentTouches.size(); --j >= 0;)
                if (currentTouches.getUnchecked(j) != nil)
                    ++totalActiveTouches;
           
            if (totalActiveTouches == 0)
                isCancel = true;
        }
       
        if (isCancel)
        {
            currentTouches.clear();
            currentModifiers = currentModifiers.withoutMouseButtons();
        }
       
        handleMouseEvent (touchIndex, pos, modsToSend, time);
    }
}

Although I'd still like to see the mouseExits unless this is the expected behaviour (i.e., no mouseExit until the next mouseEnter casued by another touch).

[martinrobinson]

Actually, adding this in at the end does provide the mouseExits and still seems to work correctly:

Code: Select all

        ...

        if (isCancel)
        {
            currentTouches.clear();
            currentModifiers = currentModifiers.withoutMouseButtons();
        }
       
        handleMouseEvent (touchIndex, pos, modsToSend, time);
       
        if (isUp || isCancel)
            handleMouseEvent (touchIndex, Point<int> (-1, -1), currentModifiers, time);
    }


[jules]

Ok.. seems to make some sense, thanks! The mouseExit is a tricky one, it seems very artificial to just yank the coordinates away to an off-screen position, and I can't help feeling that that'll break something. But.. I guess any code that isn't touch-aware will probably go wrong anyway, and I can't think of a better solution.

[martinrobinson]

Yes it does seem a bit of a kludge. My thinking was that I can't remember any scenario where I needed to inspect the position of a mouseExit but I can remember a number of instances of using a notification of a mouseExit to do something important.

[igor]

I'm sorry to hijack the thread, but I'm really curious how to utilize this code to respond to a gesture, e.g. a pinch, for zoom.

Since UIViewComponentPeer::handleTouches() feeds handleMouseEvent(), is the accepted method to override ComponentPeer::handleMouseEvent(), and then feed the information routed there to a gesture handling state machine of our own creation?

Would it make sense to set up gesture recognizers, like UIPinchGestureRecognizer in the UIViewComponentPeer, and handle the results in a callback, like a handlePinchGesture(UIGestureRecognizer)? If the registration and handling of gestures was done in more generic functions, then they could be stubbed out in the abstract parent class, and filled in on other platforms as gesture engines become available.

I'm sure there's a fatal flaw in my naïve view of things...