1

For the past few months I've been looking into developing a Kinect based multitouch interface for a variety of software music synthesizers.

The overall strategy I've come up with is to create objects, either programatically or (if possible) algorithmically to represent various controls of the soft synth. These should have;

  • X position
  • Y position
  • Height
  • Width
  • MIDI output channel
  • MIDI data scaler (convert x-y coords to midi values)

2 strategies I've considered for agorithmic creation are XML description and somehow pulling stuff right off the screen (ie given a running program, find xycoords of all controls). I have no idea how to go about that second one, which is why I express it in such specific technical language ;). I could do some intermediate solution, like using mouse clicks on the corners of controls to generate an xml file. Another thing I could do, that I've seen frequently in flash apps, is to put the screen size into a variable and use math to build all interface objects in terms of screen size. Note that it isn't strictly necessary to make the objects the same size as onscreen controls, or to represent all onscreen objects (some are just indicators, not interactive controls)

Other considerations;

Given (for now) two sets of X/Y coords as input (left and right hands), what is my best option for using them? My first instinct is/was to create some kind of focus test, where if the x/y coords fall within the interface object's bounds that object becomes active, and then becomes inactive if they fall outside some other smaller bounds for some period of time. The cheap solution I found was to use the left hand as the pointer/selector and the right as a controller, but it seems like I can do more. I have a few gesture solutions (hidden markov chains) I could screw around with. Not that they'd be easy to get to work, exactly, but it's something I could see myself doing given sufficient incentive.

So, to summarize, the problem is

  • represent the interface (necessary because the default interface always expects mouse input)
  • select a control
  • manipulate it using two sets of x/y coords (rotary/continuous controller) or, in the case of switches, preferrably use a gesture to switch it without giving/taking focus.

Any comments, especially from people who have worked/are working in multitouch io/NUI, are greatly appreciated. Links to existing projects and/or some good reading material (books, sites, etc) would be a big help.

Adam Lear
  • 32,069
jamesson
  • 537

1 Answers1

0

  1. Every touch controller I have seen that has a knob on it, you turn the knob by moving your finger in a line. So to turn a particular knob, I would touch the knob, and then slide my finger up to turn the knob clockwise, and down to turn it counterclockwise. Lifting your finger releases the knob. Your users will be accustomed to this convention, if they've ever used someone else's control surface.

    Moving a slider with my finger feels like moving a slider. Turning a knob in the usual way does not, because it relies on vertical tactile sensation. This is in large part why I bought an Oxygen 61 for my own personal setup, so I could have some real knobs to turn.

  2. Rendering your control surface and its mappings to an XML file is a matter of creating a schema and writing code to perform the translations between your XML and your control surface. To get some ideas about how this might work, consider studying Microsoft's WPF or Silverlight models; they use XAML (Microsoft's own XML schema) to render user interfaces.

    Rely heavily on third party XML libraries, and consider techniques such as serialization; these tools will greatly speed up your development effort.

  3. In terms of letting people develop their own control surfaces, it seems axiomatic that they're not going to write XML to do it. You will need to provide some kind of drag and drop from a toolbox to a designer surface, and then save that out to the XML file.

    Allow your users to patch the controls into a MIDI channel by touching them after selecting a control element. Many current music packages such as Abelton Live already support this; you click on the item you want to control, and then touch the control on your control surface that you want to map to it, and it will automatically get patched in.

Of course, all of this depends on what sort of gestures for touch (and un-touch) are available for the Kinect. Also, if the Kinect knows how to detect rotary motion of your hand, you could simply translate that into rotary motion for the knob, with say a 2:1 scaling so that you can get a complete turn without turning your hand all the way around.

IMPORTANT: If you haven't yet, as part of your research, watch the movie "Minority Report."

See also http://en.wikipedia.org/wiki/User_interface_markup_language

Robert Harvey
  • 200,592