DJ Kurlander - Microsoft Surface Table

DJ Kurlander - Project Resources

Microsoft Surface (PlayTable)

DJ Kurlander presenting the very first PlayTable prototype to Bill Gates in 2003.

Overview

Microsoft Surface Table enables a radically different way of interacting with computer technology. It has a large table top display, enhanced by sophisticated sensing technology. Using computer vision, it detects object shapes, unique tags (or bar codes), as well as the touch of any number of fingers. In response, it animates the surface, or triggers real world objects that have been placed upon it.

We originally envisioned the Microsoft Surface Table as a center point for family entertainment, enhanced by computer technology. Three high level goals motivated this product:

First, we wanted to enable people to cooperatively play and work together. As part of this, we envisioned a new "hearth" experience for the home. Family members could all interact with the Surface Table at the same time, and remote friends or family could be visually represented on the device, for a unified, collaborative experience.

Second, we aimed to enable people to interact more naturally with computers. Tangible objects, like game pieces, dice, and cards could be used as "input" to the table. Surface Table understood finger touches and gestures. It distinguished between similar objects via tags. By aiming to make the interaction more natural than traditional computers, we created a device that required less learning and was more intuitive.

Third, we strived to create a technology that enables new and enhanced entertainment. The physical controls the virtual, but the virtual also controls the physical. This enhances traditional games and entertainments, as well as totally new entertainment experiences.

History

David (DJ) Kurlander was General Manager of Microsoft's New Consumer Products effort, and the Surface Table team. In 2002, he gathered a group of people from throughout the company to brainstorm on ideas for new consumer products. Two different groups suggested a computerized game table, and below is a sketch that came out of that meeting.

Initially, DJ put together a virtual team of people with different expertise to build the prototype. Stevie Bathiche, from Microsoft Hardware, built the first prototype, and Andy Wilson, from Microsoft Research, developed the initial vision / system software. Allen Han, from Microsoft Hardware, worked on design, and DJ, Joel Dehlin, and Christina Chen from the New Consumer Products group, developed scenarios and interaction techniques. Michael Kim, also of New Consumer Products, validated the product with potential customers and found substantial interest.

At the same time, the team worked with Lego Corporation on the product concept. New Consumer Products had been asked by Steve Ballmer to explore a potential joint product with Lego Corporation. Although that joint product never materialized, it was still helpful (and incredibly fun) for Microsoft to work with Lego while scoping out the product.

Initially the we called our product, "The Virtual Game Table", but soon after it became called "PlayTable". We presented the PlayTable product to Bill Gates and other execs in 2003. Bill became a great supporter of the product.

DJ Kurlander's son Jake is pictured playing with the paint application on what he called "The Magic Table" back in 2003.

DJ Kurlander is pictured here with Dan (lower left) and Jake (lower right), situated in front of four Microsoft Surface Tables in the Disneyland Innoventions Dream Home in 2012

Scenarios and Applications

The best way to get a sense of the capabilities of the Microsoft Surface Table is to watch an introductory video. However, here I'll present a number of the applications that the team built while developing Surface, which will hopefully give you a idea of its value and potential.

Virtual Pinball - One of the earliest Table applications, virtual pinball (pictured above), simulated a virtual ball moving through a pinball machine. Using touch, people would trigger virtual bumpers to launch the ball on the Table surface, but diverse kinds of bumpers and play field obstacles could be placed on the field - for example, a physical 'black hole bumper' that would gravitationally affect the ball. Physical bumpers would also react to a ball hitting them by flashing and vibrating (triggered by lighting up the screen under the bumper).

Paint - This application allowed people to draw and create artwork on the surface of the Table, using both virtual and physical tools. They could use real paint brushes whose diameter and path were processed by the computer vision system. Virtual paint colors could be mixed with real brushes. Physical stamps allowed people to add prefabricated art into their illustrations. Any number of people could work on a piece of artwork simultaneously.

Water Simulation - This screensaver-type application simulates water over a variety of different background, such as rocks in a stream, a golf ball in a water trap, coins in a fountain, or even the internals of the Surface Table. By touching the surface, people create ripples and waves, distorting the background in a very real way.

Tanks - In this game, people use a physical remote control to drive a real toy mini-tank on the surface of the Table, and to compete with another real mini-tank driven by the table. Physical obstacles, such as walls or cliffs, placed on the table, are detected by the computer vision sensing system, and the Surface Table blocks the tanks from traveling through them (via acting as an intermediary through the remote control), and adjusts the virtual terrain accordingly. The tanks, both human driven and table driven, are constrained by a changing terrain, including virtual and physical objects. When the line-of-sight is clear, tanks can fire upon one another, which is realistically animated by the Table.

Educational Games and Imaginative Play - Spelling and math games allow kids to use real alphabet flash cards to practice their spelling on the Table, or real coins to practice their math. The Table determines which flash card was placed by a bar code / tag on back, and distinguishes between coins by their size. The Table also supported imaginitive play via Lego games, in which the Table would differentiate between lego bricks and figures by their shape or a tag on their underside. Lego was one of our early partners in developing Table scenarios.

Family Games: Trivial Pursuit / Scrabble / TransAmerica / Robobattle, etc... - The Surface Table team implemented numerous games as prototypes. For Trivial Pursuit, the Table understood rolls of real dice, and the movement of the Trivial Pursuit game pieces on the board. A visor - a real world object that only allows a single player to see a portion of the table, selectively revealed the answer. Similarly, in Scrabble, visors hid players' virtual letter tiles from each other. In games such as Robobattle, the Table simulated complex game mechanics, freeing the players from bookkeeping and game mechanics that made the game play faster and more fun.

Photo organization - The Table also provided a surface to organize digital photography. People could place a phone on the Table to retrieve photos from it, and then sort the photos into different bins, represented by real world objects, which could be brought from Table to Table, or opened virtually on a desktop.

Some other scenarios and applications include:

Video Puzzle - Users are challenged to solve a puzzle, made out of transparent tiles. The Table sees an otherwise invisible infrared tag on each piece, so it can uniquely distinguish between them. Fragments of the video frame are projected through each tile, and people try to reorganize them to show the complete, unified video.
TV Companion - The table acts as both a remote control, and secondary display providing additional information about the show being watched, as well as other shows that interest the user. The table could network in friends who were watching a show simultaneously, allowing them to share comments or video annotations.

Acknowledgments

A team of great visionary folks within Microsoft helped to make Surface Table a reality. DJ Kurlander led the ideation and early productization efforts, but he had the incredibly great fortune of working with a team of tremendous collaborators, including Stevie Bathiche, Andy Wilson, Joel Dehlin, Christina Chen, Nigel Keam, Doug Kramer, Kris Nye, Dan Robbins, Allen Han, Michael Kim, Jon Canfield, and many others

Videos

Microsoft Surface Table (PlayTable). David Kurlander and New Consumer Products Team. Microsoft. July 2003. Duration: 9:07.