Mixed Reality Demo – The Physical Parts

I have been always intrigued by the fact that people get deeply attached to the characters in the game (e.g. Second Life), or virtual pets. And with sufficient advancement in technology, the virtual characters may eventually cross the boundary and get attached to real-life people (e.g. Sci-Fi movie such as “Her”). While that is still a little far away from now, I’ve been looking to explore the 2-way communication and interaction between virtual and real world.

At AppsLab, we have enough skills to build some physical toys that we can communicate and control, but we miss a game or virtual environment that is appealing and communicative. I tried interact with Minecraft environment but stopped when it was sold. So Jake’s casual mention of MindWurld from Ed Jones (@edhjones) sparked a great interest!

MindWurld is a fantastic game.  You can choose a virtual character (Avatar) to walk around Hawaii island to free and rescue pigs, also collect treasure, and play trick of spawning pigs and catching them with Pokeball. And yes, we have full access to the source code. (see Ed’s post for details)

So we came up with a game plot quickly, as manifested in the final build:

1. Player in Real world communicates to a virtual character in MindWurld;
2. Virtual game character and object has a mirrored object in the real world;
3. Events or actions happening in sync between real and virtual objects.

This is how we put things together:

Step 1 – Toy guitar as controller

We thought of using player’s own cellphone to call a number to reach the Avatar (the virtual character in game), and just talk over the phone to tell Avatar what to do. But voice service provider was not responsive enough and we were approaching OpenWorld soon, so we ditched that approach and went for a customized controller.

Ed is a guitar player, and the virtual Avatar would be attending OpenWorld on behalf of Ed, so it is fitting that we use a toy guitar to represent him.

The toy guitar essentially provides many buttons that I can use to convey various commands and intentions, but the mod itself is a little bit more complex, as each button produce a set of signals feeding into a chip for playing music, it is not a clear simple one push to one line reading.

I used one Arduino mini pro to read signal patterns for each button push and did some noise filtering and process, and then translated them into “a player command,” which is feed into a Bluefruit EZ-key HID chip. The HID chip can connect to a computer as HID device, so each “play command” is a simple key stoke input to control the game.

Step 2 – MiP robot as mirrored avatar

MiP robot from WowWee is an inexpensive but very capable little robot. It can balance itself on two wheels, and can move back-forth, and spin on the spot, and that makes it having accurate travel along any path.

Oh, and it is quite a character. It makes happy, grumpy and lots of other noises, and shows many light patterns, to express full range of emotions!

The best part for our developers – it has an API in many languages that we can program and control the movement, sound and lights.

So for whatever events happening in the MindWurld game, such as the avatar walking around, opening treasure boxes, spawning pigs, freeing pigs and rescuing them, they are sent over a socket to a my Robot controller program, which in turn asks the Robot to perform corresponding movement and act in certain cheerful ways.

Originally, I made the MiP robot to be the mirror of virtual character in the game, in a sense that it walks the same way as his virtual part in game. It requires a large area for it to roam around. For the OAUX Exchange at OpenWorld, due to space limitation, I reprogrammed it to be a buddy of the virtual character, so it does not move too much, but makes noise and blinks light to cheer for his virtual friend.

By now, we can test out the full cast of game!

Step 3 – Juiced it up with a Pokeball

Meanwhile, Mark (@mvilrokx) had been busy printing Pokeballs:  3D printed shells, polished and painted, outfitted with unbalance motor for vibration, LED for light color effect, and  NodeMCU for network connectivity, and hooked up to a MQTT broker ready for action.

Ed quickly outfitted the virtual character with a ball in hand, throwing at pigs to rescue them.

I just quickly added a MQTT client code, replied ball-thrown and pig-rescued events to MQTT broker. And the Pokeball in real world would vibrate and flash when the virtual character throws and catches pigs in the MindWurld.

Oh, that’s the Mixed Reality game setup at OAUX Exchange. Anthony had 3 days of fun time playing Rock Star, together with “real” people, “virtual” avatar, “real” avatar, “virtual” balls and “real” balls.