Media-controlled games

Researchers at Nokia Research Center in Tampere are experimenting with new technology that could transform the way we play games on our mobile phones.

Instead of a game having a musical accompaniment wired into it, we could play the music of our choice and this could make the game easier, harder or more varied. For example, the sound of a cymbal could mean a new enemy appearing

Other game ideas include one where you have to keep your plants alive while bugs try to eat them: the faster the music, the more bugs appear and the harder the game becomes. Not only that, but different sounds produce different kinds of bugs.

Audio striptease dancer, meanwhile, reacts to certain music by moving in a certain way or removing an item of clothing.

In general, the idea has been to make use of sound in games more sophisticated. There have already been many games using the human voice, through pitch recognition software (for example karaoke machines which award points for accurate singing), but so far nothing that can react in many different and challenging ways to a complex piece of music.

The researchers have a vision of music-controlled games where players might tell their friends: "I could get through the game when I played that Queen track, but when I put on Metallica, it was just too hard." This could extend the life of a game, or create entirely new games, through mixing different media and conditions.

"The ultimate goal of the music-based research is to combine games with MP3 players," says Jukka Holm in the Music Technologies group of Nokia Research Center. "Then if you have a 4 gigabyte system peopl could play through their playlist and thus have a huge number of potential game levels available."

At the same time, researchers are looking at using images to affect games in a similar way. You could take a photograph of something - maybe a car or a motorbike - and use its visual features to affect game behaviour.

One way of using this technology is through a game of 'pong', similar to the basic ping pong video game where each player has a 'bat' and moves it up and down the screen, while a video 'ball' moves between them.

In this image-controlled version, the ball changes colour each time it hits a bat. Then you - the opponent - have to point the camera on your phone to a certain colour, for example the sky, to make your bat blue, or grass to make it green. You would lose a point if you did not make your bat the right colour in time.

Other visually-influenced games include 'Excavator', where the objective is to collect diamonds as quickly as possible. The diamonds are placed on the most distinct edges of the image, and the avatar's speed is determined by the brightness of its location on the image.

This year Holm has given plesentations at the Game Developers Conference and ACE2005. "The feedback has been really positive," he says.

Holm and his fellow researchers are keen to speak to colleagues around Nokia who would be interested in taking these technologies forward. "We're looking for collaborators in image-controlled games; especially the people who are involved with camera phones and image processing in general," says Holm. "On the music side, interesting future collaborators would be people who are making Nseries phones and other systems with several gigabyte storage."