- Fact 1: Dazzling animated displays (sprites, shaders, parallax, 3D) are more plentiful and easier to make than ever before.
- Fact 2: More natural and expressive forms of input (swiping, pinching, gesturing, talking) are being implemented and enhanced every day.
- Fact 3: Put these two together and the possible new forms of human computer interaction are endless. The only limit is our imagination.
That’s the problem: our imagination. We can’t build new interactions until A) someone imagines them, and B) the idea is conveyed to other people. As a designer in the Emerging Interactions subgroup of the AppsLab, this is my job – and I’m finding that both parts of it are getting harder to do.
If designers can’t find better ways of imagining – and by imagining I mean the whole design process from blank slate to prototype – progress will slow and our customers will be unable to unleash their full potential.
So what does it mean to imagine and how can we do it better?
Imagination starts with a daydream or an idle thought. “Those animations of colliding galaxies are cool. I wonder if we could show a corporate acquisition as colliding org charts. What would that look like?”
What separates a mere daydreamer from an actual designer is the next step: playing. To really imagine a new thing in any meaningful way you have to roll up your sleeves and actually start playing with it in detail. At first you can do this entirely in your mind – what Einstein called a “thought experiment.” This can take weeks of staring into space while your loved ones look on with increasing concern.
Playing is best done in your mind because your mind is so fluid. You can suspend the laws of physics whenever they get in the way. You can turn structures inside out in the blink of an eye, changing the rules of the game as you go. This fluidity, this fuzziness, is the mind’s greatest strength, but also its greatest weakness.
So sooner or later you have to move from playing to trying. Trying means translating the idea into a visible, tangible form and manipulating it with the laws of physics (or at least the laws of computing) re-enabled. This is where things get interesting. What was vaguely described must now be spelled out. The inflexible properties of time and space will expose inconvenient details that your mind overlooked; dealing with even the smallest of these details can derail your entire scheme – or take it wild, new directions. Trying is a collaboration with Reality.
Until recently, trying was fairly easy to do. If the thing you were inventing was a screen layout or a process flow, you could sketch it on paper or use a drawing program to make sure all the pieces fit together. But what if the thing you are inventing is moving? What if it has hundreds of parts each sliding and changing in a precise way? How do you sketch that?
My first step in the journey to a better way was to move from drawing tools like Photoshop and OmniGraffle to animation tools like Hype and Edge – or to Keynote (which can do simple animations). Some years ago I even proposed a standard “animation spec” so that developers could get precise frame-by-frame descriptions.
The problem with these tools is that you have to place everything by hand, one element at a time. I often begin by doing just that, but when your interface is composed of hundreds of shifting, spinning, morphing shapes, this soon becomes untenable. And when even the simplest user input can alter the course and speed of everything on the screen, and when that interaction is the very thing you need to explore, hand drawn animation becomes impossible.
To try out new designs involving this kind of interaction, you need data-driven animation – which means writing code. This is a significant barrier for many designers. Design is about form, color, balance, layout, typography, movement, sound, rhythm, harmony. Coding requires an entirely different skill set: installing development environments, converting file formats, constructing database queries, parsing syntaxes, debugging code, forking githubs.
A software designer needs a partial grasp of these things in order to work with developers. But most designers are not themselves coders, and do not want to become one. I was a coder in a past life, and even enjoy coding up to a point. But code-wrangling, and in particular debugging, distracts from the design process. It breaks my concentration, disrupts my flow; I get so caught up in tracking down a bug that I forget what I was trying to design in the first place.
The next stage of my journey, then, was to find relatively easy high-level programming languages that would let me keep my eye on the ball. I did several projects using Processing (actually Processing.js), a language developed specifically for artists. I did another project using Python – with all coding done on the iPad so that I could directly experience interactions on the tablet with every iteration of the code.
These projects were successful but time-consuming and painful to create. Traditional coding is like solving a Rubik’s Cube: twist and twist and twist until order suddenly emerges from chaos. This is not the way I want to play or try. I want a more organic process, something more like throwing a pot: I want to grab a clump of clay and just continuously shape it with my hands until I am satisfied.
I am not the only one looking for better ways to code. We are in the midst of an open source renaissance, an explosion of literally thousands of new languages, libraries, and tools. In my last blog post I wrote about people creating radically new and different languages as an art form, pushing the boundaries in all directions.
In “The Future of Programming,” Eric Elliott argues for reactive programming, visual IDEs, even genetic and AI-assisted coding. In “Are Prototyping Tools Becoming Essential?,” Mark Wilcox argues that exploring ideas in the Animation Era requires a whole range of new tools. But if you are only going to follow one of these links, see Bret Victor’s “Learnable Programming.”
After months of web surfing I stumbled upon an interesting open source tool originally designed for generative artists that I’ve gotten somewhat hooked on. It combines reactive programming and a visual IDE with some of Bret Victor’s elegant scrubbing interactions.
More about that tool in my next blog post…
