Programming tends to conjure images of a lone genius — maybe a Zuckerberg or a Wozniak — hunched over a keyboard in a dark room somewhere, late at night, illuminated only by the glow of a computer monitor filled with arcane expressions. And while the truth is a lot messier, these fantasies feel true, and many programmers embrace them as part of their culture. Personally, I’m not a fan.

What might programming in mixed reality look like?

This is a brief glimpse of Paraspace, the result of my one year attempt to explore that question as a master’s student at the University of Southern California’s Interactive Media and Games Division.

Background

The Affordances of Mixed Reality

The Future of Programming

Many groups have experimented with alternate approaches to programming, including visual node-based and block-based programming models. These methods have found success in specific domains such as visual effects, robotics, and programming education.

However, with few exceptions, they were designed for the same input methodologies as text-based programming environments and may not benefit from an additional spatial dimension.

Show, Don’t Tell: Programming by Demonstration

One programming methodology that does seem promising for creating programs in mixed reality is Programming by Demonstration. In traditional textual and node-based programming environments, programmers tell the computer what they want it to do via an indirect symbolic language. In Programming by Demonstration environments, they show the computer what they want to happen.

Allen Cypher, one of the central proponents of this approach, writes

The motivation behind Programming by Demonstration is simple and compelling: if a user knows how to perform a task on the computer, that should be sufficient to create a program to perform the task. It should not be necessary to learn a programming language like C or BASIC. Instead, the user should be able to instruct the computer to “Watch what I do”, and the computer should create the program that corresponds to the user’s actions.

Some of the most effective mixed reality environments (see, for example, Fantastic Contraption) allow users to complete tasks using natural, embodied actions such as grabbing and moving objects with their hands. This directness lends itself well to Programming by Demonstration.

Paraspace: Exploring Programming by Demonstration in Mixed Reality

Over the past year, I created Paraspace to explore how Programming by Demonstration might work in mixed reality. Unlike other Programming by Demonstration environments, Paraspace is collaborative and designed for the domain of visual simulations. Its target audience is visual artists who would like to experiment with behaviors but are limited by their lack of programming background. Paraspace welcomes one or more artists into a shared physical space that becomes a shared virtual space once they don Vive virtual reality headsets. The artists wield physical Vive controllers that allow them to construct a simple environment using virtual 3d primitive shapes they grab from an infinite virtual shelf. They can make this environment dynamic by creating simple movement and interaction behaviors for the shapes as they experiment, and they can pause and resume the simulation of these behaviors at any time, using a virtual timer with buttons akin to a VCR or video player.

My core design philosophy was one of consistency and conceptual simplicity; while some Programming by Demonstration systems infer user intent frequently, I wanted Paraspace to empower artists to concisely and unambiguously specify their intent. Stagecast Creator and Stop Drawing Dead Fish served as my main inspirations for this philosophy. In the following sections, I explore the design of Paraspace in the context of Programming by Demonstration.

Manipulating the Environment

Before a Programming by Demonstration system can automate something, users must show the system how to do it, which means that they must also be able to do it manually. Paraspace provides three types of objects: cubes, spheres, and cylinders; and three hand-held tools with which artists can manipulate the environment: pliers, a vacuum cleaner, and a stamp. Each tool appears as a virtual tip for a Vive controller and uses the same interaction method: press and hold the trigger while moving the controller to manipulate the one object closest to the tip. The form of each tool matches its function:

• The pliers let artists move (translate and rotate) objects
• The vacuum cleaner lets artists delete individual objects
• The stamp lets artists duplicate objects

While designing these tools, I needed to make carefully considered compromises between fluidity and precision to support the use of these tools for demonstrational programming without frustrating artists.

For example, an early version of the pliers allowed for freeform translation and rotation. Through testing, I discovered that this functionality prevented artists from making objects rotate without translating and vice-versa, because of the physical precision required. The current version of the pliers causes a rotation/translation gizmo to appear around the closest object. Grabbing from the center of the object allows the fluid free-form motion when arranging objects in the environment, while grabbing near the object endpoints or corners allows artists to translate or rotate an object along a single axis using an interface familiar to visual artists who have used 3D modeling or animation programs.

For the sake of precision, the vacuum cleaner only deletes one object at a time rather than continuously “sucking” objects as one might expect.

Programming by Demonstration

The most important (and most challenging) aspect of any Programming by Demonstration system relates to how programs are created, how they are invoked, and how they infer meaning or intent from any inherent ambiguity in a user’s actions. These concepts are all inextricably linked, so I’ll explain the nature of programs in Paraspace before exploring how they are made and the design consequences of their invocation.

Behaviors: Programs in Paraspace

In Paraspace, programs are called behaviors. Each behavior consists of a situation (an object configuration that must exist before the behavior can take effect), and a consequence (a sequence of manipulations that specifies the outcome of the behavior).

One example of a behavior might be:

• when a cube exists (situation), it should rotate a little bit (consequence).

Another might be:

• when a cube and a sphere touch (situation), the cube should be deleted (consequence).

Paraspace is a simulation environment; while the simulation is running, the system continuously performs all the behaviors that have valid situations. If there are multiple instances of the same situation in the environment, a single behavior will take effect for each one. For example, multiple cubes might all move at once because of a behavior that causes a cube to move if it exists. This contrasts with systems like Pygmalion and Mondrian, which rely on users to manually invoke programs from a palette.

The Programming Oven: Creating Paraspace Behaviors by Demonstration

Programming by Demonstration systems handle program creation in a variety of different ways. Some early systems, like Eager and The Predictive Calculator, monitor the environment for repetitive actions. Chimera maintains a graphical history of everything that has happened and allows users to select part of that history to turn into a program. A more common approach is to manually turn on a recording system for the duration of the demonstration, but this has the potential to clutter the context of the recording (the situation part of the behaviors in Paraspace). My favorite approach is Stagecast Creator’s rule tool, which starts a recording and uses the immediate surroundings of the targeted object as the context. However, this solution loses much of its elegance and effectiveness in an environment without Stagecast Creator’s two-dimensional grid.

Paraspace introduces the concept of a programming oven, a virtual appliance that is unaffected by the simulation and acts as a recording environment. Artists place objects inside the oven to establish a situation, and the oven automatically projects a copy of that situation above it. Every action artists perform on the projected objects is sequentially recorded as the consequence of the behavior specified by the situation inside the oven.

For example, to create a rule that makes cubes move along their up-axes, an artist would:

1. Use the pliers tool to pick up a cube from the environment

2. Drag the cube into the oven, and release it inside

   (a projection of the cube will appear above the oven)

3. Move the pliers close to the projected cube to reveal the movement gizmo
4. Grab the upper arrow on the movement gizmo, move the cube up a bit, and release it

The behavior exists immediately and can be said to have been “baked” into the supplied situation. Even if an artist removes or deletes the objects inside the oven, the behavior will persist.

Viewing Behaviors

A potent criticism of many traditional programming environments is that they fail to “show the data” — programmers must imagine what a program will do or is doing, even though this information should be readily available. Paraspace is by no means perfect in this regard, but it does provide some facilities for viewing the consequences of authored behaviors. Whenever the oven contains any objects, the oven plays a miniature looping simulation of the those objects next to the modifiable projection. This preview gives artists a faster feedback loop while creating behaviors before testing them in the larger environment.

While Paraspace does support individual behavior viewing via the oven, it currently fails to provide a mechanism for viewing historically created behaviors; artists must remember which situations have corresponding consequences to view or edit them. Since Paraspace only supports two types of situations at present, this is less of a problem than it otherwise might be (with three unique object types, artists can only create ten unique situations).

Editing Behaviors

Paraspace’s facilities for editing existing behaviors are quite limited, which is acceptable only because the behaviors tend to be very simple. While artists can easily add additional manipulations to the behavior corresponding to the situation within the oven, their only option for changing earlier manipulations is to reset the consequence associated with the situation inside the oven using the oven’s single ‘X’ button.

Ambiguities, Assumptions, Inferences, and Consequences

Programming by Demonstration systems are prone to ambiguity, and Paraspace is no exception. In general, I tried to choose the most obvious general interpretation of each ambiguous case, but unexpectedly, that wasn’t always possible. Consider the case of a red cube in the oven. The most specific interpretation for the corresponding situation would be: “this specific cube in the current orientation and position,” but that interpretation is clearly over-specified and useless. Conversely, the interpretation “any object in any position and orientation” is likely too general. Instead, I chose the relatively intuitive “any cube” interpretation, which is powerful in its generality without applying to everything.

If the objects in Paraspace had a more diverse set of properties, the choice would no longer be so obvious. For example, if objects could have different colors, should placing a red cube in the oven create a behavior that applies to black cubes? Each option seems equally desirable in different situations; ideally, the artist creating the behavior should be able to specify their desired interpretation. How they could do so is an open question.

Multi-object situations introduce additional ambiguities. When two objects touch in the oven, the obvious interpretation is that the situation should apply whenever those types of objects touch. However, I was unable to determine a meaningful interpretation of two objects present in the oven but not touching. Should that situation imply something about the distance between the objects? The least consistent aspect of Paraspace is that it only understands situations in which the oven contains a single object or two touching objects, and it ignores all other situations.

The consequence of a behavior is seemingly well-specified because it applies to a copy of the behavior’s situation, but recall that a behavior needs to apply to situations that are more general than the specific configuration of objects in the oven. Does moving a projected cylinder upwards in the oven make all affected cylinders in the environment move upwards in world space or in local space? For the sake of greater expressivity, Paraspace interprets all movements in local space. I subtly reinforced this idea throughout Paraspace’s design: moving the pliers close to an object makes a local transform gizmo appear around the object, and placing an object in the oven makes its initial projection appear above the oven with its global rotation removed.

Collaboration

Very few programming environments give any thought to collaborative programming. In almost all cases, collaboration is an afterthought handled through clunky external version control systems that often require manual intervention to combine parts of programs written by different authors. I think mixed reality environments have the potential to dramatically improve collaborative programming. While designing Paraspace, I considered collaboration from the outset. Artists occupy a shared virtual working environment and can see minimalist googley-eyed avatars of each other, inspired by the experiments at Daydream Labs.

In one sense, collaboration in Paraspace is very successful: one artist can freely set up a simulation while another creates behaviors for its components. This situation vastly improves on the iteration time seen in most contemporary programming environments and reduces context switching for each artist.

However, supporting collaboration made the design of many elements in Paraspace much more difficult while resulting in a degraded user experience overall. For example, user interfaces should allow users to easily recover from mistakes, but the standard solution—undo—is poorly defined in a multi-user system. Global undo is the obvious approach but fails to meet user expectations. Since developing robust multi-user undo seems to be case-dependent and equivalent in scope to a thesis project, I chose not to implement an undo system.

I made a similar concession to spatial navigation in Paraspace. While mixed reality tools like Tilt Brush allow users to resize the environment and move themselves through it without moving physically, I chose to maintain a consistent environment scale and to maintain a direct mapping between an artist’s location in physical and virtual space. I made this decision because I wanted the artists to communicate verbally in a shared physical space without colliding with each other, while simultaneously maintaining agency over their movement through the space to combat VR sickness. An obvious alternative would be to have a separate physical space for each artist and to transmit artist locations and communications over the internet. Beyond creating massively more complex technical issues, that approach would have created a host of new design issues related to personal space in mixed reality environments.

In practice, Paraspace’s programming oven metaphor restricts program editing to a single behavior at a time, but in principle, concurrent editing of separate behaviors might be worth exploring. I suspect that it would be less chaotic than concurrent editing in a text-based programming environment.

Open Questions and Future Directions

Paraspace sets the groundwork for a novel alternative method for programming in mixed reality, based in the philosophy of Programming by Demonstration. It introduces a set of consistent direct manipulation tools for manipulating a collaborative virtual environment and establishes a framework for using these tools in the context of a “programming oven” that empowers visual artists to create behaviors for objects in the environment while reducing iteration times.

Nonetheless, many open questions stand in the way of Programming by Demonstration becoming a truly viable approach for creating sophisticated virtual environments in mixed reality. As mentioned earlier, adding more properties to objects would require more sophisticated methods for users to communicate their intent in the case of ambiguities. That ability would then allow the system to support objects with mutable state such as speed or acceleration. It would also remove Paraspace’s constraint on how few types of situations may exist.

In an early version of Paraspace, I allowed artists to create behaviors that took place over longer periods of time than a single frame, but that changed their expectations about whether behaviors should operate in local or global space while raising a host of design issues such as what should happen if an object is involved in a collision while completing a previous behavior. Future explorations might tackle these problems.

For scope reasons, I purposely chose to ignore abstraction, the sequencing of behaviors, and the creation of control structures like loops and conditionals, but a more sophisticated programming environment would need these or equivalent facilities.

Finally, if a Programming by Demonstration system were used to create more interesting and complex mixed reality simulations than the ones possible in Paraspace, it would need more powerful mechanisms for viewing and debugging behaviors. Stagecast Creator has some powerful capabilities that might serve as a useful starting point.