XR is a New Medium ~ Free Your Mind
Use the unique properties of the medium to create ideal experiences
Watch the Full talk above or read it as a blog below.
All Reality Is Virtual…
Our perception of the world is what is real to us. This truth enables XR to exist through tricks of human sensory system. Knowing the true reality helps build ideal Virtual additions…
The reality we experience is a construct in our minds, based on highly incomplete data. It generally matches the real world well […] but it’s not a literal reflection of reality—it’s just an inference of the most probable state of the world, given what we know at any one time.
Michael Abrash, Chief Scientist Meta Reality Labs
VR’s Grand Challenge: Michael Abrash on the Future of Human Interaction, 2017
Language Is A Private Hallucination
Beyond this I want you to contemplate how humans make shared meaning together in the current ways.
When I say spoon you all think of something. But consider, that it’s subtly unique to each of you. My mind’s eye spoon is kinda fancy and silver, what does yours look like? We know what we’re talking about, but we can’t see exactly what each other thinks we know.
If I show you a graphic you might interpret that as meaning the word ‘spoon’. In different languages perhaps. This now gives us a focus object to agree about as ‘spoon’ and we can treat it as if it existed and was an object Yet as pointed out:
We speak about the shared world, and our inner worlds, with sounds, gestures and symbols. The field of Semiotics specifically and Communication more generally deals with this reality.
These Sign Vehicles do not contain that which they refer to. So when we speak to one another we are triggering associations in each mind, yet not able to exactly see what each of us is experiencing. Words refer to private hallucinations, and objects in the world can trigger many different word associations in different people.
XR Is New: A Shared Hallucination
XR is unique, it is an experience mediated through computer rendering. That rendering can become the illusion of various immersive and/or spatialized experiences.
XR engages three core categories: Perception, Embodiment, & Environment Providing immersion and presence for the user, while understanding and incorporating space with the user’s surroundings.
Unlike prior forms it can engage the perception of one or more people in a similar and repeatable way. It is not a media playing on a screen, but an experience happening to people. It is a kind of Shared Hallucination.
We could now have a fully interactive virtual spoon that is no longer an image or model, but an object seemingly in our hands. We could drink virtual soup with our virtual spoon!
As XR designers we get to remember that underneath it all:
There is No Spoon
You think that’s ‘Depth’ you’re perceiving now?
XR achieves depth through an illusion: showing offset images to each eye. This is to say that the the fundamental illusion of XR is in the user’s perception. So as designers we are tasked with working within the constraints of this illusion, and can also use them to our advantage in new ways.
Graphic Credit: Garcia et al
Even seemingly ‘solid' physical world objects can fool our inherited human senses.
It’s all an Illusion So Craft Ideal Illusions
Since XR is an illusion of perception, we are not limited to even simulating a ‘reality’ at all. After 7 years working as a designer for BadVR and Nanome I utilize some high level criteria to orient how I approach designing an XR experience.
The user’s entire body and environment will be involved in your experience in some way. For example if you know your user will be seated more, you have to accommodate for their physical posture and obstacles.
A simple trick to establish how users’ bodies will interact with your experience is to check by drawing out curves with the drawing tool, at maximum arm extension and comfortable arm extension. Choose different colors so you get a baseline of what is easy (green) and what is hard (red). Then use these to check if your UI is within range. Consider adding environment scans in, and doing this in mixed reality.
This will help ground your design overall since you will always be thinking about the physical possibility of doing the gestures/actions you are asking of your user.
Curves seen from back - Red shows the hard to reach area, green shows the close and easy to reach area.
Choose Your Metaphors Deliberately
Craft deliberate illusions. Don’t just try to simulate reality, or stick with ‘the way things are’, make sure you are actively selecting every aspect of your design.
For example - When working on Nanome 2 at my last job we ended up changing a core paradigm of the application based on my realization that we had to change metaphors to best serve our users.
The metaphor of Nanome 1 was that a molecular structure is like a statue that people move around in a virtual space in predetermined tiles on a floor. It enabled users to see structures in a whole new way in a familiar metaphor of the Statue, but it came with tradeoffs. Users experienced mechanics confusion, false positives and issues getting just the right view in multi-user sessions. It also brought network overhead of synchronizing the illusion that the structure was actually solid.
So for Nanome 2 we started off with this same metaphor as a baseline, re-implemented from scratch in a more standard way. It began showing the same issues. Users were spending time teleporting (or falsely teleporting) rather than interacting with the structures.
We tested through multiple rounds of real pharmaceutical customer user groups, both existing Nanome 1 Users and new users. First rounds of testing users appreciated the UI's aesthetic and simplicity but found movement in VR still challenging.
Cleaner and more streamlined.
Still have trouble with moving around the structure and other users
I became emboldened by necessity: what other metaphor could structural data with multiple users take? Well, there is no structure, it’s just data. So what if everyone had their own view? How could they still follow along?
The idea became known as ‘Non-Rivalrous’ structures internally as we worked to flesh it out and turned into what was called ‘Spotlight and Follow’ modes. Each structure can be manipulated freely or shown to others with shared cursors that keep everyone on the same page in any view. Applying this increasingly used paradigm from 2D tools into XR was a challenge but the end result was worth it.
User testing validated our hypothesis and a new paradigm for Nanome was born.
Seeing stuff from each others’ perspective is more helpful.
After trying Nanome 2, I prefer it. Just give me the structure. It gets to the point faster than before.
Spotlight/follow gives collaborators a chance to explore by themselves.
The wrong metaphor(s) in your project can make UX problems impossible to solve. Understand what your goals and user needs are and craft ideal illusions.
Work with reality of XR tech
XR is a fascinatingly rich set of technologies blended together into a coherent experience. Yet each individual piece of technology has its own characteristics to optimize for.
If you try to treat XR interactions as “magic” you will have the medium punish you with unintended side effects. Be deliberate in the usage of these amazing input and outputs modes.
Think of the headset as another perception riding along with your user. It can only see what is in its field of view and interpret what its sensors can make sense of.
When using Hand Tracking avoid having the user touch their hands together or cross one hand over the other, this is unreliable for the system to track. Reflective surfaces will confuse the environment meshing.
Account for realities of the headset and you can create fantastic experiences.
XR Interaction Fidelities
High fidelity interactions are essentially (but not limited to) simulations of the dynamics of reality. High fidelity interactions place an emphasis on immersion and details to create ideally a suspension of disbelief where the user feels they are actually using ‘real’ objects/dynamics. These are the most complex to create and have to work reliably. While immersive if done well can become frustrating if not perfectly executed.
Medium fidelity interactions are a balance between simulating aspects of reality while minimizing the details. Often a useful balance for a task that requires or benefits from user immersion yet does not require suspension of disbelief.
Low fidelity interactions are essentially binary or button press type interactions, with dwell time being a possible addition. These are the simplest and most reliable interactions, yet they have the least immersion.
Deciding on the mixture of interaction fidelities is defined by the requirements of the system, and a user experience may be composed of mixtures of fidelity levels seamlessly based on need / implementation budget.
Read more: Interaction Fidelity: The Uncanny Valley of Virtual Reality Interactions.
Design Thinking
Research and discovery
Concept Exploration
Mechanics Validation
Tip: How Might We…
“How might we” abbreviated HMW is a buffer to keep teams from jumping to solution too early by reframing goals and even later feedback as something more actionable for the design and development team.
So for example say you get two pieces of user feedback that are slightly different and on the surface, contradicting each other. User A says ‘I keep missing this button but I need it why isn’t it green?’ while User B says ‘that button should be blue because it’s the most important one here’.
Instead of addressing each exact user feedback item in isolation, you can record the intent through some statements like:
HMW ensure the user can find buttons they need?
HMW better signal which buttons are most important?
The How Might We statement gives you another lens to look at your goals and feedback without a solution in mind already, facilitating a more open process of solving them.
Make it worth wearing the headset
Make it worth putting on the headset and staying there. Consider the entire user journey of putting headset on and using and taking off, not just them using your app. Think about how your app fits into the headset OS and how that headset fits into your users’ lives.
You are creating your virtual “spoon” for a purpose, not always just to simulate a spoon. XR can achieve things no other medium can. If you know its true nature you can push it farther than anyone has before and maybe even set a new standard.
Free your mind…
Supplemental Resources
Essential talk that inspired me on how to approach ‘native medium’ design.