Comp4010 Lecture10 VR Interface Design

VR INTERFACE DESIGN
COMP 4010 Lecture Ten
Mark Billinghurst
October 12th 2021
mark.billinghurst@unisa.edu.au

VR Input Devices
• Physical devices that convey information into the application
and support interaction in the Virtual Environment

Hand Input Devices
• Devices that integrate hand input into VR
• World-Grounded input devices
• Devices fixed in real world (e.g. joystick)
• Non-Tracked handheld controllers
• Devices held in hand, but not tracked in 3D (e.g. xbox controller)
• Tracked handheld controllers
• Physical device with 6 DOF tracking inside (e.g. Vive controllers)
• Hand-Worn Devices
• Gloves, EMG bands, rings, or devices worn on hand/arm
• Bare Hand Input
• Using technology to recognize natural hand input

Bare Hands
• Using computer vision to track bare hand input
• Creates compelling sense of Presence, natural interaction
• Challenges need to be solved
• Not having sense of touch
• Line of sight required to sensor
• Fatigue from holding hands in front of sensor

Non-Hand Input Devices
• Capturing input from other parts of the body
• Head Tracking
• Use head motion for input
• Eye Tracking
• Largely unexplored for VR
• Face Tracking
• Use for lip syncing
• Microphones
• Audio input, speech
• Full-Body tracking
• Motion capture, body movement

Pedestrian Devices
• Pedestrian input in VR
• Walking/running in VR
• Virtuix Omni
• Special shoes
• http://www.virtuix.com
• Cyberith Virtualizer
• Socks + slippery surface
• http://cyberith.com

Comparison Between Devices
From Jerald (2015)

Typical VR System
• Combining multiple technology elements for good user experience
• Input devices, output modality, content databases, networking, etc.

From Content to User
Modelling
Program
Content
• 3d model
• Textures
Translation
• CAD data
Application
programming
Dynamics
Generator
Input Devices
• Gloves, Mic
• Trackers
Renderers
• 3D, sound
Output Devices
• HMD, audio
• Haptic
User Actions
• Speak
• Grab
Software
Content
User I/O

Types of VR Graphics Content
• Panoramas
• 360 images/video
• Captured 3D content
• Scanned objects/spaces
• Modelled Content
• Hand created 3D models
• Existing 3D assets

VR System Pipeline
• Using time warping and lens distortion

Typical VR Simulation Loop
• User moves head, scene updates, displayed graphics change

What Happens When Senses Don’t Match?
• 20-30% VR users experience motion sickness
• Sensory Conflict Theory
• Visual cues don’t match vestibular cues
• Eyes – “I’m moving!”, Vestibular – “No, you’re not!”

VR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

Sketching VR Interfaces
•Download 360 panorama template grid
•Draw interface ideas into grid
•Scan into 360 photo viewer for VR HMD
See https://virtualrealitypop.com/vr-sketches-56599f99b357

VR Storyboarding - Describing the Experience

Example: VR Storyboard
https://medium.com/cinematicvr/a-storyboard-for-virtual-reality-fa000a9b4497

Gravity Sketch
•Intuitive immersive 3D design platform
•Move from sketch to final 3D model render
•Natural 3D UI manipulation
•Two handed input, 3D menus, etc
•Multi-platform
•HMD (Quest, Steam), tablet, etc
•Support for collaboration

Scene Assembly
•Assemble assets into 3D scene
•Create high-fidelity view
•Collect user feedback
•Immersive Scene Assembly
•Microsoft Maquette: https://www.maquette.ms/
•Sketchbox: https://www.sketchbox3d.com/

Interactive 360 Prototyping for VR
•Create 360 images and add interactive elements
•Many possible tools
•InstaVR
•http://www.instavr.co/
•Free, fast panorama VR
•Drag and drop web interface
•Deploy to multi platforms (Quest, Vive, phone, etc)
•VR Direct
•https://www.vrdirect.com/
•Connect multiple 360 scenes
•Instant content update
•EasyVR
•https://www.360easyvr.com/

VR Visual Programming
• Drag and drop VR development
• Visual Programming for Unity
• VR Easy - http://blog.avrworks.com/
• Key VR functionality (navigation, etc)
• HMD and VR controller support
• Bolt
• Rich visual flow
• Integrated with Unity
• Playmaker - https://hutonggames.com/
• Popular game authoring tool
• Can be combined with VR toolkits

High Level Graphics Tools
• Game Engines
• Powerful, need scripting ability
• Unity, Unreal, Cry Engine, etc..
• Combine with VR plugins
• HMDs, input devices, interaction, assets, etc..

Typical Virtual Reality System
HMD
Input
Tracking
User
Interface

How can we Interact in VR?
• How can VR devices create a natural user experience?

Background: Human-computer interaction
• HCI studies communication
• Users and computers communicate via the interface
• Traditional UI design issues:
• Input device
• Interaction style
• Feedback to the user
• Gulf of execution / gulf of evaluation
• All these are relevant for 3D/VR User Interfaces

Why 3D Interaction?
• 3D / VR application should be useful
• Support immersion
• Use natural skills
• Provide immediacy of visualization
• But many current VR apps either
• Support only simple interaction
• Or, have serious usability problems
• We need good 3D user interface guidelines

Some Definitions
• 3D Interaction:
• Human-computer interaction in which the user’s tasks are
carried out in a 3D spatial context
• 3D input devices, 2D input devices mapping into 3D
• 3D user interface (3D UI):
• A UI that involves 3D interaction
• 3D interaction technique:
• A method (hardware and software) allowing a user to
accomplish a task in a 3D UI
29

Examples of 3D User Interfaces – VR and non-VR
3D physical input, 3D virtual context 3D physical input, 2D virtual context 2D physical input, 3D virtual context

What makes 3D interaction difficult?
• Spatial input
• Lack of constraints
• Lack of standards
• Lack of tools
• Lack of precision
• Fatigue
• Layout more complex
• Perception

Example: Virtual-SAP
https://www.youtube.com/watch?v=Xz_J0EK8LLs

Moving from Menus to Natural Interaction
2D UI in 3D
Natural 3D Interaction

Natural Interface Concept - WorldBuilder
https://www.youtube.com/watch?v=FheQe8rflWQ&t=43s

World Builder Today (Available on Steam)
• https://www.youtube.com/watch?v=65u3W7wjXs0

Vision vs. Reality – Still Work to Do..
Natural interface
Gesture, speech
Wide field of view
Full body input
Limited input
Wireless, limited range tracking
Reduced field of view
2D GUI in VR

Universal 3D Interaction Tasks in VR
• Object Interaction
• Selection: Picking object(s) from a set
• Manipulation: Modifying object properties
• Navigation
• Travel: motor component of viewpoint motion
• Wayfinding: cognitive component; decision-making
• System control
• Issuing a command to change system state or mode

Selection and Manipulation
• Selection:
• specifying one or more objects from a set
• Manipulation:
• modifying object properties
• position, orientation, scale, shape, color, texture, behavior, etc.

Goals of selection
•Indicate action on object
•Query object
•Make object active
•Travel to object location
•Set up manipulation

Selection performance
• Variables affecting user performance
• Object distance from user
• Object (visual) size
• Density of objects in area
• Occluders

Classification of Selection Techniques
• asdf

Selection classification
Selection
Feedback
Object
indication
Indication to
select
graphical
tactile
audio
object touching
pointing
indirect selection
button
gesture
voice

Common Selection Techniques
•Simple virtual hand
•Ray-casting
•Occlusion
•Go-go (arm-extension)

Simple virtual hand technique
• Process
• One-to-one mapping between physical and virtual hands
• Object can be selected by “touching” with virtual hand
• “Natural” mapping
• Limitation:
• Only select objects in hand reach

Ray-casting technique
• “Laser pointer” attached
to virtual hand
• First object intersected by
ray may be selected
• User only needs to
control 2 DOFs
• Proven to perform well
for remote selection
• Variants:
• Cone casting
• Snap-to-object rays

Example Ray Casting
https://www.youtube.com/watch?v=KGC8vFpRDko

Occlusion technique
• Image-plane technique - truly 2D
• Occlude/cover desired object with
selector object (e.g. finger)
• Nearest object along ray from eye
through finger may be selected

Image Plane Interaction
• Pierce, J., Forsberg, A., Conway, M., Hong, S., Zeleznik, R., & Mine, M. (1997).
Image Plane Interaction Techniques in 3D Immersive Environments.
Proceedings of the ACM Symposium on Interactive 3D Graphics, 39-44.

Example
https://www.youtube.com/watch?v=QFXPJcGdhe4

Go-Go Technique
• Arm-extension technique
• Non-linear mapping between physical and virtual hand position
• Local and distant regions (linear < D, non-linear > D)
Poupyrev, I., Billinghurst, M., Weghorst, S., & Ichikawa, T. (1996). The Go-Go Interaction
Technique: Non-linear Mapping for Direct Manipulation in VR. Proceedings of the
ACM Symposium on User Interface Software and Technology, 79-80.

Precise 3D selection techniques
• Increase selection area
• Cone-casting (Liang, 1993)
• Snapping (de Haan, 2005)
• 3D Bubble Cursor (Vanacken, 2007)
• Sphere-casting (Kopper 2011)
• Increase control/display ratio
• PRISM (Frees, 2007)
• ARM (Kopper, 2010)
Not ideal for cluttered
environments (high
density, occlusion)
May require careful
interaction

Cone-Casting
Spotlight
Aperture

Sphere-casting (SQUAD)
• Two phases
• Sphere-casting followed by QUAD-menu selection
• Features
• Multiple low precision selections
• Scales well – at most log4n+1 refinement steps
• Limitations
• Quad-menu phase is done outside spatial context
• Target needs to be unique or selectable among identical ones
Kopper, R., Bacim, F., & Bowman, D. A. (2011). Rapid and accurate 3D selection by progressive
refinement. In 3D User Interfaces (3DUI), 2011 IEEE Symposium on (pp. 67-74). IEEE.

SQUAD Selection
Sphere casting
Quad Selection

PRISM (Frees 2005)
• Change Control/Gain ratio
based on hand speed
• As hand moves slower, scale
down object motion
• As hand moves faster, us 1:1
motion mapping
• Twice the performance for
object docking tasks
Frees, S., & Kessler, G. D. (2005). Precise and rapid interaction through scaled manipulation in
immersive virtual environments. In Virtual Reality, 2005. Proceedings. VR 2005. IEEE (pp. 99-106).

Goals of manipulation
•Object placement
• Design
• Layout
• Grouping
•Tool usage
•Travel

Classification of Manipulation Techniques
• asdfa

Technique Classification by Components
Manipulation
Object Attachment
Object Position
Object Orientation
Feedback
attach to hand
attach to gaze
hand moves to object
object moves to hand
user/object scaling
no control
1-to-N hand to object motion
maintain body-hand relation
other hand mappings
indirect control
no control
1-to-N hand to object rotation
other hand mappings
indirect control
graphical
force/tactile
audio

Common Manipulation Techniques
•Simple virtual hand
•HOMER
•Scaled-world grab
•World-in-miniature

Simple Virtual Hand Manipulation

HOMER technique
Hand-Centered
Object
Manipulation
Extending
Ray-Casting
• Selection: ray-casting
• Manipulate: directly with virtual hand
• Include linear mapping to allow
wider range of placement in depth
Time
Bowman, D., & Hodges, L. (1997). An Evaluation of Techniques for Grabbing and Manipulating
Remote Objects in Immersive Virtual Environments. Proceedings of the ACM Symposium on
Interactive 3D Graphics, 35-38.

Example
https://www.youtube.com/watch?v=V6Fo3iza5cY

Scaled-world Grab Technique
• Often used w/ occlusion
• At selection, scale user up (or world down) so that virtual
hand is actually touching selected object
• User doesn‘t notice a change in the image until he moves
Mine, M., Brooks, F., & Sequin, C. (1997). Moving Objects in Space: Exploiting Proprioception in
Virtual Environment Interaction. Proceedings of ACM SIGGRAPH, 19-26

World-in-miniature (WIM) technique
• “Dollhouse” world held in
user’s hand
• Miniature objects can be
manipulated directly
• Moving miniature objects
affects full-scale objects
• Can also be used for
navigation
Stoakley, R., Conway, M., & Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in
Miniature. Proceedings of CHI: Human Factors in Computing Systems, 265-272, and
Pausch, R., Burnette, T., Brockway, D., & Weiblen, M. (1995). Navigation and Locomotion in
Virtual Worlds via Flight into Hand-Held Miniatures. Proceedings of ACM SIGGRAPH, 399-400.

Voodoo Doll Interaction
• Manipulate miniature objects
• Act on copy of objects
• Actions duplicated on actual object
• Supports action at a distance
• Two handed technique
• One hand sets stationary reference frame
• Second hand manipulates object
Pierce, J. S., Stearns, B. C., & Pausch, R. (1999). Voodoo dolls: seamless interaction at
multiple scales in virtual environments. In Proceedings of the 1999 symposium on Interactive
3D graphics (pp. 141-145). ACM.

Two-Handed Interaction
• Symmetric vs. Asymmetric
• Symmetric: both hands performing same actions
• Asymmetric: both hands performing different actions
• Dominant (D) vs. non-dominant (ND) hand
• Guiard’s principles
• ND hand provides frame of reference
• ND hand used for coarse tasks, D hand for fine-grained tasks
• Manipulation initiated by ND hand
Guiard, Y., "Asymmetric Division of Labor in Human Skilled Bimanual Action:
The Kinematic Chain as a Model," J. Motor Behavior, 19 (4), 1987, pp. 486-517.

Symmetric Bimanual Technique
• iSith (Wyss 2006)
• Using two 6 DOF controllers each ray casting
• Intersection point of two rays determines interaction point
Wyss, H. P., Blach, R., & Bues, M. (2006, March). iSith-Intersection-based spatial interaction
for two hands. In 3D User Interfaces, 2006. 3DUI 2006. IEEE Symposium on (pp. 59-61). IEEE.

Asymmetric Bimanual Technique
• Spindle + Wheel (Cho 2015)
• Two 6 DOF handheld controls
• One dominant, one ND
• Movement one hand relative
to other provides 7 DOF input
Cho, I., & Wartell, Z. (2015). Evaluation of a bimanual simultaneous 7DOF interaction technique in
virtual environments. In 3D User Interfaces, 2015 IEEE Symposium on (pp. 133-136). IEEE.

Demo: Spindle + Wheel 7 DOF Input
https://www.youtube.com/watch?v=nF4egFHyLYM

Design Guidelines for Manipulation
• There is no single best manipulation technique
• Map the interaction technique to the device
• Reduce degrees of freedom when possible
• Use techniques that can help to reduce clutching
• Consider the use of grasp-sensitive object selection
• Use pointing techniques for selection and grasping techniques for manipulation
• Use existing techniques unless there is a large amount of benefit from
designing a new application-specific method
• Consider the trade-off between technique design and environmental design

Navigation
• How we move from place to place within an environment
• The combination of travel with wayfinding
• Wayfinding: cognitive component of navigation
• Travel: motor component of navigation
• Travel without wayfinding: "exploring", "wandering”

Travel
• The motor component of navigation
• Movement between 2 locations, setting the position (and
orientation) of the user’s viewpoint
• The most basic and common VE interaction technique,
used in almost any large-scale VE

Types of Travel
• Exploration
• No explicit goal for the movement
• Search
• Moving to specific target location
• Naïve – target position not known
• Primed – position of target known
• Maneuvering
• Short, precise movements changing viewpoint

Movement Process
• Focusing on user control

Technique classification
• Physical locomotion metaphors
• treadmills, cycles, etc…
• Steering metaphor
• Route planning metaphor
• Target specification metaphor
• Manual manipulation metaphor
• Scaling metaphor

Classification of Travel and Locomotion
Virtual turning Real turning
Virtual
translation
Desktop VEs
Vehicle simulators
CAVE wand
Most HMD systems
Walking in place
Magic Carpet
Real
translation
Stationary cycles
Treadport
Biport
Wide-area tracking
UNIPORT
ODT
Can classify locomotion devices in terms of real vs. virtual travel

Taxonomy of Travel Techniques
• Focusing on
sub-task of
travel
Bowman, D. A., Koller, D.,
& Hodges, L. F. (1997,
March). Travel in immersive
virtual environments: An
evaluation of viewpoint
motion control techniques.
In Virtual Reality Annual
International Symposium,
1997., IEEE 1997 (pp. 45-
52). IEEE.

Gaze Directed Steering
• Move in direction that you are looking
• Very intuitive, natural navigation
• Can be used on simple HMDs (e.g. Google Cardboard)
• But: Can’t look in different direction while moving

Example: Gaze Directed Steering
https://www.youtube.com/watch?v=6iKxser1Wic

TelePortation
• Use controller to select end point
• Usable with 3DOF contoller
• Jump to a fixed point in VR
• Discrete motion can be confusing/cause sickness

Pointing Technique
• A “steering” technique
• Use hand tracker instead of head tracker
• Point in direction you want to go
• Slightly more complex, than gaze-directed steering
• Allows travel and gaze in different directions
• good for relative motion, look one way, move another

Example: VIVE Thrust
https://www.youtube.com/watch?v=JRgCe_8q4vE

Grabbing the Air Technique
• Use hand gestures to move yourself through the world
• Metaphor of pulling a rope
• Often a two-handed technique
• May be implemented using Pinch Gloves
Mapes, D., & Moshell, J. (1995). A Two-Handed Interface for Object Manipulation in
Virtual Environments. Presence: Teleoperators and Virtual Environments, 4(4), 403-416.

Moving Your Own Body
• Can move your own body
• In World in Miniature, or map view
• Grab avatar and move to desired point
• Immediate teleportation to new position in VE
Moving avatar in Map View Moving avatar in WIM view

Example: Navigation Using WIM
https://www.youtube.com/watch?v=VxGqIjMlTs8

Walking in Place
https://www.youtube.com/watch?v=J_yQfW1qYGI

Redirected Walking
• Address problem of limited
walking space
• Warp VR graphics view of
space
• Create illusion of walking
straight, while walking in circles
Razzaque, S., Kohn, Z., & Whitton, M. C. (2001, September). Redirected walking.
In Proceedings of EUROGRAPHICS (Vol. 9, pp. 105-106).

Redirected Walking
https://www.youtube.com/watch?v=KVQBRkAq6OY

Redirected Walking with Walls
https://www.youtube.com/watch?v=u8pw81VbMUU

Guided Navigation Technique
• Water skiing metaphor for VR movement
• Good for moving in a fixed direction, while giving user some
control

Wayfinding
• The means of
• determining (and maintaining) awareness of where one is located (in
space and time),
• and ascertaining a path through the environment to the desired
destination
• Problem: 6DOF makes wayfinding hard
• human beings have different abilities to orient themselves in an
environment, extra freedom can disorient people easily
• Purposes of wayfinding tasks in virtual environments
• Transferring spatial knowledge to the real world
• Navigation through complex environments in support of other tasks

Wayfinding – Making Cognitive Maps
• Goal of Wayfinding is to build Mental Model (Cognitive Map)
• Types of spatial knowledge in a mental model
• landmark knowledge
• procedural knowledge (sequence of actions required to follow a path)
• map-like (topological) knowledge
• Creating a mental model
• systematic study of a map
• exploration of the real space
• exploration of a copy of the real space
• Problem: Sometimes perceptual judgments are incorrect
within a virtual environment
• e.g. users wearing a HMD often underestimate dimensions of space,
possibly caused by limited field of view

Wayfinding as a Decision Making Process

Kevin Lynch – The Image of the City
• In real cities, five elements
• Path, Edge, District, Node, Landmark
• VR environments the same

Designing VE to Support Wayfinding
• Provide Landmarks
• Any obvious, distinct and non-mobile
object can serve as a landmark
• A good landmark can be seen from
several locations (e.g. tall)
• Audio beacons can also serve as
landmarks
• Use Maps
• Copy real world maps
• Ego-centric vs. Exocentric map cues
• World in Miniature
• Map based navigation

Wayfinding Aids
• Path following
• Easy method of wayfinding
• Multiple paths through a single space may be denoted by colors
• For example, hospitals that use colored lines to indicate how to get to
certain locations.
• Bread crumbs (leaving a trail)
• leaving a trail of markers - like Hänsel and Gretel
• allows participant to know when they've been somewhere before
• having too many markers can make the space be overly cluttered
• Compass
• may also be other form of direction indicator (e.g. artificial horizon)
• may specify directions in 2D space or 3D space

Design Guidelines for Navigation
• Match the travel technique to the application
• Use an appropriate combination of travel technique,
display devices, and input devices
• The most common travel tasks should require a minimum
of effort from the user
• Use physical locomotion technique if user exertion or
naturalism is required
• Use target-based techniques for goal-oriented travel and
steering techniques for exploration and search
• Provide multiple travel techniques to support different
travel tasks in the same application
• Choose travel techniques that can be easily integrated
with other interaction techniques in the application

System Control
• Issuing a command to change system state or mode
• Examples
• Launching application
• Changing system settings
• Opening a file
• Etc.
• Key points
• Make commands visible to user
• Support easy selection

Example: GearVR Interface
• 2D Interface in 3D Environment
• Head pointing and click to select

TULIP Menu
• Menu items attached to virtual finger tips
• Ideal for pinch glove interaction
• Use one finger to select menu option from another
Bowman, D. A., & Wingrave, C. A. (2001, March). Design and evaluation of menu systems for
immersive virtual environments. In Virtual Reality, 2001. Proceedings. IEEE (pp. 149-156). IEEE.

2D Menus in VR
• Many examples of 2D GUI and floating menus in VR
Nested Pie Menu
2D Menu in VR CAVE

Example: Marking Menu in VR
https://www.youtube.com/watch?v=BTTBgZ94lAc

Tools
• Use tools for system commands
• Tangible user interfaces (real tools)
• Virtual tools (3D objects)
• Design issues
• Support eyes-off use
• Use of physical affordances
• Base on familiar objects
• Provide tactile feedback
• Map real tool to virtual operation
Tangible interface for CAVE

Voice Input
• Implementation
• Wide range of speech recognition engines available
• E.g. Unity speech recognition plug-in, IBM VR speech sandbox
• Factors to consider
• Recognition rate, background noise, speaker dependent/independent
• Design Issues
• Voice interface invisible to user
• no UI affordances, overview of functions available
• Need to disambiguate system commands from user conversation
• Use push to talk or keywords
• Limited commands – use speech recognition
• Complex application – use conversational/dialogue system

Example – IBM VR Speech Sandbox
https://www.youtube.com/watch?v=NoO2R3Pz5Go
• Available from: http://ibm.biz/vr-speech-sandbox

Design Guidelines for System Control
• Avoid mode errors
• Design for discoverability
• Consider using multimodal input
• Use an appropriate spatial reference frame
• Prevent unnecessary focus and context switching
• Avoid disturbing the flow of action of an interaction task
• Structure the functions in an application and guide the user
• 3D is not always the best solution – consider hybrid interfaces

119
Conclusions
lUsability one of the most crucial issues facing VE applications
lImplementation details critical to ensure usability
lEase of coding not equal to ease of use
lSimply adapting 2D interfaces is not sufficient

Conclusions
• User interface key for good VR experience
• Need 3D user interface techniques
• Design for
• Selection/Manipulation
• Navigation
• System control
• Follow good design guidelines
• Cannot just implement 2D techniques in VR

Resources
• Excellent book
• 3D User Interfaces: Theory and Practice
• Doug Bowman, Ernst Kruijff, Joseph, LaViola, Ivan Poupyrev
• Great Website
• http://www.uxofvr.com/
• 3D UI research at Virginia Tech.
• research.cs.vt.edu/3di/

UX of VR Website - www.uxofvr.com
• Many examples of great interaction techniques
• Videos, books, articles, slides, code, etc..

Acknowledgments – Content From
• Doug Bowman, Virginia Tech
• Joe LaViola, University of Central Florida
• Ernst Kruijff, Graz Univ. of Technology
• Ivan Poupyrev, Google
Doug Bowman

www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au

INTERACTION DESIGN FOR
VIRTUAL REALITY

How Can we Design Useful VR?
• Designing VR experiences that meet real needs

What is Interaction Design ?
Designing interactive products to
support people in their everyday
and working lives”
Preece, J., (2002). Interaction
Design
• Interaction Design is the design of
user experience with technology

Bill Verplank on Interaction Design
https://www.youtube.com/watch?v=Gk6XAmALOWI

• Interaction Design involves answering three questions:
• What do you do? - How do you affect the world?
• What do you feel? – What do you sense of the world?
• What do you know? – What do you learn?
Bill Verplank

The Interaction Design Process
Evaluate
(Re)Design
Identify needs/
establish
requirements
Build an
interactive
version
Final Product
Develop alternative prototypes/concepts and compare them
And iterate, iterate, iterate....

Interaction Design Process
Evaluate
(Re)Design
Identify needs/
establish
requirements
Build an
interactive
version
Final Product
Develop alternative prototypes/concepts and compare them
And iterate, iterate, iterate....

NeedsAnalysis Goals
1. Create a deep understanding of
the user and problem space
2. Understand howVR can help
address the user needs

Key Questions
1. Who is the user?
• Different types of users
2. What are the user needs?
• Understand the user, look for insights
3. Can VR address those needs?
• VR cannot solve all problems

Who are the Users?
• Different types of users, must consider them all
• Primary: people regularly using the VR system
• Secondary: people providing tech support/developing system
• Tertiary: people providing funding/space for VR system

Methods for Identifying User Needs
Learn
from
people
Learn
from
analogous
settings
Learn
from
Experts
Immersive
yourself in
context

1. Learn from People
• Learn from target users by:
• Questionnaires and interviewing
• Running focus groups
• Observing people performing target tasks

2. Learn from Experts
• Experts have in-depth knowledge about topic
• Can give large amount of information in short time
• Look for existing process/problem documentation
• Choose participants with domain expertise
• Expertise, radical opinion, etc.

3. Immersive yourself in Context
• Put yourself in the position of the user
• Role playing, a day in the life of a user, cultural probes
• Observing the problem space around you – how do you feel?
• Take notes and capture your observations
A day in the Life of.. Cultural Probes.. Role Playing..

4. Seek Inspiration in Analogous Setting
• Inspiration in different context than problem space
• E.g. redesign library by going to Apple store
• Think of Analogies that connect with challenge
• Similar scenarios in different places
What can public libraries learn from Apple stores?

Identifying User Needs
• From understanding the user, look for needs
• Human emotional or physical necessities.
• Needs help define your design
• Needs are Verbs not Nouns
• Verbs - (activities and desires)
• Nouns (solutions)
• Identify needs from the user traits you noted, or
from contradictions between information
• disconnect between what user says and what user does..

Example: VR for Arachnophobia
• True story:
• Mark’s father, Alan, didn’t seem afraid of anything
• He went to the HIT Lab to try VR for the first time
• In a virtual kitchen he saw a VR spider and screamed
• Contradiction:
• Afraid of nothing, but screams at virtual spider

Example: VR for Arachnophobia
State the Problem
- [User] needs [verb phrase] in a way that [way]
- How might we [verb phrase] ?
Example
- Alan needs to overcome his fear of spiders in a way that
that is easy and painless
- How might we help him overcome his fear of spiders ?
User Need

Is VR the Best Solution?
• Not every problem can be solved by VR..
• Problems Ideal for Virtual Reality, have:
• visual elements
• 3D spatial interaction
• physical manipulation
• procedural learning
• Problems Not ideal for Virtual Reality, have:
• heavy reading, text editing
• many non-visual elements
• need for connection with real world
• need for tactile, haptic, olfaction feedback

The Interaction Design Process
Evaluate
(Re)Design
Identify needs/
establish
requirements
Build an
interactive
version
Final Product

Idea Generation
• Once user need is found, solutions can be proposed
• Idea generation through:
• Brainstorming
• Lateral thinking
• Ideal storming
• Formal problem solving
• Etc..

Example:
• Ideas for overcoming fear of spiders
• Watching spider videos
• Exposure to real spiders
• Using toy spiders
• Virtual Reality therapy
• Augmented Reality spider viewing

VR for Spider Phobia
https://www.youtube.com/watch?v=HNLwvNapUA4

Elaboration and Reduction
• Elaborate on Ideas and Reduce to Final Design Direction
• Elaborate - generate solutions.These are the opportunities
• Reduce - decide on the ones worth pursuing
• Repeat - elaborate and reduce again on those solutions

Use Interface Metaphors
• Design interface object to be similar to familiar
physical object that the user knows how to use
• E.g. Desktop metaphor, spreadsheet, calculator
• Benefits
• Makes learning interface easier and more accessible
• Users understand underlying conceptual model

Typical VR Interface Metaphors
• Direct Manipulation
• Reach out and directly grab objects
• Ray Casting
• Select objects through ray from head/hand
• Vehicle Movement
• Move through VR environment through vehicle movement

Example: Handle Bar Metaphor
• https://www.youtube.com/watch?v=VBCP63jD3OI

Affordances
”… the perceived and actual properties of the thing,
primarily those fundamental properties that determine just
how the thing could possibly be used. [...]
Affordances provide strong clues to the operations of
things.”
(Norman, The Psychology of Everyday Things 1988, p.9)

Perceived vs. Actual Affordances
• Perceived affordance should match actual affordance

Affordances in VR
• Design interface objects to show how they are used
• Use visual cues to show possible affordances
• Perceived affordances should match actual affordances
• Good cognitive model - map object behavior to expected
Familiar objects in Job Simulator Object shape shows how to pick up

Examples of Affordances in VR
Virtual buttons can be pushed Virtual doors can be walked through
Virtual objects can be picked up
Flying like a bird in Birdly

UX Guidelines for VR
• The Four Cores of UX Design for VR
• Make interface Interactive and Reactive
• Design for Comfort and Ease
• Use usable Text and Image Scale
• Include position audio and 3D sound
https://www.dtelepathy.com/blog/philosophy/ux-guide-designing-virtual-reality-experiences

UX Challenges
• Problems to be Addressed
• Keep the user safe
• Make it look and feel real
• Make sure users don’t get simulation sickness
• Develop easy-to-use controls and menus

Cardboard Design Lab
• Mobile VR App providing examples of best practice VR
designs and user interaction (iOS, Play app stores)

Demo: Cardboard Design Lab
https://www.youtube.com/watch?v=2Uf-ru2Ndvc

VR Human Interface Guidelines
• Interface design website - http://vrhig.com/
• Set of VR interface design best practices

Design Guidelines (from 3D UI book)
• Design for comfortable poses
• Design for relatively short sessions and encourage breaks
• Use constraints, use and invent magical techniques
• Consider real world tools and practices as a source of inspiration for 3D user
interface design
• Consider designing 3D techniques using principles from 2D interaction
• Consider using physical props and passive feedback, particularly in highly
specialized tasks
• Ensure temporal and spatial compliance between feedback dimensions

More VR Design Guidelines
• Use real-world cues when appropriate.
• If there is a horizon line, keep it steady
• Be careful about mixing 2D GUI and 3D
• Avoid rapid movement, it makes people sick
• Avoid rapid or abrupt transitions to the world space
• Keep the density of information and objects on screen low
• Do not require the user to move their head or body too much
From https://www.wired.com/2015/04/how-to-design-for-virtual-reality/

Interaction Design Process
Evaluate
(Re)Design
Identify needs/
establish
requirements
Build an
interactive
version
Final Product

NASA Hololens AR/VR Concept Demo
• Vision: Work on Mars from your office
• Story and sketches based on vision
• Led to working Demo

Final NASA HoloLens OnSight
Demo

https://www.youtube.com/watch?v=o-GP3Kx6-CE

What is Evaluation?
•Evaluation is concerned with
gathering data about the usability
of a design or product by a
specified group of users for a
particular activity within a specified
environment or work context

When to evaluate?
• Once the product has been developed
• pros : rapid development, small evaluation cost
• cons : rectifying problems
• During design and development
• pros : find and rectify problems early
• cons : higher evaluation cost, longer development
design implementation evaluation
redesign &
reimplementation
design implementation

Four Evaluation Paradigms
•‘quick and dirty’
•usability testing (lab studies)
•field studies
•predictive evaluation

Quick and Dirty
• ‘quick & dirty’ evaluation: informal feedback from
users or consultants to confirm that their ideas are
in-line with users’ needs and are liked.
• Quick & dirty evaluations are done any time.
• Emphasis is on fast input to the design process
rather than carefully documented findings.

Usability Testing
• Recording typical users’ performance on typical
tasks in controlled settings.
• As the users perform tasks they are watched &
recorded on video & their inputs are logged.
• User data is used to calculate performance times,
errors & help determine system usability
• User satisfaction questionnaires & interviews are
used to elicit users’ opinions.

Laboratory-based studies
• Laboratory-based studies
• can be used for evaluating the design, or system
• are carried out in an interruption-free usability lab
• can accurately record some work situations
• some studies are only possible in a lab environment
• some tasks can be adequately performed in a lab
• useful for comparing different designs in a controlled context

Laboratory-based studies
Controlled, instrumented environment

Field/Ethnographic Studies
• Field studies are done in natural settings
• The aim is to understand what users do naturally
and how technology impacts them.
• In product design field studies can be used to:
- identify opportunities for new technology
- determine design requirements
- decide how to introduce new technology
- evaluate technology in use.

Predictive Evaluation
• Experts apply their knowledge of typical
users, often guided by heuristics, to
predict usability problems.
• Can involve theoretically based models.
• A key feature of predictive evaluation is
that users need not be present
• Relatively quick and inexpensive

Characteristics of Approaches
Usability
testing
Field
studies
Predictive
Users do task natural not involved
Location controlled natural anywhere
When prototype early prototype
Data quantitative qualitative problems
Feed back measures &
errors
descriptions problems
Type applied naturalistic expert

Pilot Studies
• A small trial run of the main study.
• Can identify majority of issues with interface design
• Pilot studies check:
- that the evaluation plan is viable
- you can conduct the procedure
- that interview scripts, questionnaires,
experiments, etc. work appropriately
• Iron out problems before doing the main study.

Controlled Experiments
• Designer of a controlled experiment should
carefully consider:
• proposed hypothesis
• selected subjects
• measured variables
• experimental methods
• data collection
• data analysis

Subjects
• The choice of subjects is critical to the validity of the
results of an experiment
• subjects group should represent expected user population
expected user population
• Consider subject factors such as:
• age group, education, skills, culture, technology background
• The sample size should be large enough (10+) to be
statistically representative of the user population

Hypothesis and Variables
• Hypothesis: prediction of the experiment outcome
• Experiments manipulate and measure variables
under controlled conditions
• There are two types of variables
• independent: variables that are manipulated to create
different experimental conditions
• e.g. number of items in menus, colour of the icons
• dependent: variables that are measured to find out the
effects of changing the independent variables
• e.g. speed of menu selection, speed of locating icons

Experimental Methods
• It is important to select the right experimental method so that the results of the
experiment can be generalized
• There are mainly two experimental methods
• between-groups: each subject is assigned to one experimental condition
• within-groups: each subject performs under all
the different conditions

Experimental Methods
Randomly
assigned
Statistical data analysis
Experimental
task
Condition
2
Condition
3
Condition
1
Subjects
data data data
Between-
groups
Randomly
assigned
Statistical data analysis
Subjects
data data data
Within-
groups
Experimental
tasks
Condition
2
Condition
3
Condition
1
Experimental
tasks
Condition
1
Condition
3
Condition
2
Experimental
tasks
Condition
1
Condition
2
Condition
3

Data Collection and Analysis
• The choice of a method is dependent on the type of data that
needs to be collected
• In order to test a hypothesis the data has to be analysed using a
statistical method
• The choice of a statistical method depends on the type of collected
data
• All the decisions about an experiment should be made before the
experiment is carried out

Data Types
• Subjective (Qualitative)
• Subjective survey
• Likert Scale, condition rankings
• Observations
• Think Aloud
• Interview responses
• Objective (Quantitative)
• Performance measures
• Time, accuracy, errors
• Process measures
• Video/audio analysis
How easy was the task
1 2 3 4 5
Not very easy Very easy

Example: VR Navigation using Head Tilt
• CHI 2017 paper from Tregillus, Al Zayer, and Folmer
• Problem
• Navigation in mobile VR difficult due to limited input options
• Solution
• Use head tilt to provide simulated joystick input
Tregillus, S., Al Zayer, M., & Folmer, E. (2017, May). Handsfree Omnidirectional
VR Navigation using Head Tilt. In Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems (pp. 4063-4068). ACM.

Implementation
• Calculate head tilt angle
• Difference between vertical head vector and gravity vector
• Once head tilt is greater than threshold, move forward
• However using head tilt alone prevents looking around
• Head tilt navigation triggered when walking detected (from IMU)
• Implemented in Unity and Google Cardboard SDK/Viewer

Demo Video
https://www.youtube.com/watch?v=e6fyfnVTAYs

User Study
• Goal: To compare head tilt input to joystick input
for navigation in mobile VR
• Conditions
• TILT: Head tilt input only
• WIP-TILT: Head tilt + using walking to trigger tilt input
• Joystick: Joystick input
• Measures
• Quantitative: Performance time, Number of obstacles
hit
• Qualitative: Simulator sickness (SSQ), user
preferences

Experiment Design
• 25 Subjects (6 female, 19 male)
• Within subjects design
• All subjects do all conditions
• Experience conditions in counterbalanced order
• For each condition
• Training then navigate through 5 virtual corridors
• At end of condition take SSQ survey
• Rate condition on Likert scale for efficiency, accuracy, etc.
• After all conditions
• Interview subjects for more feedback

Results: Performance, Sickness
• Performance time, Obstacles hit, SSQ sickness scores
• Use one way ANOVA test for significance between conditions
• TILT significantly faster and more accurate than WIP-TILT, joystick
• No significant difference between sickness scores
• Using p < 0.05 significance

Results: User Preference
• One way ANOVA comparing Likert scores (1 – 7)
• significant diff. between TILT and WIP-TILT for efficiency, learnability,
errors, likeability and immersion
• significant diff. between TILT and joystick for learnability and immersion

Discussion
• TILT
• Performed fastest because user didn’t need to walk in place
• Liked condition best, except for immersion
• TILT not ideal for VR applications where user needs to look around
• WIP-TILT
• Slower than TILT, more difficult to learn due to walking
• User felt most immersive due to proprioceptive input
• Shows that head tilt could be viable input for mobile VR

Lessons Learned About Expt. Design
• Decide on type of experiment
• Within subject vs. between subject
• Have well designed task with measurable outcomes
• Use both qualitative and quantitative measures
• Performance + user preference
• Have enough subjects for significant results
• Use the appropriate statistics
• Compare conditions + perform post hoc analysis
• Provide subject training on task
• Observe user behavior and interview subjects

Conclusion
• Interaction Design methods can be used to develop effective Virtual Realty
interfaces
• Needs Analysis
• Several methods available for determining user needs
• Design
• Use metaphors and affordances, good UI guidelines
• Prototyping
• Many rapid prototyping tools available
• Evaluation
• Use multiple methods for best evaluation

Comp4010 Lecture10 VR Interface Design

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Comp4010 Lecture10 VR Interface Design

Similar to Comp4010 Lecture10 VR Interface Design (20)

More from Mark Billinghurst

More from Mark Billinghurst (14)

Recently uploaded

Recently uploaded (20)

Comp4010 Lecture10 VR Interface Design