Lecture 7 from the COMP 4010 class on AR and VR. This lecture was about Designing AR systems. It was taught on September 7th 2021 by Mark Billinghurst from the University of South Australia.

1. DESIGNING AR SYSTEMS
COMP 4010 Lecture Seven
Mark Billinghurst
September 7th 2021
mark.billinghurst@unisa.edu.au

2. REVIEW

3. XR Prototyping Tools
Low Fidelity (Concept, visual design)
• Sketching
• Photoshop
• PowerPoint
• Video
High Fidelity (Interaction, experience design)
• Interactive sketching
• Desktop & on-device authoring
• Immersive authoring & visual scripting
• XR development toolkits

4. XR Prototyping Techniques
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills
Sketching
Paper Prototyping
Video Prototyping
Wireframing
Bodystorming
Wizard of Oz

5. Interactive Sketching
•Pop App
● Pop - https://marvelapp.com/pop
● Combining sketching and interactivity on mobiles
● Take pictures of sketches, link pictures together

6. Proto.io
• Web based prototyping tool
• Visual drag and drop interface
• Rich transitions
• Scroll, swipe, buttons, etc
• Deploy on device
• mobile, PC, browser
• Ideal for mobile interfaces
• iOS, Android template
• For low and high fidelity prototypes

7. Digital Authoring Tools for AR
Vuforia Studio
Lens Studio
• Support visual authoring of marker-
based and/or marker-less AR apps
• Provide default markers and support
for custom markers
• Typically enable AR previews
through emulator but need to deploy
to AR device for testing

8. Zappar
• Zapworks Studio
• Code-free interactivity
• Desktop authoring for mobile AR
• Integrated computer vision (ARkit, ARCore)
• Scripting, visual programming
• Multiple publishing options
• Zappar App, WebAR, App enabled
• Zapbox
• Inexpensive mobile AR HMD solution
• Two handed input
ZapBox

9. Snap LensStudio - https://lensstudio.snapchat.com/
Author and preview AR prototypes
● Tool behind Snapchat Lenses, but also a powerful AR prototyping tool
● Can do face (using front camera) and world lenses (rear camera)
● Simulated previews using webcam
Deploy and use advanced AR features
● Can deploy to phone running Snapchat app via Snapcode
● Has advanced AR tracking and segmentation capabilities

10. Immersive Authoring Tools for AR
• Enable visual authoring of 3D
content in AR
• Make it possible to edit while
previewing AR experience in the
environment
• Provide basic support for interactive
behaviors
• Sometimes support export to
WebXR
Apple Reality Composer
Adobe Aero

11. Creating On Device
•Adobe Aero
•Create AR on mobile devices
•Touch based interaction and authoring
•Only iOS support for now
•https://www.adobe.com/nz/products/aero.html

12. Apple Reality Composer
• Rapidly create 3D scenes and AR experiences
• Creation on device (iPhone, iPad)
• Drag and drop interface
• Loading 2D/3D content
• Simple interactivity – trigger/action
• Anchor content in real world (AR view)
• Planes (vertical, horizontal), faces, images

13. Digital Prototyping
Lo-
Fi
Hi-
Fi
Easy
Hard
Digital
Authoring
Immersive
Authoring
Web-Based
Development*
Cross-Platform
Development*
Native
Development*
* requires scripting and 3D programming skills

14. XR Tools Landscape
Digital & Immersive Authoring
Proto.io, Tour Creator, ...
Tilt Brush, Blocks, Quill, …
Web-Based Development
THREE.js, Babylon.js, …
A-Frame, AR.js, …
Cross-Platform Development
Unity + SDKs
Unreal + SDKs
Native Development
Cardboard/Oculus/Vive/... SDK
Vuforia/ARCore/ARKit/… SDK

15. XR Tools Landscape
Digital & Immersive Authoring
Good for storyboarding but limited
support for interactions
Web-Based Development
Good for basic XR apps but often
interactions feel unfinished
Native Development
Good for full-fledged XR apps but
limited to a particular platform
Cross-Platform Development
Good for full-fledged XR apps but
usually high learning curve

16. XR Toolkits
Card-
board
AR
Kit
AR
Core
Oculus VIVE
Holo
Lens
WMR
Web
Cam
A-Frame
AR.js
SteamVR
MRTK
Vuforia
AR Foundation
XR Interaction
WebXR

17. WebXR: A-Frame
• Based on Three.js and WebGL
• New HTML tags for 3D scenes
• A-Frame Inspector (not editor)
• Asset management (img, video,
audio, & 3D models)
• ECS architecture with many open-
source components
• Cross-platform XR

18. AR.js – WebXR Tracking
• Web based AR tracking library
• Marker tracking: ARToolkit markers
• Image tracking: Natural feature tracking
• Location tracking: GPS and compass
• Key Features
• Very Fast : It runs efficiently even on phones
• Web-based : It is a pure web solution, so no installation required.
• Full javascript based on three.js + A-Frame + jsartoolkit5
• Open Source : It is completely open source and free of charge!
• Standards : It works on any phone with webgl and webrtc
• See https://ar-js-org.github.io/AR.js-Docs/

19. Unity – unity.com
• Started out as game engine
• Has integrated support for many
types of XR apps
• Powerful scene editor
• Asset management & store
• Basically all XR device vendors
provide Unity SDKs

20. Vuforia
• Highly optimized computer vision tracking
• Multiple types of tracking
• Image tracking, object tracking, model tracking, area tracking, etc.
• Interaction features
• Virtual buttons, occlusion, visual effects,
• Multi-platform
• Mobile AR, AR headsets See https://www.vuforia.com/

21. AR Foundation
• A unified Framework for AR
• Multi-platform API
• Includes core features from ARKit, ARCore, Magic Leap, and HoloLens
• Set of behaviours and API with following features
• Tracking, light estimation, occlusion, meshing , video pass-through, etc.
• Integrates with Unity MARS
• See https://unity.com/unity/features/arfoundation

22. Unity XR Interation ToolKit (preview package)
• Easy way to add interactivity to AR/VR experience
• Object interactions
• UI interactions
• Locomotion
• Enabling common interactions without writing code
• AR gesture, object placement, annotations
• https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@1.0/

23. Unity MARS
• Features
• Visually author AR apps (WYSIWYG)
• Test apps in Unity editor
• Develop apps that can interact with real world
• Intelligent real-world recognition
• Multi-platform development
• Based on ARFoundation
• ARKit, ARCore, Magic Leap and Hololens
• See unity.com/mars

24. Mixed Reality ToolKit (MRTK)
• Open-Source Mixed Reality ToolKit
• Set of Unity modules/Unreal plugin
• Interaction Models
• Controllers, gesture, gaze, voice, etc.
• UX elements
• Foundational elements
• Material, text, light, etc.
• Controls and behaviours
• button, menu, slider, etc.
• Tutorials, documentation, guidelines
• See https://github.com/microsoft/MixedRealityToolkit-Unity

25. DESIGNING AR SYSTEMS

26. Design in Interaction Design
Key Prototyping
Steps

27. Good vs. Bad AR Design

28. https://www.youtube.com/watch?v=YJg02ivYzSs

29. AR. Design Considerations
• 1. Design for Humans
• Use Human Information Processing model
• 2. Design for Different User Groups
• Different users may have unique needs
• 3. Design for the Whole User
• Social, cultural, emotional, physical cognitive
• 4. Use UI Best Practices
• Adapt known UI guidelines to AR/VR
• 5. Use of Interface Metaphors/Affordances
• Decide best metaphor for AR/VR application

30. 1. Design for Human Information Processing
• High level staged model from Wickens and Carswell (1997)
• Relates perception, cognition, and physical ergonomics
Perception Cognition Ergonomics

31. Design for Perception
• Need to understand perception to design AR
• Visual perception
• Many types of visual cues (stereo, oculomotor, etc.)
• Auditory system
• Binaural cues, vestibular cues
• Somatosensory
• Haptic, tactile, kinesthetic, proprioceptive cues
• Chemical Sensing System
• Taste and smell

32. Depth Perception Problems
• Without proper depth cues AR interfaces look unreal

33. Which of these POI are near or far?

34. Types of Depth Cues

35. Improving Depth Perception
Cutaways
Occlusion
Shadows

36. Cutaway Example
• Providing depth perception cues for AR
https://www.youtube.com/watch?v=2mXRO48w_E4

37. Design for Cognition
• Design for Working and Long-term memory
• Working memory
• Short term storage, Limited storage (~5-9 items)
• Long term memory
• Memory recall trigger by associative cues
• Situational Awareness
• Model of current state of user’s environment
• Used for wayfinding, object interaction, spatial awareness, etc..
• Provide cognitive cues to help with situational awareness
• Landmarks, procedural cues, map knowledge
• Support both ego-centric and exo-centric views

38. Micro-Interactions
▪ Using mobile phones people split their attention
between the display and the real world

39. Time Looking at Screen
Oulasvirta, A. (2005). The fragmentation of attention in mobile
interaction, and what to do with it. interactions, 12(6), 16-18.

40. Dividing Attention to World
• Number of times looking away from mobile screen

41. Design for Micro Interactions
▪ Design interaction for less than a few seconds
• Tiny bursts of interaction
• One task per interaction
• One input per interaction
▪ Benefits
• Use limited input
• Minimize interruptions
• Reduce attention fragmentation

42. NHTSA Guidelines - www.nhtsa.gov
For technology in cars:
• Any task by a driver should be interruptible at any time.
• The driver should control the pace of task interactions.
• Tasks should be completed with glances away from the
roadway of 2 seconds or less
• Cumulative time glancing away from the road <=12 secs.

43. Make it Glanceable
• Seek to rigorously reduce information density. Successful designs afford for
recognition, not reading.
Bad Good

44. Reduce Information Chunks
You are designing for recognition, not reading. Reducing the total # of information
chunks will greatly increase the glanceability of your design.
1
2
3
1
2
3
4
5 (6)
Eye movements
For 1: 1-2 460ms
For 2: 1 230ms
For 3: 1 230ms
~920ms
Eye movements
For 1: 1 230ms
For 2: 1 230ms
For 3: 1 230ms
For 4: 3 690ms
For 5: 2 460ms
~1,840ms

45. Ego-centric and Exo-centric views
• Combining ego-centric and exo-centric cue for better situational awareness

46. Cognitive Issues in Mobile AR
• Information Presentation
• Amount, Representation, Placement, View combination
• Physical Interaction
• Navigation, Direct manipulation, Content creation
• Shared Experience
• Social context, Bodily Configuration, Artifact manipulation, Display space
Li, N., & Duh, H. B. L. (2013). Cognitive issues in mobile augmented reality: an embodied perspective.
In Human factors in augmented reality environments (pp. 109-135). Springer, New York, NY.

47. Information Presentation
• Consider
• The amount of information
• Clutter, complexity
• The representation of information
• Navigation cues, POI representation
• The placement of information
• Head, body, world stabilized
• Using view combinations
• Multiple views

48. Example: Twitter 360
• www.twitter-360.com
• iPhone application
• See geo-located tweets in real world
• Twitter.com supports geo tagging

49. But: Information Clutter from Many Tweets
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah

50. Solution: Information Filtering

51. Information Filtering
Before After

52. OutdoorAR:Limited FOV

53. • Show POI outside FOV
• Zooms between map and panorama views
Zooming Views

54. • https://www.youtube.com/watch?v=JLxLH9Cya20

55. Design for Physical Ergonomics
• Design for the human motion range
• Consider human comfort and natural posture
• Design for hand input
• Coarse and fine scale motions, gripping and grasping
• Avoid “Gorilla arm syndrome” from holding arm pose

56. Gorilla Arm in AR
• Design interface to reduce mid-air gestures

57. XRgonomics
• Uses physiological model to calculate ergonomic interaction cost
• Difficulty of reaching points around the user
• Customizable for different users
• Programmable API, Hololens demonstrator
• GitHub Repository
• https://github.com/joaobelo92/xrgonomics
Evangelista Belo, J. M., Feit, A. M., Feuchtner, T., & Grønbæk, K. (2021, May). XRgonomics: Facilitating the Creation of
Ergonomic 3D Interfaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11).

58. XRgonomics
https://www.youtube.com/watch?v=cQW9jfVXf4g

59. 2. Designing for Different User Groups
• Design for Difference Ages
• Children require different interface design than adults
• Older uses have different needs than younger
• Prior Experience with AR systems
• Familiar with HMDs, AR input devices
• People with Different Physical Characteristics
• Height and arm reach, handedness
• Perceptual, Cognitive and Motor Abilities
• Colour perception varies between people
• Spatial ability, cognitive or motor disabilities

60. Designing for Children
• HMDS
• inter pupillary distance, head fit, size and weight
• Tablets
• Poor dexterity, need to hold large tablet
• Content
• Reading ability, spatial perception

61. 3. Design for the Whole User

62. Consider Your User
• Consider context of user
• Physical, social, emotional, cognitive, etc.
• Mobile Phone AR User
• Probably Mobile
• One hand interaction
• Short application use
• Need to be able to multitask
• Use in outdoor or indoor environment
• Want to enhance interaction with real world

63. Would you wear this HMD?

64. Whole User Needs
• Social
• Don’t make your user look stupid
• Cultural
• Follow local cultural norms
• Physical
• Can the user physically use the interface?
• Cognitive
• Can the user understand how the interface works?
• Emotional
• Make the user feel good and in control

65. Example: Social Acceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

66. TAT AugmentedID

69. 4. Use UI Best Practices
• General UI design principles can be applied to AR
• E.g. Shneiderman’s UI guidelines from 1998
• Providing interface feedback
• Mixture of reactive, instrumental and operational feedback
• Maintain spatial and temporal correspondence
• Use constraints
• Specify relations between variables that must be satisfied
• E.g. physical constraints reduce freedom of movement
• Support Two-Handed control
• Use Guiard’s framework of bimanual manipulation
• Dominant vs. non-dominant hands

70. Follow Good HCI Principles
• Provide good conceptual model/Metaphor
• customers want to understand how UI works
• Make things visible
• if object has function, interface should show it
• Map interface controls to customer s model
• infix -vs- postfix calculator -- whose model?
• Provide feedback
• what you see is what you get!

71. Example: Guiard’s model of bimanual manipulation
Guiard, Y. (1987). Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of Motor Behavior, 19, 486-517.
Dominant
hand
Non-dominant
hand
Dominant
hand
Non-dominant
hand
Non-Dominant: Leads, set spatial reference frame, performs coarse motions
Dominant: Follows, works in reference frame, performs fine motions

72. Adapting Existing Guidelines
• Mobile Phone AR
• Phone HCI Guidelines
• Mobile HCI Guidelines
• HMD Based AR
• 3D User Interface Guidelines
• VR Interface Guidelines
• Desktop AR
• Desktop UI Guidelines

73. Example: Apple iOS Interface Guidelines
• Make it obvious how to use your content.
• Avoid clutter, unused blank space, and busy backgrounds.
• Minimize required user input.
• Express essential information succinctly.
• Provide a fingertip-sized target for all controls.
• Avoid unnecessary interactivity.
• Provide feedback when necessary
From: https://developer.apple.com/ios/human-interface-guidelines/

74. Applying Principles to Mobile AR
• Clean
• Large Video View
• Large Icons
• Text Overlay
• Feedback

75. •Interface Components
• Physical components
• Display elements
• Visual/audio
• Interaction metaphors
Physical
Elements
Display
Elements
Interaction
Metaphor
Input Output
5. Use Interface Metaphors

76. AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Expressiveness, Intuitiveness

77. AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Design for Layers

78. Information Layers
• Head-stabilized
• Heads-up display
• Body-stabilized
• E.g., virtual tool-belt
• World-stabilized
• E.g., billboard or signpost

79. Head stabilized
• Information attached to view – always visible

80. Body Stabilized
• Information moves with person

81. Body Stabilized Interface
• Elements you want always available, but not always visible

82. World Stabilized
• Information fixed in world

83. • Elements you want fixed relative to real world objects

84. “Diegetic UI”
• Integrated with world

85. Example: Fragments
• UI Elements embedded
in real world
• Real world occlusion

86. Fragments Demo
• https://www.youtube.com/watch?v=kBGWZztPZ4A

87. Design to Device Constraints
• Understand the platform and design for limitations
• Hardware, software platforms
• E.g. Handheld AR game with visual tracking
• Use large screen icons
• Consider screen reflectivity
• Support one-hand interaction
• Consider the natural viewing angle
• Do not tire users out physically
• Do not encourage fast actions
• Keep at least one tracking surface in view
Art of Defense Game

88. Handheld AR Constraints/Affordances
• Camera and screen are linked
• Fast motions a problem when looking at screen
• Intuitive “navigation”
• Phone in hand
• Two handed activities: awkward or intuitive
• Extended periods of holding phone tiring
• Awareness of surrounding environment
• Small screen
• Extended periods of looking at screen tiring
• In general, small awkward platform
• Vibration, sound
• Can provide feedback when looking elsewhere

89. Common Mobile AR Metaphors
• Tangible AR Lens Viewing
• Look through screen into AR scene
• Interact with screen to interact with AR content
• Touch screen input
• E.g. Invisible Train
• Metaphor – holding a window into the AR world

90. The Invisible Train
https://www.youtube.com/watch?v=6LE98k0YMLM

91. Common Mobile AR Metaphors
• Tangible AR Lens Manipulation
• Select AR object and attach to device
• Physically move phone to move AR object
• Use motion of device as input
• E.g. AR Lego
• Metaphor – phone as physical handle for device

92. • https://www.youtube.com/watch?v=icmqv32HEPU

93. AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Design for Affordances

94. Tangible AR Metaphor
• AR overcomes limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design

95. Tangible AR Design Principles
• Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual content
• Support for spatial 3D interaction techniques
• Time and space multiplexed interaction
• Support for multi-handed interaction
• Match object affordances to task requirements
• Support parallel activity with multiple objects
• Allow collaboration between multiple users

96. AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design

97. Affordances
”… the perceived and actual properties of the thing, primarily
those fundamental properties that determine just how the
thing could possibly be used. [...]
Affordances provide strong clues to the operations of things.”
(Norman, The Psychology of Everyday Things 1988, p.9)

98. Affordances

99. Affordance Matrix

100. Affordance Matrix
Fake door
Hidden door
Real door
No door

101. Physical vs. Virtual Affordances
• Physical Affordance
• Look and feel of real objects
• Shape, texture, colour, weight, etc.
• Industrial Design
• Virtual Affordance
• Look of virtual objects
• Copy real objects
• Interface Design

102. •AR design is mixture of physical
affordance and virtual affordance
•Physical
•Tangible controllers and objects
•Virtual
•Virtual graphics and audio

103. Affordances in AR
• Design AR interface objects to show how they are used
• Use visual and physical cues to show possible affordances
• Perceived affordances should match actual affordances
• Physical and virtual affordances should match
Merge Cube Tangible Molecules

104. Case Study 1: 3D AR Lens
Goal: Develop a lens based AR interface
• MagicLenses
• Developed at Xerox PARC in 1993
• View a region of the workspace differently to the rest
• Overlap MagicLenses to create composite effects

105. 3D MagicLenses
MagicLenses extended to 3D (Veiga et. al. 96)
§ Volumetric and flat lenses

106. AR Lens Design Principles
• Physical Components
• Lens handle
• Virtual lens attached to real object
• Display Elements
• Lens view
• Reveal layers in dataset
• Interaction Metaphor
• Physically holding lens

107. 3D AR Lenses: Model Viewer
§ Displays models made up of multiple parts
§ Each part can be shown or hidden through the lens
§ Allows the user to peer inside the model
§ Maintains focus + context

108. AR Lens Demo

109. AR Lens Implementation
Stencil Buffer Outside Lens
Inside Lens Virtual Magnifying Glass

110. Case Study 2 : LevelHead
• Block based game

111. Case Study 2: LevelHead
• Physical Components
• Real blocks
• Display Elements
• Virtual person and rooms
• Interaction Metaphor
• Blocks are rooms

113. Level Head Demo

114. Case Study 3: AR Chemistry (Fjeld 2002)
• Tangible AR chemistry education

115. Goal: An AR application to teach
molecular structure in chemistry
•Physical Components
• Real book, rotation cube, scoop, tracking
markers
•Display Elements
• AR atoms and molecules
• Interaction Metaphor
• Build your own molecule

116. AR Chemistry Input Devices

118. AR Chemistry Demo

119. Case Study 4: Transitional Interfaces
Goal: An AR interface supporting
transitions from reality to virtual reality
•Physical Components
• Real book
•Display Elements
• AR and VR content
• Interaction Metaphor
• Book pages hold virtual scenes

120. Milgram’s Continuum (1994)
Reality
(Tangible
Interfaces)
Virtualit
y
(Virtual
Reality)
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Mixed Reality (MR)
Central Hypothesis
• The next generation of interfaces will support
transitions along the Reality-Virtuality continuum

121. Transitions
• Interfaces of the future will need to support
transitions along the RV continuum
• Augmented Reality is preferred for:
• co-located collaboration
• Immersive Virtual Reality is preferred for:
• experiencing world immersively (egocentric)
• sharing views
• remote collaboration

122. The MagicBook
•Design Goals:
•Allows user to move smoothly
between reality and virtual reality
•Support collaboration

123. MagicBook Metaphor

124. MagicBook Demo

125. Features
• Seamless transition from Reality to Virtuality
• Reliance on real decreases as virtual increases
• Supports egocentric and exocentric views
• User can pick appropriate view
• Computer becomes invisible
• Consistent interface metaphors
• Virtual content seems real
• Supports collaboration

126. Collaboration in MagicBook
• Collaboration on multiple levels:
• Physical Object
• AR Object
• Immersive Virtual Space
• Egocentric + exocentric collaboration
• multiple multi-scale users
• Independent Views
• Privacy, role division, scalability

127. Technology
• Reality
• No technology
• Augmented Reality
• Camera – tracking
• Switch – fly in
• Virtual Reality
• Compass – tracking
• Pressure pad – move
• Switch – fly out

128. Summary
•When designing AR interfaces, think of:
• Physical Components
• Physical affordances
• Virtual Components
• Virtual affordances
• Interface Metaphors
• Tangible AR or similar

129. AR INTERFACE DESIGN
GUIDELINES

130. Design Guidelines
By Vendors
Platform driven
By Designers
User oriented
By Practitioners
Experience based
By Researchers
Empirically derived

131. Design Patterns
“Each pattern describes a problem which occurs
over and over again in our environment, and then
describes the core of the solution to that problem in
such a way that you can use this solution a million
times over, without ever doing it the same way twice.”
– Christopher Alexander et al.
Use Design Patterns to Address Reoccurring Problems
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

132. Example UI Design Patterns
• http://ui-patterns.com/patterns

134. Design Patterns for Handheld AR
• Set of design patterns for Handheld AR
• Title: a short phase that is memorable.
• Definition: what experiences the prepattern supports
• Description: how and why the prepattern works,
what aspects of game design it is based on.
• Examples: Illustrate the meaning of the pre-pattern.
• Using the pre-patterns: reveal the challenges and
context of applying the pre-patterns.
Xu, Y., Barba, E., Radu, I., Gandy, M., Shemaka, R., Schrank, B., ... & Tseng, T.
(2011, October). Pre-patterns for designing embodied interactions in handheld
augmented reality games. In 2011 IEEE International Symposium on Mixed and
Augmented Reality-Arts, Media, and Humanities (pp. 19-28). IEEE.

135. Handheld AR Design Patterns
Title Meaning Embodied Skills
Device Metaphors Using metaphor to suggest available player
actions
Body A&S Naïve physics
Control Mapping Intuitive mapping between physical and digital
objects
Body A&S Naïve physics
Seamful Design Making sense of and integrating the
technological seams through game design
Body A&S
World Consistency Whether the laws and rules in
physical world hold in digital world
Naïve physics
Environmental A&S
Landmarks Reinforcing the connection between digital-
physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in the
game decides how much they feel like living in
the digital game world
Environmental A&S
Naïve physics
Living Creatures Game characters that are responsive to
physical, social events that mimic behaviours
of living beings
Social A&S Body A&S
Body constraints Movement of one’s body position
constrains another player’s action
Body A&S Social A&S
Hidden information The information that can be hidden and
revealed can foster emergent social play
Social A&S Body A&S
*A&S = awareness and skills

137. Example: Seamless Design
• Design to reduce seams in the user experience
• Eg: AR tracking failure, change in interaction mode
• Paparazzi Game
• Change between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games,
Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media, and
Humanities, p.19-28, October 26-29, 2011

138. Demo: Paparazzi Game
• https://www.youtube.com/watch?v=MIGH5WGMnbs

139. Example: Living Creatures
• Virtual creatures should respond to real world events
• eg. Player motion, wind, light, etc
• Creates illusion creatures are alive in the real world
• Sony EyePet
• Responds to player blowing on creature

149. Google ARCore Interface Guidelines
https://developers.google.com/ar/design

150. ARCore Elements App
• Mobile AR app demonstrating
interface guidelines
• Multiple Interface Guidelines
• User interface
• User environment
• Object manipulation
• Off-screen markers
• Etc..
• Test on Device
• https://play.google.com/store/apps/details?id=com.google.ar.unity.ddelements

151. ARCore Elements
• https://www.youtube.com/watch?v=pRHmLuXIs0s

152. ARKit Interface Guidelines
• developer.apple.com/design/human-interface-guidelines/ios/system-capabilities/augmented-reality/

153. Microsoft Mixed Reality Design Guidelines
• https://docs.microsoft.com/en-us/windows/mixed-reality/design/design

154. MRTK Interface Examples
• Examples of UX Building Blocks
• http://aka.ms/MRTK

155. The Trouble with AR Design Guidelines
1) Rapidly evolving best practices
Still a moving target, lots to learn about AR design
Slowly emerging design patterns, but often change with OS updates
Already major differences between device platforms
2) Challenges with scoping guidelines
Often too high level, like “keep the user safe and comfortable”
Or, too application/device/vendor-specific
3) Best guidelines come from learning by doing
Test your designs early and often, learn from your own “mistakes”
Mind differences between VR and AR, but less so between devices

156. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au