AVI 2014 Best Paper: http://dx.doi.org/10.1145/2598153.2598160 See patent: Visual focus-aware techniques for visualizing display changes - https://patents.google.com/patent/US20150227334A1 Increasing screen real-estate allows for the development of applications where a single user can manage a large amount of data and related tasks through a distributed user interface. However, such users can easily become overloaded and become unaware of display changes as they alternate their attention towards different displays. We propose AwToolkit, a novel widget set for developers that supports users in maintaining awareness in multi-display systems. The AwToolkit widgets automatically determine which display a user is looking at and provide users with notifications with different levels of subtlety to make the user aware of any unattended display changes. The toolkit uses four notification levels (unnoticeable, subtle, intrusive and disruptive), ranging from an almost imperceptible visual change to a clear and visually salient change. We describe AwToolkit’s six widgets, which have been designed for C# developers, and the design of a user study with an application oriented towards healthcare environments. The evaluation results reveal a marked increase in user awareness in comparison to the same application implemented without AwToolkit.

1. AwToolkit:
Attention-Aware User Interface
Widgets
Attention-Aware User Interface
Widgets
Juan E. Garrido, Victor M. R. Penichet, Maria D. Lozano
Computer Science Research Institute,
University of Castilla-La Mancha
Aaron Quigley, Per Ola Kristensson
SACHI, School of Computer Science,
University of St Andrews

2. 2AVI’14: AwToolkit Slide of 20
The problem

3. 3AVI’14: AwToolkit Slide of 20
The opportunity
• Determine which display a user is attending to
• Determine what changes are occurring on a non-
attended display and where
• Determine a means to notify a user of these changes
• either on the display they are currently attending to or
• in-situ, when they return their gaze to a previously
unattended display
• Determine the granularity to notify of change
• Determine if this has an impact on performance.

4. 4AVI’14: AwToolkit Slide of 20
Overview
• Related work
• AwToolkit concept and features in widgets
• Spectrum of change notification
• Properties
• AWPanel and change notification techniques
• AwButton, AwLabel, AwTextBox, AwComboBox and
AwPlayer
• Evaluation: User’s View
• Evaluation: Developer’s View
• Conclusions and future work

5. 5AVI’14: AwToolkit Slide of 20
Related work
MAUI Toolkit, CSCW 2004 Mnemonic rendering, UIST 2006
DiffDisplays, IUI 2014TeleEye, CollaborateCom 2008

6. 6AVI’14: AwToolkit Slide of 20
DiffDisplays: Gaze dependent
techniques for visualizing display
changes (IUI 2013)
Jakub Dostal, Per Ola Kristensson, and Aaron Quigley. 2013. Subtle gaze-dependent techniques for visualising display changes in multi-display environments.
In Proceedings of the 2013 international conference on Intelligent user interfaces (IUI '13). ACM, New York, NY, USA, 137-148.

7. 7AVI’14: AwToolkit Slide of 20
DiffDisplays
Jakub Dostal, Per Ola Kristensson, and Aaron Quigley. 2013. Subtle gaze-dependent techniques for visualising display changes in multi-display environments.
In Proceedings of the 2013 international conference on Intelligent user interfaces (IUI '13). ACM, New York, NY, USA, 137-148.

8. 8AVI’14: AwToolkit Slide of 20
DiffDisplays
PixMap Aura
WindowMap

9. 9AVI’14: AwToolkit Slide of 20
DiffDisplays

10. 10AVI’14: AwToolkit Slide of 20
SpiderEyes
QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Dostal, J., Hinrichs, U., Kristensson, P.O. and Quigley, A. 2014. SpiderEyes: designing attention- and proximity-aware collaborative interfaces for wall-sized
displays. In Proceedings of the 19th
ACM International Conference on Intelligent User Interfaces (IUI 2014).

11. 11AVI’14: AwToolkit Slide of 20
Overview
• Related work
• AwToolkit concept and features in widgets
• Spectrum of change notification
• Properties
• AWPanel and change notification techniques
• AwButton, AwLabel, AwTextBox, AwComboBox and
AwPlayer
• Evaluation: User’s View
• Evaluation: Developer’s View
• Conclusions and future work

12. 12AVI’14: AwToolkit Slide of 20
Widgets
http://www.eclipse.org/swt/widgets/

13. 13AVI’14: AwToolkit Slide of 20
AWToolkit concept
• Goal: generate change notifications
upon return of gaze to display
• Approach: Gaze-aware user interface
components (widgets)
• subtle change within widgets
• Three categories: entire screen, part
of screen, mirroring techniques
• AwToolkit C# toolkit for developers for
gaze sensitive user interface controls
• Examples: panel, button, label, textbox, comboBox and
media player

14. 14AVI’14: AwToolkit Slide of 20
AWToolkit features
• Toolkit for detection and reaction
• Objective: preserve look and feel of existing
components as in, for example, Visual Studio
• Detection based on the Diff Displays system (widgets
connected to server)
• Two gaze aware states
• Properties of each widget allow the developer to
modify the visual appearance and behaviour ie. the
degree of saliency in change notification.

15. 15AVI’14: AwToolkit Slide of 20
Spectrum of change notification
• Disruptive:
• the user is alerted and the current activity is disrupted
• Intrusive:
• the user is alerted and the goal is clearly to avert their gaze to the
location of the displayed change notification
• Subtle:
• the user is alerted with less obvious visual cues. The goal is to
draw their gaze to the display change notification
• Unnoticeable:
• the user is alerted in such a subtle manner, that they may not even
notice the display change notification.

16. 16AVI’14: AwToolkit Slide of 20
Properties
• Developer can specify
• Choice of display change notifications
• Appearance of graphic elements
• Length of time for change notification
• Or use AwDefaultTechnique to automate level selection
• default selection of visual change elements
• based on user study discussed later
• Three methods for localisation of widget content change
• Widget screenshot, widget text, state change.

17. 17AVI’14: AwToolkit Slide of 20
AWPanel
• Based on standard panel control
• Typically used to organise application content
• Polygon with content and margin (a continuous line)
• Widget detects changes by comparing screen-shots over
time (on a pixel level)
• This allows AWPanel to be content agnostic

18. 18AVI’14: AwToolkit Slide of 20
3 Change notification techniques
• Modifying the widget margin:
colour, line stippling pattern, corner marks
• Choice of these vs standard margin can range
from unnoticeable to intrusive
• Quadrant view:
panel conceived as four
regions tied to the four
corners (change and where)
• Book mode:
dependent on the number of changes
series of navigable screen shots.
18

19. 19AVI’14: AwToolkit Slide of 20
AwButton, AwLabel, AwTextBox,
AwComboBox and AwPlayer
• Modifying the widget margin: colour or line stippling
pattern
• Highlight changes within
the active area of the
widget
• AwPlayer offers a
disruptive notification
connected to a media player
stopping or restarting a
video depending on a users
gaze.

20. 20AVI’14: AwToolkit Slide of 20
Ubi4health
tasks to be performed
alert control

21. 21AVI’14: AwToolkit Slide of 20
Evaluation: User’s View
• Evaluation was performed with
12 participants (5m, 7f: age 21-48,
avg 36) with Ubi4health
• a healthcare application with a three-screen configuration
• used to manage essential tasks in residential care homes
• typical use case: left screen (tasks to be performed),
centre screen (alert control), right screen (staff
management)
• two versions of Ubi4health
• One with AwPanel, AwButton, AwLabel, AwTextBox and
AwComboBox (and one without).
QuickTime™ and a
AVC Coding decompressor
are needed to see this picture.

22. 22AVI’14: AwToolkit Slide of 20
Evaluation: Tasks
• Scenario: a user responsible for the completion of pending
tasks, staff organisation and the immediate attention of
emergencies in a real residential care home
• Task 1: start of shift, task progression monitoring, resident
alert monitoring, moving to staff allocation (as tasks not
performed in time)
• Task 2: staff organisation (single screen focus 15 sec),
move to alter monitoring
• Task 3: staff reorganisation while monitoring emergencies
(25 sec), return to pending tasks
• Task 4: task monitoring (15 sec), move to staff organisation.

23. 23AVI’14: AwToolkit Slide of 20
Evaluation: Results
• Within-subjects design with toolkit as the independent
variable, accuracy as the dependent variable
• Conditions made it difficult for the participants to compare the
current state of a screen with a previous one
• Improvement in tasks 1, 3 and 4 (p < 0.0001; RM-ANOVA,
Bonferroni corrected)
• Task 2, Ubi4health
already has a clear
visual change on the
screen
• (1,3,4) without AwToolkit:
11-23% accuracy With 83-100% accuracy.

24. 24AVI’14: AwToolkit Slide of 20
Evaluation: Developer’s View
• 12 developers (good knowledge of C# and Visual Studio)
• Task 1: import the library with AwToolkit to be able to
see and use its items in the toolbar
• Task 2: insert AwButton and AwPanel into an application
and define how widgets behave when user is not looking
• Task 3: insert a 10-seconds timer to be run when Aw-
Button is pressed
• Task 4: disable the button after the 10-second timeout
and introduce any change in the AwPanel
• Avg accuracy 100% on tasks 1 and 2, 83% on task 3
and 92% on task 4.

25. 25AVI’14: AwToolkit Slide of 20
Conclusions and future work
• A novel toolkit for developers to support notifying users of
changes that appear in the user interface when users are
not looking at the display
• Awareness of changes at the widget level at varying
degrees of subtly
• Future:
• extensions of the toolkit to further interface components
• exploration of mirroring and multi-screen notifications
• exploration of spectrum of change in multi-display
environments including mobile interface elements.

26. Thank you
aquigley@st-andrews.ac.uk