The document discusses research on cross-media information spaces and architectures led by Prof. Beat Signer. Some key areas of research include interactive paper, personal information management across digital and physical media, tangible user interfaces, and dynamic data physicalization using holograms and other technologies. The goal is to develop more fluid and open document formats that go beyond simulating paper on desktop computers.
9953056974 Young Call Girls In Mahavir enclave Indian Quality Escort service
Â
Cross-Media Information Spaces and Architectures (CISA
1. 2 December 2005
Cross-Media Information Spaces and Architectures
Prof. Beat Signer
http://www.beatsigner.com
Web & Information Systems Engineering Lab
Department of Computer Science
Vrije Universiteit Brussel
WEB & INFORMATION
SYSTEMS ENGINEERING
3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3October 19, 2018
Information
Systems &
Management
Information
Visualisation
& Navigation
Information
Visualisation
nd NavigationHuman-Machine &
Human-Information
Interaction
CISA
RSL [4]
MindXpres [8]
OC2 [5]
associative file system [2]
data-driven storytelling [11]
XIMA [13]
cross-device interaction [9]
iServer/iPaper [3]
ViDaX [16]
data physicalisation [32]
map-based interaction [18]
non-linear storytelling [27]
PaperProof [29]iGesture [23]
PaperSketch [30]
TangHo [31]
Context Modelling Toolkit [10]
Midas [21]
SpeeG2 [24]
PaperPoint [26]
Print-n-Link [17]
digital libraries [12]
Mudra [22] mixed reality [28]
ArtVis [15]
open cross-media linking [1]
PimVis [14]
DocTr [6]
EdFest [25]
source code visualisation [19]
INFEX [7]
collaborative filtering [20]
Cross-Media Information Spaces and Architectures
4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4October 19, 2018
Models and Frameworks
dynamic data physicalisation framework
RSL hypermedia metamodel iPaper framework iGesture framework
Personal Information Management (PIM) framework MindXpres presentation platform
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5October 19, 2018
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
RSL Hypermedia Metamodel
MindXpres
Paper-Digital Interfaces
PimVisTangible Holograms
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6October 19, 2018
Prof. Dr. Beat Signer
Interactive Paper, Cross-Media
Information Architectures
Audrey Sanctorum
User-defined Cross-Device and
Cross-Media Interaction
Ahmed K.A. Abdullah
TangHo, Mixed Reality,
Dynamic Data Physicalisation
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
WEB & INFORMATION
SYSTEMS ENGINEERING
CROSS-MEDIA INFORMATION SPACES
AND ARCHITECTURES (CISA)
Payam Ebrahimi
Dynamic Data Physicalisation,
Mixed Reality, TangHo
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7October 19, 2018
Jan Maushagen
Learning Analytics, Adaptive
Persuasive ICT Tools
WEB & INFORMATION
SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
CROSS-MEDIA INFORMATION SPACES
AND ARCHITECTURES (CISA)
Reinout Roels
MindXpres: Extensible Content-
driven Presentation Tool
Dr. Ahmed A.O. Tayeh
Open Cross-Media Authoring,
Fluid Document Formats
Dr. Sandra Trullemans
Cross-Media Personal
Information Management
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8October 19, 2018
What is Wrong with Digital Documents?
ī§ Existing document formats are based on the simulation
of paper affordances on desktop computers
ī§ How to manage mixed-media "documents" in open and
fluid cross-media information spaces?
ī§ on the data level
- context-sensitive adapatation
- cross-media transclusion
ī§ on the visualisation and navigation level
- zoomable user interfaces
ī§ on the cross-media interaction level
- fluid multimodal cross-media interfaces
ī§ Remediation of the "paper simulation" approach
ī§ WYSIWYG is only one out of many options!
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9October 19, 2018
Research Methodology
Conceptual
Modelling
General
Frameworks
Applications
User
Studies
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10October 19, 2018
PARC User Interface (Mid 70s)
ī§ Desktop metaphor
ī§ WIMP interaction
ī§ "filing cabinets" with
hierarchical folders
ī§ Paper paradigm
ī§ "What You See Is What
You Get" (WYSIWYG)
ī§ what about richer
document formats that
existed at that time?
Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11October 19, 2018
Digital Documents as a Paper Simulator?
Vannevar Bush
Ted Nelson
Most people don't understand the logic of the
concept: "What You See Is What You Get" is
based on printing the document out ("get"
means "get WHEN YOU PRINT IT OUT"). And that
means a metaphysical shift: a document can
only consist of what can be printed! [...] No
overlays [...] â PAPER UNDER GLASS.
When data of any sort are placed in storage,
they are filed alphabetically or numerically,
and information is found (when it is) by
tracing it down from subclass to subclass. It
can be in only one place, unless duplicates
are used [...] The human mind does not work
that way. It operates by association.
As We May Think, Atlantic Monthly, July 1945
Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12October 19, 2018
Evolution of Interfaces
date January 1984 February 2018 + 34 years
price $2,500 $2,299 x 0.92
CPU 68000 Motorola
8 MHz
0.7 MIPS
quad-core Intel core i5
4.2 GHz
238,310 MIPS
x 525
x 340,442
memory 128 kB 8 GB x 62,500
storage 400 kB floppy drive 2 TB fusion drive x 5,000,000
monitor 9" black and white
512 x 342
68 dpi
27" colour
5120 x 2880
218 dpi
x 3
x 84
x 3.2
input devices mouse
keyboard
mouse
keyboard
same
same
GUI desktop WIMP desktop WIMP same
original Macintosh 27-inch iMac comparison
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13October 19, 2018
RSL Metamodel and iServer
Global Information Systems Group, ETH Zurich
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14October 19, 2018
Links
Users
Layers
Plug-ins
Structure
Active
Content
Global Information Systems Group, ETH Zurich
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15October 19, 2018
Bridging the Paper-Digital Divide
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16October 19, 2018
Paper-Digital Integration
Amazon Kindle Anoto Digital Pen and Paper Technology
Augmented / Interactive PaperElectronic Paper
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17October 19, 2018
Cross-Media Link Server (iServer)
Global Information Systems Group
ETH Zurich
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18October 19, 2018
Interactive Paper Applications
ī§ Enhanced Reading
ī§ paper-digital maps
ī§ interactive festival brochures
ī§ educational materials
ī§ auction brochures
ī§ scientific publications
ī§ âĻ
ī§ Enhanced Writing
ī§ photo album
ī§ researcher's notebook
ī§ mammography annotation
ī§ query by sketching
ī§ âĻ
ī§ Paper-based Interfaces
ī§ PaperPoint
ī§ document proof-editing
ī§ digital libraries
ī§ video analysis tool
ī§ educational games
ī§ Art Installations
ī§ Lost Cosmonaut
ī§ Generosa Enterprise
ī§ Interactive Tabletops
ī§ photo browser
ī§ collaborative sketch tool
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19October 19, 2018
Interactive Festival Guide (EdFest)
Global Information Systems Group
ETH Zurich
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20October 19, 2018
Visual and Interaction Design
ī§ Visualisation of interactive
areas and functionality
ī§ design patterns
ī§ Interaction design
ī§ online versus offline input
processing
ī§ lack of modal dialogues
ī§ multimodal interaction
ī§ Findings might be appli-
cable to traditional GUIs
ī§ e.g. cross-application widget
interaction
Global Information Systems Group, ETH Zurich
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21October 19, 2018
Paper-Digital Maps
Szabolcs Becze
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22October 19, 2018
RSL-based Associative File System
signer
My Pictures
Vancouver ER 2010
Granville
Island
Vancouver
Harbour
Public
Market
Keynote
J. Thorp
Olga and
Lamia
Dinner
ER 2010
multiple classification
structural link
navigational link
My Videos
Vancouver
Aquarium
ER 2010
Keynote
cross-media association
My Presentations
ER 2010 WISE 2009
cross-media
transclusion
single slide
transclusion
(via OOXML
RSL selector)
Gregory Cardone
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23October 19, 2018
RSL-based Associative File System ...
Gregory Cardone
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24October 19, 2018
Personal Information Management (PIM)
ī§ Keeping, organising and
re-finding information
ī§ digital and physical
ī§ Study of human-
information interaction
ī§ files, piles, mixtures, âĻ
ī§ OC2 PIM model
ī§ based on RSL hypermedia
metamodel
ī§ Cross-Media PIM system
ī§ explicit as well as implicit
associations between entities
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25October 19, 2018
Object-Concept-Context (OC2) Framework
object layer
structural link
concept layer
context layer
navigational link
associative link
weighted link
Object 1 Object n
Object 2
Context 1 Context n
Concept 1
Concept 2 Concept n
extent link
entity
link
Links
selector resource
(1,*)(1,*)
(1,1) (0,*)
(0,*) (0,*)
RefersTo
HasTargetHasSource
partition
Entities
(0,*)
link
Navigational
Links
link
Extent
Links
partition
ResourcesSelectors
context
Contexts
link
Structural
Links
link
Associative
Links
resource
Physical
Objects
resource
Digital
Objects
resource
Concepts
resource
Objects
HasAssoc
Target
(0,*)
(1,*)
HasExt
Target
HasExt
Source
HasAssoc
Source
(1,1)
(1,*)
(0,*)
(0,*)
(0,*)
(1,1)
disjoint
partition
translation
OC2 conceptual model RSL-based metamodel
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26October 19, 2018
PimVis
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27October 19, 2018
PimVis Setup
28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28October 19, 2018
PimVis Context View
Audrey Sanctorum
29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29October 19, 2018
PimVis Document View
Audrey Sanctorum
30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30October 19, 2018
PimVis Focus View
Audrey Sanctorum
31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31October 19, 2018
Document Tracking (DocTr) Framework
ī§ Unified document (object)
tracking solution
ī§ plug-ins for QR codes, RFID
tags, SIFT algorithm, âĻ
ī§ tracking of documents across
organisational structures
ī§ avoid fragmentation when
using multiple tracking
setups
Document Management
PendingPool
DataModel
IdentificationMechanism
OC2
GUI
REST Input Interface
REST Output Interface
Third-party
Applications
DocTr
Delegator
Tracking Setups
32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32October 19, 2018
Cross-Device and Cross-Media Interfaces
ī§ Rapid prototyping platform
for cross-device and
cross-media integration
ī§ End-user authoring
ī§ customised distribution of
user interface components
ī§ mashup tool for digital and
physical (IoT) components
33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33October 19, 2018
ArtVis: TUI-based Information Exploration
ī§ Advanced visualisation
techniques in combination
with a TUI
ī§ explore Web Gallery of Art
ī§ faceted browsing
ī§ phidgets-based TUI
ī§ RFID-tagged physical
objects
ī§ Three main components
to explore, analyse and
browse the information
Bram Moerman
34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34October 19, 2018
ArtVis Setup
Bram Moerman
35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35October 19, 2018
ArtVis Visualisation
Bram Moerman
36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36October 19, 2018
Video: ArtVis
Bram Moerman
37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37October 19, 2018
Tangible Hologram (TangHo) Setup
Timothy J. Curtin
38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38October 19, 2018
Tangible Graph Hologram
39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39October 19, 2018
Collaborative TangHo Interaction
40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40October 19, 2018
Real-Time Cross-Media Data Exploration
41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41October 19, 2018
Dynamic Data Physicalisation
ī§ Physical objects used for
input as well as output
ī§ How can we achieve
dynamic data physicali-
sation with dynamic
affordances
ī§ use physical variables such
as temperature or texture
ī§ exploration of big data sets
ī§ experimental tangible
holograms (TangHo)
platform
42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42October 19, 2018
TangHo Prototype
ī§ Lego Mindstorms-based
6DOF arm prototype
ī§ motors can be locked or set to
float mode
- useful for bidirectional I/O
ī§ Replaceable end effector for different
types of non-visual feedback
ī§ Challenges
ī§ inverse kinematics (final system
is body mounted)
- use robotics toolbox (MATLAB)
ī§ limited HoloLens hand tracking
ī§ 3D printing of final arm
Timothy J. Curtin
43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43October 19, 2018
TangHo Interactions
44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44October 19, 2018
TangHo Interactions âĻ
45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45October 19, 2018
Data Physicalisation Framework
ī§ Understanding the design space
ī§ how do users map data to physicalisations
ī§ Understanding the perceptual effectiveness
ī§ e.g. what are the just-noticeable differences that a physical
variable can convey?
ī§ Framework for dynamic data physicalisation
ī§ data processing with device-independent output
- TextDIO: text for static or dynamic reports
- PhysDIO: data points, data variables and physical variables
ī§ text and visualisation drivers for representation on specific
devices (e.g. TangHo)
ī§ new data physicalisation design guidelines (as in InfoVis)
46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46October 19, 2018
Data Physicalisation Framework ...
47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47October 19, 2018
Dynamic Data Physicalisation
ī§ Understanding the design space
ī§ how do users map data to physicalisations
ī§ Data physicalisation design guidelines
ī§ what are the just-perceptual differences that
a physical variable can convey?
ī§ Dynamic data physicalisation framework
ī§ data physicalisation grammar
ī§ data processing with device-independent output
ī§ software drivers for different physicalisations (e.g. TangHo)
ī§ New application domains
Jacques Bertin, 1967
48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48October 19, 2018
HCI and HCII in Smart Environments
49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49October 19, 2018
Context Modelling Toolkit (CMT)
ī§ Multi-layered context
modelling approach
ī§ programmer, expert user and
end user
ī§ Beyond simple "if this then
that" rules
ī§ Client-server architecture
ī§ server: context reasoning
based on Drools
ī§ client: sensor input as well as
applications
End User
Expert User
Functions
Actions
Template
Filled in
template
Situation
Situations
Facts
Rule
Programmer
Rule
(4)
(5)
(6)
(7)
(8)
ī§ Increase trust via intelligibility
50. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50October 19, 2018
Context Modelling Toolkit (CMT) âĻ
51. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 51October 19, 2018
Cross-Domain Internet of Things (IoT)
ī§ Middleware for interoper-
ability of smart behaviour
across IoT platforms
ī§ Cross-domain conflict
resolution
ī§ End-user authoring and
intelligibility
52. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 52October 19, 2018
SpeeG2: Multimodal Text Input
User
Speech recognition
(Microsoft SAPI 5.4)
Skeletal tracking
(Microsoft Kinect)
5
4
2
3
SpeeG2 GUI
6
1
Sven De Kock
53. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 53October 19, 2018
Video: SpeeG2 Performance (21 WPM)
Sven De Kock
54. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 54October 19, 2018
INFEX: Cross-Device Information Exchange
55. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 55October 19, 2018
INFEX Architecture
Smartphone
HR Reader
Camera
Laptop
...
Tangible
Remote Device
RFID QR Codes PixelSense
Interface Lookup Service
Bluetooth TCP/IP WebSockets
FTP HTTP REST NFC PBAB
List
Inspect
Transfer
Input Handler
Renderer
Images
PDF
Contacts
Tabletop
Projector
+
Kinect
Laptop
HoloLens
...
56. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 56October 19, 2018
INFEX Scenario
57. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 57October 19, 2018
Open Cross-Media Linking
ī§ Integration of a new resource type includes
ī§ development of a data plug-in
ī§ implementation of a visual plug-in
ī§ registration with the resource plug-in repository
ī§ Link browser loads plug-ins on demand
ī§ internal or external (third party application) visualisation
ī§ Third-party applications use the link browser for
visualisation
ī§ similar to the integration of third-party applications with web
browsers
ī§ Non-monolithic annotation model and authoring tool
ī§ resource-specific plug-ins
59. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 59October 19, 2018
Cross-Document Linking
local visual plug-ins
external visual plug-ins
communicate
Link Browser
60. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 60October 19, 2018
Document Retrieval and Discovery
ī§ Use explicit as well as implicit document relationships
61. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 61October 19, 2018
Presentation Solutions
ī§ PaperPoint
ī§ interactive PowerPoint paper
handouts
ī§ non-linear navigation and
annotations
ī§ in use since 2005
ī§ MindXpres
ī§ extensible presentation
platform
ī§ non-linear navigation via
zoomable user interface
ī§ rich media plug-ins
ī§ developed since 2011
62. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 62October 19, 2018
PaperPoint Features
ī§ Enhanced mobility during presentations
ī§ Improved presentation overview for the presenter
ī§ see content of current and nearby slides (orientation)
ī§ Annotation of static slide content
ī§ highlighting of content
ī§ dynamic creation of new content at presentation time
ī§ Improved control of embedded media (e.g. video)
ī§ Digital whiteboard
ī§ Multi-pen support
ī§ collaborative presentations
ī§ brainstorming
63. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 63October 19, 2018
What is Wrong with Slideware?
ī§ Simulation of physical slides
ī§ limited space due to the slide concept
ī§ linear navigation from slide to slide
ī§ difficult to reuse content and embed rich media types
ī§ MindXpres addresses these issues
ī§ unlimited canvas with zoomable user interface
ī§ non-linear navigation and associative linking
ī§ content-based approach with automatic visualisation
64. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 64October 19, 2018
MindXpres Presentation Tool
ī§ Extensible prototyping
platform for novel presen-
tation concepts
ī§ content-based approach
- separation of content and pre-
sentation (automatic visualisation)
ī§ cross-media content reuse
ī§ non-linear navigation via
zoomable user interface
ī§ connectivity and interactivity
65. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 65October 19, 2018
MindXpres Presentation Tool ...
66. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 66October 19, 2018
MindXpres Platform
ī§ MindXpres presentations are currently represented in the
XML-based MindXpres document format
ī§ Compiler (node.js application) translates XML to HTML
ī§ Presentation engine based on HTML5 and related APIs
ī§ e.g. WebSockets for connectivity
67. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 67October 19, 2018
MindXpres Communication Platform
68. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 68October 19, 2018
Navigation Mirroring Plug-in
69. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 69October 19, 2018
Polls and Quizzes Plug-in
70. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 70October 19, 2018
Interactive Source Code Plug-in
Paul Mestereaga
71. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 71October 19, 2018
Interactive Data Visualisation
72. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 72October 19, 2018
Interactive Data Visualisation ...
73. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 73October 19, 2018
MindXpres Presentation Tool
ī§ Functional MindXpres prototype
ī§ Ongoing work
ī§ visual authoring tool
- alternative to XML-based authoring
ī§ rich media plug-ins
- videos with subtitles, annotations etc.
- statistical data with alternative visualisations
- live sensor data
- ...
ī§ audience-driven presentations
ī§ collaborative annotations and information sharing
ī§ multimodal interaction (e.g. integration of PaperPoint concepts)
ī§ import of existing slideware formats
74. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 74October 19, 2018
Multimodal and Multi-Touch Interaction
EPOC, emotiv Echo Smartpen, Livescribe
Kinect,Microsoft
75. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 75October 19, 2018
WISE Lab: Office of the Future
76. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 76October 19, 2018
ī§ Midas/Mudra Framework
ī§ declarative definition of multimodal
and multi-touch interactions
ī§ rule-based language approach
ī§ rapid protoyping and application development
ī§ iGesture Workbench
ī§ create and test gesture sets and algorithms
ī§ different modalities: digital pen, Wii remote, âĻ
ī§ open source (www.igesture.org)
ī§ HephaisTK Toolkit
ī§ domain-specific human-computer dialogue
language (SMUIML) for multimodal I/O
ī§ speech recognisers, RFID readers, âĻ
Multimodal and Multi-Touch Interaction
77. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 77October 19, 2018
Mudra
ī§ Declarative rule-based language
ī§ Fusion across different levels of abstraction
ī§ fact base
ī§ Rapid prototyping
ī§ simple integration of new input devices
ī§ integration of external gesture recognisers
78. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 78October 19, 2018
VolTra
Brecht De Rooms
79. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 79October 19, 2018
Video: Kinect Presenter
80. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 80October 19, 2018
Video: Augmented Reality (Mental Nomad)
Toon Duwee
81. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 81October 19, 2018
Interactive Tables
Pen-Based Google Earth Browser Pen-Based Drawing Tool
Photo Browser Photo Browser (Slider)
Global Information Systems Group, ETH Zurich
82. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 82October 19, 2018
The Lost Cosmonaut
ī§ Writing as a collaborative
act of memory and story-
telling
ī§ Reading, writing and
narrative as an act of
making sense
ī§ Handwriting as a tool for
human-computer
interaction
In collaboration with Axel Vogelsang,
Artists in Labs Programme
83. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 83October 19, 2018
Fluid Multimodal Cross-Media Interfaces
ī§ Investigation of mobile and distributed document formats
ī§ Mobile Cross-Media Applications
ī§ Passive or active information units (crosslets)
ī§ processing and storage of sensor data
ī§ binding of specific multimodal interactions to resources
ī§ Implicit context-aware human-computer interaction (HCII)
84. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 84October 19, 2018
Are You Interested in a Thesis?
ī§ Various possibilities for BA, MA and PhD theses
ī§ Mixed Reality Solutions
- e.g. for photography, museums etc.
ī§ Tangible User Interfaces
ī§ Data Physicalisation Framework
- tangible hologram platform
- big data exploration interfaces
ī§ MindXpres Presentation Platform
ī§ Cross-Device and Cross-Media Interfaces
ī§ Smart Environments and Cross-Domain Internet of Things (IoT)
ī§ Personal Information Management (PIM)
ī§ Interactive Paper Solutions
ī§ ...
ī§ Do you have your own ideas? Come along to discuss them...
85. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 85October 19, 2018
Conclusions
ī§ Innovative cross-media document formats which go
beyond simulating paper on desktop computers
ī§ RSL-based representation of open and fluid cross-media
information spaces
ī§ interactive paper and data physicalisation solutions
ī§ personal information management and associative file systems
ī§ MindXpres presentation tool
ī§ Multimodal and multi-touch interaction
ī§ Midas/Mudra, HephaisTK and iGesture
ī§ SpeeG2, Kinect Presenter
ī§ Mixed reality and dynamic data physicalisation
ī§ ArtVis, Mental Nomad and Tangible Holograms
86. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 86October 19, 2018
References
ī§ B. Signer, Fundamental Concepts for
Interactive Paper and Cross-Media Information
Spaces, Second Edition, ISBN 978-3-8370-2713-6,
August 2017
ī§ ArtVis Project
ī§ http://www.beatsigner.com/flyers/ArtVis.pdf
ī§ MindXpres Project
ī§ http://mindxpres.com
ī§ Dynamic Data Physicalisation and Tangible Holograms
ī§ http://beatsigner.com/dataPhysicalisation.html
ī§ http://beatsigner.com/tangibleHolograms.html
87. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 87October 19, 2018
References ...
[1] B. Signer and M.C. Norrie, A Model and
Architecture for Open Cross-Media Annotation and
Link Services, Information Systems 36(3), Elsevier,
May 2011
ī§ http://beatsigner.com/publications/signer_IS2011.pdf
[2] B. Signer, What is Wrong with Digital Documents?
A Conceptual Model for Structural Cross-Media
Content Composition and Reuse, Proceedings of the
29th International Conference on Conceptual Modeling
(ER 2010), Vancouver, Canada, November 2010
ī§ http://beatsigner.com/publications/signer_ER2010.pdf
88. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 88October 19, 2018
References ...
[3] B. Signer, Fundamental Concepts for Interactive
Paper and Cross-Media Information Spaces,
Dissertation ETH No. 16218, Zurich, Switzerland,
2005
ī§ http://beatsigner.com/publications/signer2006.pdf
[4] B. Signer and M.C. Norrie, As We May Link: A General
Metamodel for Hypermedia Systems, Proceedings of
ER 2007, 26th International Conference on Conceptual
Modeling, Auckland, New Zealand, November 2007
ī§ http://beatsigner.com/publications/signer_ER2010.pdf
89. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 89October 19, 2018
References ...
[5] S. Trullemans and B. Signer, Towards a Con-
ceptual Framework and Metamodel for Context-Aware
Personal Cross-Media Information Management
Systems, Proceedings of ER 2014, 33rd International
Conference on Conceptual Modelling, Atlanta, USA,
October, 2014
ī§ http://beatsigner.com/publications/trullemans_ER2014.pdf
[6] S. Trullemans, A. Vercruysse and B. Signer, DocTr:
A Unifying Framework for Tracking Physical Docu-
ments and Organisational Structures, Proceedings of
EICS 2016, Brussels, Belgium, June 2016
ī§ http://beatsigner.com/publications/trullemans_EICS2016.pdf
90. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 90October 19, 2018
References ...
[7] R. Roels, A. De Witte and B. Signer, INFEX:
A Unifying Framework for Cross-Device Information
Exploration and Exchange, Proceedings of the ACM
on Human-Computer Interaction (PACMHCI), 1(3),
2017
[8] R. Roels and B. Signer, MindXpres: An Exten-
sible Content-driven Cross-Media Presentation
Platform, Proceedings of WISE 2014, 15th Interna-
tional Conference on Web Information System
Engineering, Thessaloniki, Greece, October, 2014
ī§ http://beatsigner.com/publications/roels_WISE2014.pdf
91. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 91October 19, 2018
References ...
[9] A. Sanctorum and B. Signer, Towards User-
defined Cross-Device Interaction, Proceedings of DUI
2016, 5th Workshop on Distributed User Interfaces,
Lugano, Switzerland, June 2016
ī§ http://beatsigner.com/publications/sanctorum_DUI2016.pdf
[10] S. Trullemans and B. Signer, A Multi-layered
Context Modelling Approach for End Users, Expert
Users and Programmers, Proceedings of SERVE
2016, International Workshop on Smart Ecosystems
cReation by Visual dEsign, Bari, Italy, June 2016
ī§ http://beatsigner.com/publications/trullemans_SERVE2016.pdf
92. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 92October 19, 2018
References ...
[11] R. Roels, Y. Baeten and B. Signer, An Inter-
active Data Visualisation Approach for Next
Generation Presentation Tools: Towards Rich
Presentation-based Data Exploration and Storytelling,
Proceedings of CSEDU 2016, Rome, Italy, April, 2016
ī§ http://beatsigner.com/publications/roels_CSEDU2016.pdf
[12] M.C. Norrie, B. Signer and N. Weibel, Interactive
Paper as a Reading Medium in Digital Libraries, Pro-
ceedings of ECDL 2008, 12th European Conference
on Research and Advanced Technology for Digital
Libraries, Aarhus, Denmark, September 2008
ī§ http://beatsigner.com/publications/norrie_ECDL2008.pdf
93. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 93October 19, 2018
References ...
[13] B. Signer, M.C. Norrie, P. Geissbuehler and
D. Heiniger, Telephone Interface for Avalanche War-
nings based on Information Server for Adaptable Con-
tent Delivery, Proceedings of Pervasive 2002, Zurich,
Switzerland, August 2002
ī§ http://beatsigner.com/publications/signer_Pervasive2002.pdf
[14] S. Trullemans, A. Sanctorum and B. Signer,
PimVis: Exploring and Re-finding Documents in Cross-
Media Information Spaces, Proceedings of AVI 2016,
International Working Conference on Advanced Visual
Interfaces, Bari, Italy, June 2016
ī§ http://beatsigner.com/publications/trullemans_AVI2016.pdf
94. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 94October 19, 2018
References ...
[15] B. Dumas, B. Moerman, S. Trullemans and
B. Signer, ArtVis: Combining Advanced Visualisation
and Tangible Interaction for the Exploration, Analysis
and Browsing of Digital Artwork Collections, Proceed-
ings of AVI 2014, Como, Italy, May 2014
ī§ http://beatsigner.com/publications/dumas_AVI2014.pdf
[16] B. Dumas, T. BrochÊ, L. Hoste and B. Signer,
ViDaX: An Interactive Semantic Data Visualisation and
Exploration Tool, Proceedings of AVI 2012, Internatio-
nal Working Conference on Advanced Visual Inter-
faces, Capri Island, Italy, May 2012
ī§ http://beatsigner.com/publications/dumas_AVI2012.pdf
95. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 95October 19, 2018
References ...
[17] Moira C. Norrie, Beat Signer and Nadir Weibel,
Print-n-Link: Weaving the Paper Web, Proceedings of
DocEng 2006, ACM Symposium on Document
Engineering, Amsterdam, The Netherlands,
October 2006
ī§ http://beatsigner.com/publications/norrie_DocEng2006.pdf
[18] M. Grossniklaus, M.C. Norrie, B. Signer and
N. Weibel ,Putting Location-Based Services on the
Map, Proceedings of W2GIS 2006, Hong Kong, China,
December 2006
ī§ http://beatsigner.com/publications/grossniklaus_W2GIS2006.pdf
96. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 96October 19, 2018
References ...
[19] R. Roels, P. Mestereaga and B. Signer, An
Interactive Source Code Visualisation Plug-in for the
MindXpres Presentation Platform, Communications in
Computer and Information Science (CCIS), 583, 2016
ī§ http://beatsigner.com/publications/roels_CCIS2016.pdf
[20] A. de Spindler, M.C. Norrie, M. Grossniklaus and
B. Signer, Spatio-Temporal Proximity as a Basis for
Collaborative Filtering in Mobile Environments,
Proceedings of UMICS 2006, Workshop on Ubiquitous
Mobile Information and Collaboration Systems,
Luxembourg, Grand Duchy of Luxembourg, June 2006
ī§ http://beatsigner.com/publications/despindler_UMICS2006.pdf
97. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 97October 19, 2018
References ...
[21] C. Scholiers, L. Hoste, B. Signer and W. De
Meuter, Midas: A Declarative Multi-Touch Interaction
Framework, Proceedings of TEI 2011, 5th Interna-
tional Conference on Tangible, Embedded and Em-
bodied Interaction, Funchal, Portugal, January 2006
ī§ http://beatsigner.com/publications/scholliers_TEI2011.pdf
[22] L. Hoste, B. Dumas and B. Signer, Mudra: A Unified
Multimodal Interaction Framework, Proceedings of
ICMI 2011, 13th International Conference on Multi-
modal Interaction, Alicante, Spain, November 2011
ī§ http://beatsigner.com/publications/scholliers_TEI2011.pdf
98. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 98October 19, 2018
References ...
[23] B. Signer, U. Kurmann and M.C. Norrie,
iGesture: A General Gesture Recognition Framework,
Proceedings of ICDAR 2007, 9th International
Conference on Document Analysis and Recognition,
Curitiba, Brazil, September 2007
ī§ http://beatsigner.com/publications/signer_ICDAR2007.pdf
[24] L. Hoste and B. Signer, SpeeG2: A Speech- and
Gesture-based Interface for Efficient Controller-free
Text Entry, Proceedings of ICMI 2013, 15th Inter-
national Conference on Multimodal Interaction,
Sydney, Australia, December 2013
ī§ http://beatsigner.com/publications/hoste_ICMI2013.pdf
99. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 99October 19, 2018
References ...
[25] B. Signer, M. Grossniklaus and M.C. Norrie,
Interactive Paper as a Mobile Client for a Multi-
Channel Web Information System, World Wide Web
Journal (WWW), Vol. 10, No. 4, Springer,
December 2007
ī§ http://beatsigner.com/publications/signer_WWWJ2007.pdf
[26] B. Signer and M.C. Norrie, PaperPoint: A Paper-
based Presentation and Interactive Paper Prototyping
Tool, Proceedings of TEI 2007, First International
Conference on Tangible and Embedded Interaction,
Baton Rouge, USA, February 2007
ī§ http://beatsigner.com/publications/signer_TEI2007.pdf
100. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 100October 19, 2018
References ...
[27] A. Vogelsang and B. Signer, The Lost Cosmo-
naut: An Interactive Narrative Environment on Basis of
Digitally Enhanced Paper, Proceedings of the Inter-
national Conference on Virtual Storytelling 2005,
Strasbourg, France, December 2005
ī§ http://beatsigner.com/publications/vogelsang_ICVS2005.pdf
[28] L. Hoste and B. Signer, Expressive Control of Indirect
Augmented Reality During Live Music Performances,
Proceedings of NIME 2013, 13th International Confe-
rence on New Interfaces for Musical Expression,
Daejeon, Korea Republic, May 2013
ī§ http://beatsigner.com/publications/hoste_NIME2013.pdf
101. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 101October 19, 2018
References ...
[29] N. Weibel, A. Ispas, B. Signer and M.C. Norrie,
PaperProof: A Paper-Digital Proof-Editing System,
Proceedings of CHI 2008, 26th International Confe-
rence on Human Factors in Computing Systems
(Interactivity Track), Florence, Italy, April 2008
ī§ http://beatsigner.com/publications/weibel_CHI2008.pdf
[30] N. Weibel, B. Signer, M.C. Norrie, H. Hofstetter,
H.-C. Jetter and H. Reiterer, PaperSketch: A Paper-
Digital Collaborative Remote Sketching Tool, Proceed-
ings of IUI 2011, International Conference on Intelli-
gent User Interfaces, Palo Alto, USA, February 2011
ī§ http://beatsigner.com/publications/weibel_IUI2011.pdf
102. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 102October 19, 2018
References ...
[31] B. Signer and T.J. Curtin, Tangible Holograms:
Towards Mobile Physical Augmentation of Virtual
Objects, Technical Report WISE Lab, WISE-2017-01,
March 2017
ī§ http://beatsigner.com/publications/signer_arXiv2017.pdf
[32] B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed
K.A. Abdullah, Towards a Framework for Dynamic
Data Physicalisation, Proceedings of the International
Workshop Toward a Design Language for Data
Physicalisation, Berlin, Germany, October 2018
ī§ http://beatsigner.com/publications/signer_DataPhys2018.pdf
103. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 103October 19, 2018
References ...
[33] B. Signer and M.C. Norrie, Interactive Paper:
Past, Present and Future, Proceedings of PaperComp
2010, 1st International Workshop on Paper Com-
puting, Copenhagen Denmark, September 2010
ī§ http://beatsigner.com/publications/signer_PaperComp2010.pdf
[34] A.A.O Tayeh and B. Signer, A Dynamically Extensible
Open Cross-Document Link Service, Proceedings of
WISE 2015, 16th International Conference on Web
Information Systems Engineering, Miami, USA,
November, 2015
ī§ http://beatsigner.com/publications/tayeh_WISE2015.pdf
104. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 104October 19, 2018
References ...
[35] S. Trullemans and B. Signer, From User Needs
to Opportunities in Personal Information Management:
A Case Study on Organisational Strategies in Cross-
Media Information Spaces, Proceedings of DL 2014,
London, UK, September, 2014
ī§ http://beatsigner.com/publications/trullemans_DL2014.pdf
[36] R. Roels, C. Vermeylen and B. Signer, A Unified
Communication Platform for Enriching and Enhancing
Presentations with Active Learning Components, Pro-
ceedings of ICALT 2014, Athens, Greece, July 2014
ī§ http://beatsigner.com/publications/roels_ICALT2014.pdf
105. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 105October 19, 2018
References ...
[37] A. Ispas, H. Schuldt, M.C. Norrie and B. Signer,
Towards Query by Sketch, Michael Springmann,
Second DELOS Conference on Digital Libraries, Pisa,
Italy, December 2007
ī§ http://beatsigner.com/publications/springmann_DELOS2007.pdf