SlideShare a Scribd company logo
1 of 295
Download to read offline
Fusing Multimedia Data Into
Dynamic Virtual Environments
Ruofei Du
ruofei@cs.umd.edu
Dissertation Committee: Dr. Varshney, Dr. Zwicker, Dr. Huang, Dr. JaJa, and Dr. Chuang
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
Motivation
Popularity of VR and AR devices
2
Motivation
Popularity of VR and AR devices
3
Motivation
Popularity of VR and AR devices
4
Motivation
Popularity of VR and AR devices
5
Motivation
Popularity of VR and AR devices
6
4.5million VR and AR devices shipped worldwide
https://www.statista.com/statistics/671403/global-virtual-reality-device-shipments-by-vendor
Motivation
Assorted VR and AR applications
Remote Education
7
Motivation
Assorted VR and AR applications
Immersive Entertainment
8
Motivation
Assorted VR and AR applications
Business Meetings
9
Fusing Multimedia Data Into Dynamic Virtual Environments
Where is the
3D data going
to come from?
Motivation
Lack of contents in VR and AR
12
Motivation
Lack of contents in VR and AR
13
Automatically
fusing multimedia
data into
dynamic virtual
environments
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
15
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Montage4D
ACM I3D’18
To appear in JCGT’18
Social Street View
Best Paper Award
ACM Web3D’16
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
16
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Video Fields
ACM Web3D’16
Montage4D
ACM I3D’18
To appear in JCGT’18
Social Street View
Best Paper Award
ACM Web3D’16
Geollery
Under Review, ACM CHI’18
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
17
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Video Fields
ACM Web3D’16
Montage4D
ACM I3D’18
JCGT’18
Haptics and Gestures
• VRSurus (CHI EA’16)
• HandSight (ECCVW’14,TACCESS’16,
SIGACCESS’16)
FoveatedRendering
• Kernel Foveated Rendering (I3D’18)
Learning Sketchy Scenes
• Sketchy Scenes (ECCV’18)
• LUCCS (Under Review, TOG’18)
Visualizationand Cryptography
• AtmoSPHERE(CHI EA’15)
• TopicFields
• ARCrypt(Under Review VR’19)
Social Street View
Best Paper Award
ACM Web3D’16
Geollery
Under Review, ACM CHI’19
Under Review, IEEE VR’19
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
HAPTICS IN VR,
VISUAL CRYPTOGRAPHY IN
AR, VISUALIZATION, ETC.
18
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Social Street View
Best Paper Award
ACM Web3D’16
19
Social Street View: Blending Immersive Street Views
with Geotagged Social Media
Ruofei Du and Amitabh Varshney
{ruofei, varshney} @ cs.umd.edu
www.SocialStreetView.com
In Proceedings of the International Conference of Web3D 2016. [Best Paper Award]
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
image courtesy: plannedparenthood.org
Introduction
Social Media
21
image courtesy: huffingtonpost.com
Introduction
Social Media – Reflect real life
23
image courtesy: forbes.com
Introduction
Social Media - Versatility
24
25
Metadata are useful for understanding
spatial relevance, temporal impacts,
relationship amongst users, and the associated events.
EXIF
image courtesy: B&T Magazine
26
Introduction
Meta data - Time of Creation
image courtesy: Brian Clegg
27
Introduction
Meta data - Location of Creation
image courtesy: squarespace.com
28
Introduction
Meta data - Camera Information
image courtesy: naldzgraphics.net
29
image courtesy:
instagram.com,
facebook.com,
twitter.com
Related Work
Linear narrative visualization
30
image courtesy:
pinterest.com
31
Related Work
2D layout
Related Work
Natural Immersive Virtual Reality?
image courtesy: drivenhealthy.com
32
Related Work
Karnath et al. and Loomis et al.
33
Related Work
VR and Visual Memories
34
Related Work
VR and Visual Memories
35
Related Work
VR and Visual Memories
36
Related Work
VR and Visual Memories
37
Immersive virtual environment can improve
social interactions and spatial awareness relates to visual memories.
38
Very little work has been carried out in
designing immersive interfaces that interleave
visual navigation of our surroundings
with social media content
39
Visualizing social media in a 2D world map
Related Work
Visualization of Geo-tagged Information
40
image courtesy: twitterstand.umiacs.umd.edu
Related Work
TwitterStand: News in Tweets
Sankaranarayanan et al. SIGGIS 2009
41
Related Work
Visualization of Geo-tagged Information
42
image courtesy: flickr.com
Related Work
Flickr - World Map
Serdyukov et al. SIGIR 2009
43
Related Work
Visualization of Geo-tagged Information
44
image courtesy: panoramio.com
Related Work
Panoramio - Photos of the World
Cuenca and Manchón, 2005-2015
45
Related Work
Visualization of Geo-tagged Information
46
image courtesy: photostand.umiacs.umd.edu
Related Work
PhotoStand for News Photos
Samet et al. VLDB 2013
47
All these systems are limited by the 2D world map.
Related Work
Visualization of Geo-tagged Information
(cont.)
48
What about 3D solutions?
Related Work
Visualization of Geo-tagged Information
(cont.)
49Registers user-annotated text and images to a particular point in 3D space.
Related Work
Visualization of Geo-tagged Information
(cont.)
50However, both the 3D model and the annotation are predefined for rendering such a scene.
Related Work
Visualization of Geo-tagged Information
(cont.)
51
Related Work
Visualization of Geo-tagged Information
(cont.)
52
Related Work
Visualization of Geo-tagged Information
(cont.)
53
Related Work
Visualization of Geo-tagged Information
(cont.)
54
They use offline 3D reconstruction algorithm or existing 3D models
for registering the photos onto the meshes.
Such methods usually suffer from HOURS of processing time for a single location.
So what about our approach?
Social Street View
Social Street View
Demonstration
The Augmentarium, UMIACS
6000 x 3000 pixels
58
59
60
Conception, architecting & implementation
Social Street View
A mixed reality system that can depict geo-tagged
social media in immersive 3D web environments
61
Blending multiple modalities of
Street View + Social Media
Depth maps, normal maps, and road orientation
GPS coordinates and time creation
62
Enhancing visual augmentation
Maximal Poisson-disk sampling
Evaluated by image saliency metrics
63
Achieving cross-platform compatibility by
WebGL + Three.js
smartphones, tablets, desktop, high-resolution large-
area wide field of view tiled display walls, as well as
head-mounted displays.
64
Technical Challenges?
65
66
Architecture
Social Street View System Flowchart
67
Street View Cars - Cameras, LIDARs and GPS
Image courtesy from Google Street View
image courtesy: Google
68
Tiles of Panoramic Images
Image courtesy from Google Street View
image courtesy: Google
69
image courtesy: Google
70
71
Depth Map
Decoded from Google Maps API v3
72
Normal Map
Decoded from Google Maps API v3
73
Road Orientations
Decoded from Google Maps API v3
74
image courtesy: Instagram
75
Multi-level Strategy for Streaming Social Media
Image courtesy from Instagram users in public domains
image courtesy: Instagram
76
image courtesy: Instagram
77
Data Mining
Multi-level Strategy
… …
8 METERS
16 METERS
1024 METERS
SQL Database
Indexed by B+ Tree
logarithmic insertion & query
78
SQL Database
Indexed by B+ Tree
logarithmic insertion & query
79
Longitude, latitude, timestamps are indexed by B+ trees.
Haversine Formula
Andrew, 1805
80
The distance between social media and street view panorama
Haversine formula, which defines the distance on the surface of a sphere.
81
geotagged social media
spatiotemporal filters custom location query
immersive 360° map
2D mini-map
82
83
How can we render and layout the social media?
Baseline
Random Uniform Sampling
84
Baseline
Random Uniform Sampling
85
Without uniform random sampling
Without uniform
sampling
Accumulation
86
Baseline
Random Uniform Sampling
87
With uniform random sampling
Not preferred
Overlays high saliency regions
88
Add depth map
Remove sky and ground (most)
89
Sky:
Distance Limit:
Add depth map
Remove sky and ground (most)
90
91
Can we ensure each image aligns with the building geometries?
Add normal map
Remove all ground, align images
92
Define the ground:
93
94
Can we further reduce visual clutter and occlusion?
Maximal Poisson-disk Sampling
Gamito et al. Remove visual clutter and occlusion
95
The Poisson-disk distribution is uniform.
Minimum distance between each pair of social media is greater than R.
Terminates the sampling when it reaches the maximal coverage.
Dart-throwing Algorithm
PixelPie by Ip et al. using vertex and fragment shaders
image courtesy: PeterPan23@Wikipedia
96
Pixel-Pie Algorithm
Remove when occlusion occurs
97
Pixel-Pie by Ip. et al. uses GPU depth-testing feature for efficient sampling.
Project Social Media Pictures
By Maximal Poisson-disc Sampling
98
Sampling in regions which are not Sky nor Ground, and with a limitation of depth values;
Place social media as billboards upon the building geometries;
Cast soft shadows as if lightings were 45 degree to the normal vectors.
Sampling Comparison
Remove visual clutter and occlusion
99
Results for Urban Areas
100
101
Results for Urban Areas
Results for Urban Areas
102
103
The algorithm works well in dense urban areas such as the Manhattan District
104
What if there are no buildings in the scene?
Scenic Landscapes
Using orientation of the road
105
We use only the orientations of the road.
Scenic Landscapes
Using the road orientations
106
Scenic Landscapes
Using the road orientations
107
Scenic Landscapes
Using the road orientations
108
Scenic Landscapes
Using the road orientations
109
What if …
Temporal information is used for filtering and rendering?
image courtesy: mindtheproduct.com
110
111
Experiment
100 panoramas, 80,000+ social media
112
Experiment
Nvidia Quadro K6000
113
Experiment
Conditions of panoramic images
Immersive high-resolution screens
Common Consumer-level Displays
114
Initialization Time
Panorama takes a while to load
115
Panoramas takes much more time than social media for initializing a new view.
After Prefetching
¾ - ⅘ time reduced
116
~250msPreload the panoramic texture into the GPU memory:
Initialization Time
Panorama takes a while to load
117
Stitching and shipping panoramas to GPU is the most time-consuming part.
Rendering Time
Almost 58~60 FPS
118
After initialization, the system runs in real time at about 58~60 FPS.
How effective is the Maximal Poisson-disk Sampling
for reducing the visual clutter?
Saliency Map
Hou et al. TPAMI 2011
120
How much social media covers the salient regions of the panorama?
Saliency Map
Hou et al. TPAMI 2011
121
Saliency maps use color, intensity, and orientation contrasts to
estimate where humans will look at.
Saliency Map
Hou et al. TPAMI 2011
122
We prefer the social media to cover less high-saliency regions.
Social Media
Coverage
100 panoramas for each algorithm
123
Social Media
Coverage
100 panoramas for each algorithm
124
The first two algorithms occupy 30% - 50% of the high-saliency regions.
Social Media
Coverage
100 panoramas for each algorithm
125
In the last algorithm, all the images are separated with each other and uniformly distributed on the building surfaces.
The last two algorithms largely reduce the coverage of saliency map to less than 20%.
What could be the potential applications of Social Street View?
127
Stuck in traffic on our way to
Cabo with this awesome view
#roadtrip #cabo #view #mexico
Daniela on Instagram
July 12, 2014
128
129
130
Business Advertising
Museum, restaurant, real-estate ...
131
... dinner started off with
amazing oysters paired with my
favorite Ruinart blanc de blancs
champagne
By frankiextah on Instagram
132
Business Advertising
Museum, restaurant, real-estate ...
133
Social Street View enhances future consumers’ visual memories and
makes it easier for them to seek the “amazing oysters” around this place.
Business Advertising
Museum, restaurant ...
134
Learning Culture
Taierzhuang, Chinese Spring Festivals
135
Crowd-sourced Tourism
136
In conclusion, Social Street View provides a novel solution for
automatically fusing geotagged social media in an immersive 3D environments.
We envision social media and panoramic videos are
significant parts of the big data requirements for VR and AR applications
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
140
Spherical Harmonics for Saliency Computation and Virtual
Cinematography in 360° Videos
Ruofei Du and Amitabh Varshney
{ruofei, varshney} @ cs.umd.edu
Best Student Poster Award at ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.
Under Review, IEEE Transactions on Visualization and Computer Graphics.
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
Motivation
Spherical 360 degree videos
142
image courtesy: https://www.wareable.com/media/images/2016/04/nokia-ozo-1461975451-F4jp-column-width-inline.jpg
Motivation
Spherical 360 degree videos
143image courtesy: http://www.newsshooter.com/wp-content/uploads/2016/11/Screen-Shot-2016-11-01-at-11.23.46-PM.png
Motivation
Spherical 360 degree videos
144image courtesy: http://www.newsshooter.com/wp-content/uploads/2016/11/Screen-Shot-2016-11-01-at-11.23.46-PM.png
Motivation
Spherical 360 degree videos
145image courtesy: http://danielschristian.com/learning-ecosystems/wp-content/uploads/2016/07/bubl-in-classroom-july2016.jpg
Motivation
Spherical 360 degree videos
146
Motivation
Almost 90% of the pixels are outside the
binocular field of view
147
Visual
Medium
Binocular Field of View (FoV)
Ratio Beyond FoV
Horizontal Vertical
Human Eyes 114° 135°
HTC Vive,
Oculus Rift
85° 95°
Samsung
Gear VR
75° 85°
Google
Cardboard
65° 75°
Motivation
Almost 90% of the pixels are outside the
binocular field of view
148
Visual
Medium
Binocular Field of View (FoV)
Ratio Beyond FoV
Horizontal Vertical
Human Eyes 114° 135° 76.25%
HTC Vive,
Oculus Rift
85° 95° 87.53%
Samsung
Gear VR
75° 85° 90.16%
Google
Cardboard
65° 75° 92.48%
Almost 90% of the pixels are outside a typical HMD's field of view (FoV)
Motivation
Understanding where people are likely to
look at
149
Background
Saliency Maps
150
Saliency maps represent the conspicuity at every location
in the visual field by a scalar quantity.
Background
Saliency Maps, Itti et al., PAMI 1998
151
The goal of saliency detection is to model the visual attention system,
inspired by the behavior and the neuronal architecture of the early primate visual system.
Motivation
Saliency Prediction
152
Conventional saliency models can hardly deal with
horizontal clipping and spherical rotations
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
153
The classic Itti et al’s model treats the input as an rectilinear image.
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
154
What if the spherical image is horizontally translated?
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
155
Will the results be the same?
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
156
A simple horizontal clipping causes a false negative result.
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
157
What about a simple spherical rotation?
Motivation
Classic Saliency Approaches are not
consistent (Itti et al.’s model)
158
A spherical rotation may cause both false negative and false positive.
Motivation
How can we build an efficient and consistent
saliency model in the spherical space?
159
The Input Frame
Source
Horizontal
Clipping
Spherical
Rotation
Our ModelItti et al’s Model
How can we build an efficient and consistent saliency model in the spherical space?
161
Spherical Harmonics
Video demo
162
Spherical Harmonics (SH) coefficients can be used for
summarizing the feature maps and computing saliency maps in the 𝑺𝑶(𝟐) space
Algorithm
Evaluating SH Functions
163
Pre-compute SH functions
for every pair of (𝜃, 𝜙)
𝑂(𝑁𝑀𝐿2)
Load to the GPU memory
Several minutes
Algorithm
Extracting Static Features
164
Compute SH coefficients
using prefix sum on the GPU
in real time
𝑂(𝐿2
𝑙𝑜𝑔𝑁𝑀)
Here, 𝑓𝑖,𝑗 is the feature value,
𝐻𝑖,𝑗 is the precomputed value with
SH functions at each pixel
Algorithm
Extracting Static Features
165
Intensity Red-Green Contrast Blue-Yellow Contrast
Algorithm
Evaluating SH coefficients
166
For a single feature map and L = 15 bands of spherical harmonics,
It produces 152 = 225 SH coefficients in real time.
Reconstruction
From 15 bands of SH coefficients
167
Clouds
Mountains
Humans
Caves
Landscape
168
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
The High Band - QTheLowBand-P
Difference
Algorithm
Spherical spectral residual model
169
Our spherical spectral residual (SSR) model only takes
𝑄2 − 𝑃2 = 176 multiplication and summation
Algorithm
Nonlinear normalization
170
We use the non-linear normalization technique (Itti et al.) to
combine different spatial and temporal saliency maps.
171
Results
Visual Comparison Between Itti et al’s
model and our SSR model
172
Results
Visual Comparison Between Itti et al’s
model and our SSR model
173
Results
Timing Comparison Between Itti et
al’s model and our SSR model
174
Resolution
The Average Timing Per Frame
Itti et al. (CPU) SSR (CPU) SSR (GPU)
1920x1080 104.46 ms 21.34 ms 10.81 ms
4096x2048 314.94 ms 48.18 ms 13.20 ms
7680x3840 934.26 ms 69.53 ms 26.58 ms
Virtual Cinematography
How can saliency maps be used?
175
How can saliency maps guide automatic camera control for 360° videos?
176
The most salient objects may vary from frame to frame,
due to the varying occlusions, colors, and self-movement.
Virtual Cinematography
Most salient objects are not reliable
177
An approach that relies on just tracking the most salient objects
may incur rapid motion of the camera, and
worse still, may induce motion sickness in virtual reality.
Virtual Cinematography
Camera jittering
178
SpatioTemporal Model
Virtual Cinematography
179
Ratio of the observed sum of saliency values The virtual camera distance between adjacent frames
Evaluation
Compute 2048 potential camera positions
180
32×64 pairs of camera positions uniformly over the sphere
The optimal position considers both saliency coverage and temporal movement.
Evaluation
Compute 2048 potential camera positions
181
The video frame rate is much smaller than the rendering frame rate
182
Convert the spherical coordinates to quaternions:
Spherical interpolation using the spline curves:
Interpolation
Quaternion + Spline curves
Spline Interpolation
Yellow: Optimal camera position
Blue: Interpolated path
183
184
185
0
20
40
60
80
100
120
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
101
106
111
116
121
126
131
136
141
146
151
156
161
166
171
176
181
186
191
196
201
206
211
216
221
226
231
236
241
246
251
256
261
266
271
276
281
286
291
296
301
306
311
316
321
326
331
336
341
346
351
356
TemporalMotion(diagonaldegrees)
Frame Number
Comparison of Temporal Artifacts Between MaxCoverage and SpatioTemporal Models
MaxCoverage SpatioTemporal
Conclusion
Spherical Harmonics Saliency
186
In conclusion, we have presented a novel GPU-driven pipeline which employs spherical
harmonics to directly compute the saliency maps for 360° videos in the 𝑆𝑂(2) space.
Conclusion
Spherical Harmonics Saliency
187
Our method remains consistent for challenging cases like horizontal clipping, spherical
rotations, and equatorial bias, and can be computed on the GPU in real time.
Conclusion
Spherical Harmonics Saliency
188
We demonstrate the application of using spherical harmonics saliency for virtual
cinematography in 360° videos.
Conclusion
Spherical Harmonics Saliency
189
We present a novel spatiotemporal optimization model to ensure large spatial saliency
coverage and reduce temporal jitters of the camera motion.
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Montage4D
ACM I3D’18
To appear in JCGT’18
190
Montage4D: Interactive Seamless Fusion of
Multiview Video Textures
Ruofei Du†‡, Ming Chuang‡¶
, Wayne Chang‡, Hugues Hoppe‡§
, and Amitabh Varshney†
†Augmentarium and GVIL | UMIACS | University of Maryland, College Park
‡Microsoft Research, Redmond ¶
Apple Inc. §
Google Inc.
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND
Introduction
Fusion4D and Holoportation
Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
RGB Depth
RGB Depth Mask
RGB Depth Mask
RGB Depth Mask
Depth estimation &
segmentation
8 Pods
Capture
SITE A SITE B
Volumetric
fusion
Color
Rendering
Remote
rendering
Mesh,
color,
audio
streams
Network
192
Fusing multiview video textures onto dynamic meshes
with real-time constraint is a challenging task
194Screen recorded with OBS. Computed in real-time from 8 videos.
195
of the participants does not believe the 3D reconstructed person looks real
196
of the participants does not believe the 3D reconstructed person looks real
Screen recorded with OBS. Computed in real-time from 8 videos.
Related Work
3D Texture Montage
197
Related Work
3D Texture Montage
198
Related Work
3D Texture Montage
199
Related Work
3D Texture Montage
200
Related Work
3D Texture Montage
201
Related Work
3D Texture Montage
202
Related Work
3D Texture Montage
203
𝑤𝑖 = 𝑉 ⋅ max 0, ො𝑛 ⋅ ෝ𝑣𝑖
𝛼
Normal Weighted Blending
(200 FPS)
Visibility test
Normal vector
Texture camera view direction
Motivation
Visual Quality Matters
204
Holoportation (Escolano et al. 2016) Montage4D (Du et al. 2018)
Motivation
Visual Quality Matters
Montage4D (Du et al. 2018)
205
Holoportation (Escolano et al. 2016)
What is our approach for real-time seamless
texture fusion?
Workflow
Identify and diffuse the seams
207
Workflow
Identify and diffuse the seams
208
Workflow
Identify and diffuse the seams
209
Workflow
Identify and diffuse the seams
210
What are the causes for the seams?
Motivation
Causes for Seams
212
Self-occlusion (depth seams)
Field-of-View (background seams)
Majority-voting (color seams)
Self-occlusion (Depth Seams)
One or two vertices of the triangle are occluded in the
depth map while the others are not.
Seams
Causes
213
Seams
Causes
214
Depth: 1.3
Depth: 1.4
Depth: 30
Seams
Causes
215
Raw projection mapping results Seams after occlusion test
Majority Voting (Color Seams)
Each vertex is assigned with 8 colors coming from the 8
cameras. These colors are classified into different clusters
in LAB color space with a 0.1 threshold. The mean color
of the largest cluster is named majority voting color.
Seams
Causes
216
Majority Voting (Color Seams)
The triangle vertices have different results of the majority
voting colors, which may be caused by either mis-
registration or self-occlusion.
Seams
Causes
217
Seams
Causes
218
Mean L*A*B Color
Mean L*A*B Color
Mean L*A*B Color
Seams
Causes
219
Raw projection mapping results Seams after occlusion test Seams after majority voting test
Field of View (Background Seams)
One or two triangle vertices lie outside the
camera’s field of view or in the subtracted background
region while the rest are not.
Seams
Causes
220
Seams
Causes
221
Inside
Field-of-View
Inside
Field-of-View
Outside Field-of-View
Seams
Causes
222
Raw projection mapping results Seams after field-of-view test
Seams
Causes
223
224
of the million-level triangles are seams (for each view)
Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
For a static frame, how can we get rid of the
annoying seams at interactive frame rates?
How can we spatially smooth the texture
(weight) field near the seams so that we
cannot see visible seams in the results?
Workflow
Identify and diffuse the seams
227
Geodesics
For diffusing the seams
228
Geodesic is the shortest route between two points on the surface.
Geodesics
For diffusing the seams
229
On triangle meshes, this is challenging because of the computation of tangent directions.
And shortest paths are defined on edges instead of the vertices.
Geodesics
For diffusing the seams
230
We use the algorithm by Surazhsky and Hoppe for computing the approximate geodesics.
The idea is to maintain only 2~3 shortest paths along each edge
to reduce the computational cost.
231
What are the causes for the blurring?
Motivation
Causes for blurring
233
Texture projection errors
Careful calibration + Bundle adjustment
Normal-weighted blending
Imprecise geometries / Noisy point clouds /
Different specular highlights
𝑤𝑖 = 𝑉 ⋅ max 0, ො𝑛 ⋅ ෝ𝑣𝑖
𝛼
Motivation
Causes for blurring
234
Texture projection errors
Normal-weighted blending
View-dependent rendering
𝑤𝑖 = 𝑉 ⋅ max 0, ො𝑣 ⋅ ෝ𝑣𝑖
𝛼
𝑤𝑖 = 𝑉 ⋅ max 0, ො𝑛 ⋅ ෝ𝑣𝑖
𝛼
235
Visibility test
Visibility% of view i
User camera’s view direction
Texture camera view direction
Geodesics
𝑫 𝑣
𝑖 (𝑡)
For a frame at time 𝑡, for each camera 𝑖, for each vertex 𝑣,
We define the Desired Texture Field:
236
237
Workflow
Identify and diffuse the seams
238
Temporal Texture Field
Temporally smooth the texture fields
239
For frame at time 𝑡, for each camera 𝑖, for each vertex 𝑣,
We define the Temporal Texture Field (exponential smoothing)
𝑻 𝑣
𝑖
𝑡 = 1 − 𝜆 𝑻 𝑣
𝑖
𝑡 − 1 + 𝜆𝑫 𝑣
𝑖
(𝑡)
Texture field of the previous frame
Temporal smoothing factor
1
𝐹𝑃𝑆
Temporal Texture
Fields
Transition between views
240
241
Holoportaion
Montage4D
242
Holoportaion
Montage4D
243
244
245
Note: Results use default parameters of Floating Textures, which may not be optimal for our datasets.
Still, for optical flow based approach, it would be better if seam vertices are assigned with less texture weights.
246
With additional computation for seams,
geodesics, and temporal texture fields, is
our approach still in real time?
Exmperiment
Cross-validation
249
Montage4D achieves better quality with over 90 FPS on NVIDIA GTX 1080
• Root mean square error (RMSE) ↓
• Structural similarity (SSIM) ↑
• Signal-to-noise ratio (PSNR) ↑
250
of the participants does not believe the 3D reconstructed person looks real
Experiment
Break-down of a typical frame
251
Most of the time is used in communication between CPU and GPU
In conclusion, Montage4D uses seam identification, geodesic fields,
and temporal texture field to provides a practical texturing solution for
real-time 3D reconstructions. In the future, we envision that Montage4D is useful
for fusing the massive multi-view video data into VR applications like
remote business meeting, remote training, and live broadcasting.
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Video Fields
ACM Web3D’16
Montage4D
ACM I3D’18
To appear in JCGT’18
Haptics
• VRSurus (CHI EA’16)
• HandSight (ECCVW’14,TACCESS’16,
SIGACCESS’16)
FoveatedRendering
• Kernel Foveated Rendering (I3D’18)
Learning Sketchy Scenes
• Sketchy Scenes (ECCV’18)
• LUCCS (Under Review, TOG’18)
Visualization and Cryptography
• AtmoSPHERE (CHI EA’15)
• TopicFields (In Submission to VR’18)
• ARCrypt(In Submission to VR’18)
Social Street View
Best Paper Award
ACM Web3D’16
Geollery
Under Review, ACM CHI’18
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
HAPTICS IN VR,
VISUAL CRYPTOGRAPHY IN
AR, VISUALIZATION, ETC.
253
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Social Street View
Best Paper Award
ACM Web3D’16
Geollery
Under Review, ACM CHI’18
254
Geollery: An Interactive Social Platform in Mixed Reality
Ruofei Du, David Li, and Amitabh Varshney
{ruofei, dli7319, varshney} @ umd.edu
www.Geollery.com
Under Review, ACM CHI Conference on Human Factors in Computing Systems, 2018.
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
Research Question
Social Street View Navigation
256
How to enable user to
virtually walk in Social Street View?
257
How to create a 3D world in real time
where street views are not available?
Challenges
Registration of billboards
258
259
260
Outline
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Video Fields
ACM Web3D’16
261
262
VRSurus: Enhancing Interactivity and Tangibility of
Puppets in Virtual Reality
Ruofei Du and Liang He
University of Maryland, College Park
ACM CHI Conference on Human Factors in Computing Systems, 2015.
4th place in UIST 2014 Student Innovation Contest.
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
Demo
VRSurus
264
Conclusion
Global market constraint for
VR and AR: Weak Content
265
Conclusion
Fusing Multimedia Data Into
Dynamic Virtual Environments
PANORAMAS MULTIVIEW
VIDEOS
SOCIAL MEDIA
MESHES
Video Fields
ACM Web3D’16
Montage4D
ACM I3D’18
To appear in JCGT’18
Haptics and Gestures
• VRSurus (CHI EA’16)
• HandSight (ECCVW’14,TACCESS’16,
SIGACCESS’16)
FoveatedRendering
• Kernel Foveated Rendering (I3D’18)
Learning Sketchy Scenes
• Sketchy Scenes (ECCV’18)
• LUCCS (Under Review, TOG’18)
Visualization and Cryptography
• AtmoSPHERE(CHI EA’15)
• TopicFields (In Submission to VR’18)
• ARCrypt(In Submission to VR’18)
Social Street View
Best Paper Award
ACM Web3D’16
Geollery
Under Review, ACM CHI’18
Spherical Harmonics Saliency
Best Student Poster Award
ACM I3D’18
Under Review, IEEE TVCG’18
HAPTICS IN VR,
VISUAL CRYPTOGRAPHY IN
AR, VISUALIZATION, ETC.
266
Conclusion
Application: Mixed-reality social
platforms
Conclusion
Application: Immersive
surveillance
Conclusion
Application: Immersive
surveillance + digital city
Image courtesy: https://www.docwirenews.com/docwire-pick/future-of-medicine-picks/use-of-holograms-in-medical-training/
Conclusion
Application: Telepresence
Future Directions
The ultimate XR system
Future Directions
Fusing past events
Future Directions
With the present
Future Directions
And look into the future
Future Directions
Change the way we communicate
in 3D and consume the information
Future Directions
Change the way we communicate
in 3D and consume the information
Peer-reviewed Publications
277
1. Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney. Montage4D: Interactive Seamless
Fusion of Multiview Video Textures. Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and
Games (I3D), pp. 124–133, 2018.
2. Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney. Montage4D: Real-time Seamless
Fusion and Stylization of Multiview Video Textures. Under Review. Journal of Computer Graphics Techniques, 2018.
3. Ruofei Du and Amitabh Varshney. Spherical Harmonics for Saliency Computation and Virtual Cinematography in 360°
Videos. [Best Student Poster Award] ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D),
2018. Under Review, IEEE Transactions on Visualization and Computer Graphics, 2018.
4. Ruofei Du and Amitabh Varshney. Social Street View: Blending Immersive Street Views with Geo-Tagged Social
Media. [Best Paper Award] In Proceedings of the 21st International Conference on Web3D Technology, pp. 77–85,
2016.
5. Ruofei Du, David Li, and Amitabh Varshney. Geollery: Designing an Interactive Mirrored World with Geo-tagged
Social Media. Under Review, ACM CHI Conference on Human Factors in Computing Systems, 2018.
6. Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. Kernel Foveated Rendering. Proceedings of the
ACM on Computer Graphics and Interactive Techniques, 1(5), pp. 1–20, 2018.
7. Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao
Zhang. SketchyScene: Richly-Annotated Scene Sketches. In European Conference of Computer Vision (ECCV), pp.
421–436, 2018.
Peer-reviewed Publications
278
8. Ruofei Du, Sujal Bista, and Amitabh Varshney. Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual
Environment. Proceedings of the 21st International Conference on Web3D Technology, pp. 165–172, 2016.
9. Ruofei Du and Liang He. VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality. Proceedings of the
34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI), pp. 2454--2461, 2016.
10. Ruofei Du, Kent Wills, Max Potasznik, and Jon Froehlich. AtmoSPHERE: Representing Space and Movement Using Sand
Traces in an Interactive Zen Garden Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in
Computing Systems, pp. 1627–1632, 2016.
11. Changqing Zou, Haoran Mo, Ruofei Du, Xing Wu, Chengying Gao, and Hongbo Fu. LUCCS: Language-based User-
customized Colorization of Scene Sketches. arXiv preprint, arXiv:1808.10544, Under Review. ACM Transaction on Graphics,
2018.
12. Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David A. Ross, Rama Chellappa, and Jon E. Froehlich.
Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted
Cameras ACM Transactions on Accessible Computing (TACCESS), 9(1):1, 2014.
13. Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon E. Froehlich. The Design and
Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind
European Conference on Computer Vision (ECCV), Workshop on Assistive Computer Vision and Robotics, pp. 615–631, 2014.
14. Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon E. Froehlich. Supporting Everyday
Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch Proceedings of the 17th
International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS 2015), pp. 383–384, 2015.
Acknowledgement
NSF | Nvidia | MPower | UMIACS
279
Acknowledgement
Collaborators, advisors, labmates,
and committee members
280
Ben Cutler
bcutler@microsoft.com
Spencer Fowers
sfowers@microsoft.com
Wayne Chang
wechang@microsoft.com
Sameh Khamis
sameh.khamis@gmail.com
Shahram Izadi
shahram.izadi.uk@gmail.com
Sujal Bista
Sujal@cs.umd.edu
Amitabh Varshney
varshney@cs.umd.edu
Eric Krokos
ekrokos@cs.umd.edu
Hugues Hoppe
hhoppe@gmail.com
Ming Chuang
mingchuang82@gmail.com
Matthias Zwicker
zwicker@cs.umd.edu
Furong Huang
furongh@cs.umd.edu
Committee Augmentarium / Family
MSR
Hsueh-ChienCheng
hccheng@cs.umd.edu
Mentors / Collaborators
Sai Yuan
syuan@umd.edu
Xuetong Sun
xtsunl@cs.umd.edu
Xiaoxu Meng
xmeng525@umiacs.umd.edu
Jon Froehlich
jonf@cs.washington.edu
Leah Findlater
leahkf@uw.edu
Sida Li
sidali@umiacs.umd.edu
Eric Lee
ericlee@umiacs.umd.edu
LiangHe
edigahe@gmail.com
Jesph F. JaJa
josephj@umd.edu
David Li
ericlee@umiacs.umd.edu
Gregory Kramida
algomorph@gmail.com
Jonathan Heagerty
ericlee@umiacs.umd.edu
Shuo Li
li3shuo1@gmail.com
Alexander Rowden
alrowden@teprmail.umd.edu
Thank you
Ruofei Du
ruofei@cs.umd.edu
Committee: Dr. Varshney, Dr. Zwicker, Dr. Huang, Dr. JaJa, and Dr. Chuang
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
Fusing Multimedia Data Into
Dynamic Virtual Environments
Ruofei Du
ruofei@cs.umd.edu
Committee: Dr. Varshney, Dr. Zwicker, Dr. Huang, Dr. JaJa, and Dr. Chuang
THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LABORATORY
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLANDUMIACSGVIL
283
284
285
286
287
288
289
290
Fusing Multimedia Data Into Dynamic Virtual Environments
Fusing Multimedia Data Into Dynamic Virtual Environments
Thank you
With a Starry Night Stylization
Ruofei Du
ruofei@cs.umd.edu
Amitabh Varshney
varshney@cs.umd.edu
Wayne Chang
wechang@microsoft.com
Ming Chuang
mingchuang82@gmail.com
Hugues Hoppe
hhoppe@gmail.com
• www.montage4d.com
• www.duruofei.com
• shadertoy.com/user/starea
• github.com/ruofeidu
• I am graduating December 2018.
Introduction
Fusion4D and Holoportation
image courtesy: Escolano et al. Holoportation: Virtual 3D Teleportation in Real-time (UIST 2016)
Limitations
Holoportation
image courtesy: afterpsychotherapy.com
1.6 Gbps per second
Introduction
Mobile Holoportation
image courtesy: Wayne Chang and Spencer Fowers
Introduction
Mobile Holoportation
image courtesy: Jeff Kramer
30-50 Mbps
298
Geollery: An Interactive Social Platform in Mixed Reality
Ruofei Du, David Li, and Amitabh Varshney
The Augmentarium| UMIACS | University of Maryland, College Park
{ruofei, dli7319, varshney} @ umd.edu
www.Geollery.com

More Related Content

Similar to Fusing Multimedia Data Into Dynamic Virtual Environments

Augmented Reality -A quick surface view
Augmented Reality -A quick surface viewAugmented Reality -A quick surface view
Augmented Reality -A quick surface viewMusfira Ameer
 
Current and Future Trends of Media and Information - Part 2.pptx
Current and Future Trends of Media and Information - Part 2.pptxCurrent and Future Trends of Media and Information - Part 2.pptx
Current and Future Trends of Media and Information - Part 2.pptxMarylieNoran2
 
Networks, Deep Learning (and COVID-19)
Networks, Deep Learning (and COVID-19)Networks, Deep Learning (and COVID-19)
Networks, Deep Learning (and COVID-19)tm1966
 
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...Ruofei Du
 
Challenges on geo spatial visual analytics eurographics
Challenges on geo spatial visual analytics eurographicsChallenges on geo spatial visual analytics eurographics
Challenges on geo spatial visual analytics eurographicsRaffaele de Amicis
 
AR, the TODAY
AR, the TODAYAR, the TODAY
AR, the TODAYJongHyoun
 
Montage4D: Interactive Seamless Fusion of Multiview Video Textures
Montage4D: Interactive Seamless Fusion of Multiview Video TexturesMontage4D: Interactive Seamless Fusion of Multiview Video Textures
Montage4D: Interactive Seamless Fusion of Multiview Video TexturesRuofei Du
 
Imagining a Physical Future for Digital Journalism
Imagining a Physical Future for Digital JournalismImagining a Physical Future for Digital Journalism
Imagining a Physical Future for Digital JournalismDataJournalismUK
 
An Audit of Augmented Reality Technology Utilization
An Audit of Augmented Reality Technology UtilizationAn Audit of Augmented Reality Technology Utilization
An Audit of Augmented Reality Technology UtilizationIRJET Journal
 
09187784d assignment 1
09187784d assignment 109187784d assignment 1
09187784d assignment 1lsgi4321
 
Understanding Users Behaviours in User-Centric Immersive Communications
Understanding Users Behaviours in User-Centric Immersive CommunicationsUnderstanding Users Behaviours in User-Centric Immersive Communications
Understanding Users Behaviours in User-Centric Immersive CommunicationsFörderverein Technische Fakultät
 
A Survey on Different Approaches to Augmented Reality inthe Educational Field
A Survey on Different Approaches to Augmented Reality inthe Educational FieldA Survey on Different Approaches to Augmented Reality inthe Educational Field
A Survey on Different Approaches to Augmented Reality inthe Educational FieldIRJET Journal
 
138290 633676467827677500
138290 633676467827677500138290 633676467827677500
138290 633676467827677500bndrbilli
 
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITY
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITYA L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITY
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITYcsandit
 
VR2015-revision-v7-embedded
VR2015-revision-v7-embeddedVR2015-revision-v7-embedded
VR2015-revision-v7-embeddedYun Suk Chang
 
Augmented Reality And Virtual Reality
Augmented Reality And Virtual RealityAugmented Reality And Virtual Reality
Augmented Reality And Virtual RealityBrandi Gonzales
 
M.Sc. Thesis - Automatic People Counting in Crowded Scenes
M.Sc. Thesis - Automatic People Counting in Crowded ScenesM.Sc. Thesis - Automatic People Counting in Crowded Scenes
M.Sc. Thesis - Automatic People Counting in Crowded ScenesAhmed Gad
 

Similar to Fusing Multimedia Data Into Dynamic Virtual Environments (20)

Augmented Reality -A quick surface view
Augmented Reality -A quick surface viewAugmented Reality -A quick surface view
Augmented Reality -A quick surface view
 
Current and Future Trends of Media and Information - Part 2.pptx
Current and Future Trends of Media and Information - Part 2.pptxCurrent and Future Trends of Media and Information - Part 2.pptx
Current and Future Trends of Media and Information - Part 2.pptx
 
Networks, Deep Learning (and COVID-19)
Networks, Deep Learning (and COVID-19)Networks, Deep Learning (and COVID-19)
Networks, Deep Learning (and COVID-19)
 
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...
 
Challenges on geo spatial visual analytics eurographics
Challenges on geo spatial visual analytics eurographicsChallenges on geo spatial visual analytics eurographics
Challenges on geo spatial visual analytics eurographics
 
AR, the TODAY
AR, the TODAYAR, the TODAY
AR, the TODAY
 
Montage4D: Interactive Seamless Fusion of Multiview Video Textures
Montage4D: Interactive Seamless Fusion of Multiview Video TexturesMontage4D: Interactive Seamless Fusion of Multiview Video Textures
Montage4D: Interactive Seamless Fusion of Multiview Video Textures
 
Imagining a Physical Future for Digital Journalism
Imagining a Physical Future for Digital JournalismImagining a Physical Future for Digital Journalism
Imagining a Physical Future for Digital Journalism
 
An Audit of Augmented Reality Technology Utilization
An Audit of Augmented Reality Technology UtilizationAn Audit of Augmented Reality Technology Utilization
An Audit of Augmented Reality Technology Utilization
 
09187784d assignment 1
09187784d assignment 109187784d assignment 1
09187784d assignment 1
 
Understanding Users Behaviours in User-Centric Immersive Communications
Understanding Users Behaviours in User-Centric Immersive CommunicationsUnderstanding Users Behaviours in User-Centric Immersive Communications
Understanding Users Behaviours in User-Centric Immersive Communications
 
A Survey on Different Approaches to Augmented Reality inthe Educational Field
A Survey on Different Approaches to Augmented Reality inthe Educational FieldA Survey on Different Approaches to Augmented Reality inthe Educational Field
A Survey on Different Approaches to Augmented Reality inthe Educational Field
 
Virtual Art Gallery
Virtual Art GalleryVirtual Art Gallery
Virtual Art Gallery
 
138290 633676467827677500
138290 633676467827677500138290 633676467827677500
138290 633676467827677500
 
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITY
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITYA L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITY
A L -H ARM E XPANSION M OVIE B ASED ON V IRTUAL R EALITY
 
Our 1st Seminar
Our 1st SeminarOur 1st Seminar
Our 1st Seminar
 
Ncseee 24
Ncseee 24Ncseee 24
Ncseee 24
 
VR2015-revision-v7-embedded
VR2015-revision-v7-embeddedVR2015-revision-v7-embedded
VR2015-revision-v7-embedded
 
Augmented Reality And Virtual Reality
Augmented Reality And Virtual RealityAugmented Reality And Virtual Reality
Augmented Reality And Virtual Reality
 
M.Sc. Thesis - Automatic People Counting in Crowded Scenes
M.Sc. Thesis - Automatic People Counting in Crowded ScenesM.Sc. Thesis - Automatic People Counting in Crowded Scenes
M.Sc. Thesis - Automatic People Counting in Crowded Scenes
 

More from Ruofei Du

Geollery: A Mixed Reality Social Media Platform
Geollery: A Mixed Reality Social Media PlatformGeollery: A Mixed Reality Social Media Platform
Geollery: A Mixed Reality Social Media PlatformRuofei Du
 
CTUAAA Summit 2017 Schedule
CTUAAA Summit 2017 ScheduleCTUAAA Summit 2017 Schedule
CTUAAA Summit 2017 ScheduleRuofei Du
 
交大历史与梅竹赛
交大历史与梅竹赛交大历史与梅竹赛
交大历史与梅竹赛Ruofei Du
 
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...Ruofei Du
 
Chinese Caligraphy 品读书法·感悟中华 (2010)
Chinese Caligraphy 品读书法·感悟中华 (2010)Chinese Caligraphy 品读书法·感悟中华 (2010)
Chinese Caligraphy 品读书法·感悟中华 (2010)Ruofei Du
 
Statistics for K-mer Based Splicing Analysis
Statistics for K-mer Based Splicing AnalysisStatistics for K-mer Based Splicing Analysis
Statistics for K-mer Based Splicing AnalysisRuofei Du
 
基于视频的疲劳驾驶检测系统
基于视频的疲劳驾驶检测系统基于视频的疲劳驾驶检测系统
基于视频的疲劳驾驶检测系统Ruofei Du
 
Online Vigilance Analysis Combining Video and Electrooculography Features
Online Vigilance Analysis Combining Video and Electrooculography FeaturesOnline Vigilance Analysis Combining Video and Electrooculography Features
Online Vigilance Analysis Combining Video and Electrooculography FeaturesRuofei Du
 
Deliberately Planning and Acting for Angry Birds with Refinement Methods
Deliberately Planning and Acting for Angry Birds with Refinement MethodsDeliberately Planning and Acting for Angry Birds with Refinement Methods
Deliberately Planning and Acting for Angry Birds with Refinement MethodsRuofei Du
 

More from Ruofei Du (9)

Geollery: A Mixed Reality Social Media Platform
Geollery: A Mixed Reality Social Media PlatformGeollery: A Mixed Reality Social Media Platform
Geollery: A Mixed Reality Social Media Platform
 
CTUAAA Summit 2017 Schedule
CTUAAA Summit 2017 ScheduleCTUAAA Summit 2017 Schedule
CTUAAA Summit 2017 Schedule
 
交大历史与梅竹赛
交大历史与梅竹赛交大历史与梅竹赛
交大历史与梅竹赛
 
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Envi...
 
Chinese Caligraphy 品读书法·感悟中华 (2010)
Chinese Caligraphy 品读书法·感悟中华 (2010)Chinese Caligraphy 品读书法·感悟中华 (2010)
Chinese Caligraphy 品读书法·感悟中华 (2010)
 
Statistics for K-mer Based Splicing Analysis
Statistics for K-mer Based Splicing AnalysisStatistics for K-mer Based Splicing Analysis
Statistics for K-mer Based Splicing Analysis
 
基于视频的疲劳驾驶检测系统
基于视频的疲劳驾驶检测系统基于视频的疲劳驾驶检测系统
基于视频的疲劳驾驶检测系统
 
Online Vigilance Analysis Combining Video and Electrooculography Features
Online Vigilance Analysis Combining Video and Electrooculography FeaturesOnline Vigilance Analysis Combining Video and Electrooculography Features
Online Vigilance Analysis Combining Video and Electrooculography Features
 
Deliberately Planning and Acting for Angry Birds with Refinement Methods
Deliberately Planning and Acting for Angry Birds with Refinement MethodsDeliberately Planning and Acting for Angry Birds with Refinement Methods
Deliberately Planning and Acting for Angry Birds with Refinement Methods
 

Recently uploaded

TORSION IN GASTROPODS- Anatomical event (Zoology)
TORSION IN GASTROPODS- Anatomical event (Zoology)TORSION IN GASTROPODS- Anatomical event (Zoology)
TORSION IN GASTROPODS- Anatomical event (Zoology)chatterjeesoumili50
 
Substances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestSubstances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestAkashDTejwani
 
Principles & Formulation of Hair Care Products
Principles & Formulation of Hair Care  ProductsPrinciples & Formulation of Hair Care  Products
Principles & Formulation of Hair Care Productspurwaborkar@gmail.com
 
Contracts with Interdependent Preferences (2)
Contracts with Interdependent Preferences (2)Contracts with Interdependent Preferences (2)
Contracts with Interdependent Preferences (2)GRAPE
 
famous scientist presentation psp 1st year
famous scientist presentation psp 1st yearfamous scientist presentation psp 1st year
famous scientist presentation psp 1st yearmarwaahmad357
 
Physics Serway Jewett 6th edition for Scientists and Engineers
Physics Serway Jewett 6th edition for Scientists and EngineersPhysics Serway Jewett 6th edition for Scientists and Engineers
Physics Serway Jewett 6th edition for Scientists and EngineersAndreaLucarelli
 
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Sérgio Sacani
 
Human brain.. It's parts and function.
Human brain.. It's parts and function. Human brain.. It's parts and function.
Human brain.. It's parts and function. MUKTA MANJARI SAHOO
 
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky Way
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky WayShiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky Way
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky WaySérgio Sacani
 
Alternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusAlternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusPradnya Wadekar
 
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdf
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdfwepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdf
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdfVishalSuryawanshi31
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptxVaishnaviAware
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPirithiRaju
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxmarwaahmad357
 
IB Biology New syllabus B3.2 Transport.pptx
IB Biology New syllabus B3.2 Transport.pptxIB Biology New syllabus B3.2 Transport.pptx
IB Biology New syllabus B3.2 Transport.pptxUalikhanKalkhojayev1
 
biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsSafaFallah
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptSachin Teotia
 
Cosmetic Excipients: Surfactants Used in cosmetic formulation
Cosmetic Excipients: Surfactants Used in cosmetic formulationCosmetic Excipients: Surfactants Used in cosmetic formulation
Cosmetic Excipients: Surfactants Used in cosmetic formulationDhanshriTikekar
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxAkinrotimiOluwadunsi
 

Recently uploaded (20)

TORSION IN GASTROPODS- Anatomical event (Zoology)
TORSION IN GASTROPODS- Anatomical event (Zoology)TORSION IN GASTROPODS- Anatomical event (Zoology)
TORSION IN GASTROPODS- Anatomical event (Zoology)
 
Substances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening TestSubstances in Common Use for Shahu College Screening Test
Substances in Common Use for Shahu College Screening Test
 
Principles & Formulation of Hair Care Products
Principles & Formulation of Hair Care  ProductsPrinciples & Formulation of Hair Care  Products
Principles & Formulation of Hair Care Products
 
Contracts with Interdependent Preferences (2)
Contracts with Interdependent Preferences (2)Contracts with Interdependent Preferences (2)
Contracts with Interdependent Preferences (2)
 
famous scientist presentation psp 1st year
famous scientist presentation psp 1st yearfamous scientist presentation psp 1st year
famous scientist presentation psp 1st year
 
Physics Serway Jewett 6th edition for Scientists and Engineers
Physics Serway Jewett 6th edition for Scientists and EngineersPhysics Serway Jewett 6th edition for Scientists and Engineers
Physics Serway Jewett 6th edition for Scientists and Engineers
 
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
Legacy Analysis of Dark Matter Annihilation from the Milky Way Dwarf Spheroid...
 
Human brain.. It's parts and function.
Human brain.. It's parts and function. Human brain.. It's parts and function.
Human brain.. It's parts and function.
 
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky Way
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky WayShiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky Way
Shiva and Shakti: Presumed Proto-Galactic Fragments in the Inner Milky Way
 
Alternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusAlternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabus
 
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdf
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdfwepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdf
wepik-exploring-the-dynamics-of-kinetic-energy-20240316075156zSuS.pdf
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptx
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
 
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
Applying Cheminformatics to Develop a Structure Searchable Database of Analyt...
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docx
 
IB Biology New syllabus B3.2 Transport.pptx
IB Biology New syllabus B3.2 Transport.pptxIB Biology New syllabus B3.2 Transport.pptx
IB Biology New syllabus B3.2 Transport.pptx
 
biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibiotics
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.ppt
 
Cosmetic Excipients: Surfactants Used in cosmetic formulation
Cosmetic Excipients: Surfactants Used in cosmetic formulationCosmetic Excipients: Surfactants Used in cosmetic formulation
Cosmetic Excipients: Surfactants Used in cosmetic formulation
 
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptxTHE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
THE HISTOLOGY OF THE CARDIOVASCULAR SYSTEM 2024.pptx
 

Fusing Multimedia Data Into Dynamic Virtual Environments