Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019. Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.

1. Project Geollery.com: Reconstructing a Live Mirrored World
With Geotagged Social Media
Ruofei Du†
, David Li†
, and Amitabh Varshney
{ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | Web3D 2019, Los Angeles, USA
UMIACS THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Project Geollery.com: Reconstructing a Live Mirrored World
With Geotagged Social Media

2. image courtesy: plannedparenthood.org
Introduction
Social Media
2

3. image courtesy: huffingtonpost.com
Introduction
Social Media + Topics
3

4. Motivation
Social Media + XR
4

5. image courtesy:
instagram.com,
facebook.com,
twitter.com
Motivation
Social Media + XR
5

6. Motivation
2D layout
image courtesy:
pinterest.com
6

7. Motivation
Immersive Mixed Reality?
image courtesy:
viralized.com
7

8. Motivation
Pros and cons of the classic
8

9. Motivation
Pros and cons of the classic
9

10. Related Work
2D Geospatial Visualization
10

11. Related Work
2D Geospatial Visualization
11

12. Related Work
3D Geospatial Visualization
12

13. Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
13

14. Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
14

15. Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
15

16. Related Work
Social Street View, Du and Varshney
Web3D 2016 Best Paper Award
16

17. Related Work
VirtualOulu: Toni Alatalo et al.
Web3D 2016
17

18. Related Work
Virtual Oulu, Kukka et al.
CSCW 2017
18

19. Related Work
3D Visual Popularity
Bulbul and Dahyot, 2017
19

20. Related Work
Immersive Trip Reports
Brejcha et al. UIST 2018
20

21. Related Work
High Fidelity, Inc.
21

22. Related Work
Facebook Spaces, 2017
22

23. Geollery
A Mixed-Reality Social Media
Platform, CHI 2019
23

24. Geollery.com
A Mixed-Reality Social Media
Platform
24

25. What's Next?
Research Question ⅓
Du et al. Geollery, CHI 2019.
What may a social media
platform look like in mixed reality?
25

26. What's Next?
Research Question ⅔
Du et al. Geollery, CHI 2019.
What if we could allow social media
sharing in a live mirrored world?
26

27. What's Next?
Research Question 3/3
Du et al. Geollery, CHI 2019.
What use cases can we benefit
from social media platform in XR?
27

28. System Overview
Geollery V1 Workflow
28

29. Design Space
Geollery vs. Social Street View
Du et al. Geollery, CHI 2019.
29

30. User Study
Quantitative Evaluation
Du et al. Geollery, CHI 2019.
30

31. Insights
What we learned
31
High-quality content and seed users play key roles

32. Insights
What we learned
32
Interactivity and panoramic textures increase immersion.

33. [I will use it for] exploring new places. If
I am going on vacation somewhere, I
could immerse myself into the location.
If there are avatars around that area, I
could ask questions.
P1 / M
33

34. I think it (Geollery) will be useful for
families. I just taught my grandpa how
to use Facetime last week and it would
great if I could teleport to their house
and meet with them, then we could
chat and share photos with our
avatars.
P2 / F
34

35. What if we could reconstruct a high-quality, all
textured, walkable mirrored world with
geotagged social media in real time?
35

36. Geollery.com
v2: a major leap
36

37. System Overview
Geollery Workflow
37

38. System Overview
Geollery Workflow
38

39. System Overview
Given (latitude, longitude)
39

40. System Overview
2D Map Data
40

41. System Overview
2D Map Data
41

42. System Overview
+Avatar
42

43. System Overview
+Avatar +Trees +Clouds
43

44. System Overview
+Avatar +Trees +Clouds +Night
44

45. System Overview
Street View Panoramas
45

46. System Overview
Street View Panoramas
46

47. System Overview
Street View Panoramas
47

48. System Overview
Geollery Workflow
48

49. System Overview
Geollery Workflow
49

50. System Overview
Geollery Workflow
50

51. System Overview
Geollery Workflow
51
All data we used is publicly and widely available on the Internet.

52. Rendering Pipeline
Close-view Rendering
52

53. Rendering Pipeline
Initial spherical geometries
53

54. Rendering Pipeline
Depth correction
54

55. Rendering Pipeline
Intersection removal
55

56. Rendering Pipeline
Texturing individual geometry
56

57. Rendering Pipeline
Texturing with alpha blending
57

58. Rendering Pipeline
Rendering result in the fine
detail
58

59. Rendering Pipeline
Rendering result in the fine
detail
59

60. Rendering Pipeline
Rendering result in the fine
detail
60

61. Rendering Pipeline
Close-view Rendering
61

62. Rendering Pipeline
Gap Alignment
62

63. Rendering Pipeline
Gap Alignment
63

64. Rendering Pipeline
Gap Alignment
64

65. Rendering Pipeline
Seam Blending
65

66. Rendering Pipeline
Seam Blending
66

67. Rendering Pipeline
Seam Blending
67

68. Rendering Pipeline
Street View vs. Satellite Images
68

69. Rendering Pipeline
Street View vs. Satellite Images
69

70. Rendering Pipeline
Street View vs. Satellite Images
70

71. Rendering Pipeline
Gaussian Filtering
71

72. Rendering Pipeline
Gaussian Filtering
72

73. Rendering Pipeline
Gaussian Filtering
73

74. Rendering Pipeline
Occlusion Test
74

75. Rendering Pipeline
Occlusion Test
75

76. Rendering Pipeline
Occlusion Test
76

77. LRU Cache
Least Recently Used
77
5 adjacent street views are cached while users are walking.
Each geometry has 131,074 vertices to be processed by the GPU

78. Limitations
Inaccurate depth maps etc.
78

79. Deployment
Geollery.com
79

80. Rendering Pipeline
Experimental Features
80

81. Geollery/Social Street View has its own set of
distinct offerings, as it is anchored within
real-world settings, just mapped onto VR, whereas
these are definitely more ‘fantasy’ type of arenas.
In that way, as you have already done, I think there
are multitude game challenges/tasks/feedback, like
the balloons, to add in!
Email feedback from pilot users
81

82. I think it’d be cool if you could see posts by people
in real time, along with the establishment they’re in
(like someone tweeting from inside McDonald’s or
a movie theater), if that makes sense. Sort of like
checking in to a place on Facebook
Email feedback from pilot users
82

83. An interactive rendering pipeline of
Fusing 360° Panoramas
at two levels of detail.
83

84. Contributing a large-scale real-time system to
Reconstruct a Mirrored World
without the prior knowledge of any 3D models but only street view
images and depth maps, which may be estimated from deep learning
pipeline etc.
84

85. Establishing a web-based platform at
Geollery.com
for visualizing geotagged social media in a collaborative
mixed-reality setting.
85

86. Discussion
Use Case: Audio Tour
86
...

87. Discussion
Taking the Feedback:
Geollery v2, Web3D & VR 2019
87
...

88. Discussion
Taking the Feedback
88

89. Challenge
Global Market Restraint:
Weak Content for XR
89

90. Research Goal
Fuse the information from
physical and virtual world
90
ACM Web3D '19

91. Future Directions
The Ultimate XR Platform
91

92. Future Directions
Fuses Past Events
92

93. Future Directions
With the present
93

94. Future Directions
And look into the future
94

95. Future Directions
Change the way we
communicate in 3D and
consume the information
95

96. Future Directions
Consume the information
throughout the world
96

97. Acknowledgement
NSF | MPower | UMIACS
97

98. Acknowledgement
Coauthors
98

99. Thank you!
Ruofei Du, David Li, and Amitabh Varshney
{ruofei, dli7319, varshney}@cs.umd.edu | www.Geollery.com | CHI 2019
UMIACS THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Thank you!

100. Project Geollery.com: Reconstructing a Live Mirrored World
With Geotagged Social Media
Ruofei Du, David Li, and Amitabh Varshney
{ruofei, dli7319, varshney}@umiacs.umd.edu | www.Geollery.com | CHI 2019 | Demo at D-2 (INT-40)
UMIACS THE AUGMENTARIUM
VIRTUAL AND AUGMENTED REALITY LAB
AT THE UNIVERSITY OF MARYLAND
COMPUTER SCIENCE
UNIVERSITY OF MARYLAND, COLLEGE PARK
Project Geollery.com: Reconstructing a Live Mirrored World
With Geotagged Social Media

101. User Study
Background Interview (5 min)
101

102. User Study
Quantitative Evaluation
102

103. User Study
Quantitative Evaluation
103

104. User Study
Post-interview
104

105. Post Interview
Question 1/3
Suppose that we have a polished 3D social media
platform like Geollery or Social Street View, would you
like to use it? If so, how much time would you like to
spend on it?
105

106. Post Interview
Question 1/3
106

107. I would like to use it every day
when I go to work, or travel
during weekends.
P6 / F
107

108. If it’s not distracting like
Facebook and Instagram, I
would use it every day on a
couple of things.
P17 / F
108

109. I am a follower on most social
media sites. I would only join a
3D social media platform once
my friends are there.
P4 / M
109

110. If my friends are all on this, I
can see myself spend a couple
of hours every week.
P12 / M
110

111. I don’t think I will use this. I
prefer to use Yelp to see
comments [of nearby
restaurants]
P12 / M
111

112. Post Interview
Question 2/3
Can you imagine your use cases for Geollery and Social
Street View? What would you like to use 3D social media
platforms for?
112

113. I would like to use it for the food in
different restaurants. I am always
hesitating of different restaurants. It
will be very easy to see all restaurants
with street views. In Yelp, I can only see
one restaurant at a time.
P6 / F
113

114. [I will use it for] exploring new places. If
I am going on vacation somewhere, I
could immerse myself into the location.
If there are avatars around that area, I
could ask questions.
P1 / M
114

115. I think it (Geollery) will be useful for
families. I just taught my grandpa how
to use Facetime last week and it would
great if I could teleport to their house
and meet with them, then we could
chat and share photos with our
avatars.
P2 / F
115

116. … for communicating with my families,
maybe, and distant friends, [so] they
can see New York. And, getting to
know more people, connecting with
people based on similar interests.
P2 / F
116

117. Post Interview
Question 3/3
If you were a designer or product manager for Geollery
or Social Street View, what features would you like to
add to the systems?
117

118. A mapping of the texture,
high-resolution texture, will be
great.
P12 / M
118

119. if there is a way to unify the
interaction between them,
there will be more realistic
buildings [and] you could have
more roof structures. Terrains
will be interesting to add on.
P18 / M
119

120. I would like to see kitties and
puppies running around, and
birds flying in the air
P13 / F
120

121. I could also add a bike, add a
vehicle, a motorcycle in
Geollery, this will add some
fun.
P17 / F
121