Navigation-aware adaptive streaming strategies for omnidirectional video

Navigation-Aware Adaptive
Streaming Strategies for
Omnidirectional Video
Silvia Rossi and Laura Toni
UCL - University College London
MMSP 2017 - IEEE 19th International Workshop on Multimedia Signal Processing
Luton (London)

• Introduction
• Motivation and system model
• Problem formulation
• Results
• Conclusion and future work
Outline
2

3
360˚ video streaming
Main challenges
Silvia Rossi

3
• Immersive sensation: zero latency and high quality
Main challenges
Silvia Rossi

3
• Interactive users: unknown requests
Main challenges
Silvia Rossi

3
• Large volume of media content to send
Main challenges
Silvia Rossi

3
• Perceived quality affected by geometry and map projection
Main challenges
Silvia Rossi

• Perceived quality affected by geometry and map projection
3
How to cope with limited network resources and still
optimize the navigation quality?
Main challenges
Silvia Rossi

• [2017] - “Viewport-Adaptive Navigable 360-Degree Video Delivery”
International Conference on Communications (ICC) 2017
Authors: Corbillon, Simon, Devlic and Chakareski
• [2016] - “Adaptive 360 VR video streaming: Divide and conquer!”
International Symposium on Multimedia (ISM) 2016
Authors: Hosseini, Swaminathan
• [2016] - “Optimizing 360 Video Delivery Over Cellular Networks”
All Things Cellular (ATC) 2016
Authors: Qian, Han, Ji and Gopalakrishnan
State-of-the-art
4

[2016] - “Adaptive 360 VR video streaming: Divide and conquer!”
✓ Viewport-adaptive system
✗ Limited degree of freedom in the representations
✗ Transmission strategy overlooked
4
State-of-the-art

State-of-the-art
✓ Tile-based DASH system (high degree of freedom)
✓ Bandwidth saving
✗ Transmission strategy overlooked
4

State-of-the-art
✓ Tiled-based coding (high degree of freedom)
✓ Head movement predictive algorithm
✗ Only a subset of tiles is transmitted
4

• Tile-based adaptive streaming system
• Optimized transmission strategy, taking into account:
-users' navigation paths
-geometry-based MSE as quality metric
5
Proposed solution
To efficiently transmit a 360˚ videos, we propose:
Silvia Rossi

System model
6
Server side
Map
Projection Silvia Rossi

System model
6
Server side
Map
Projection
• : # tilesN
Silvia Rossi

System model
6
Server side
Map
Projection
• : # tiles
• encoding rate set
N
R = {R1, R2, ..., RQ}
Silvia Rossi

System model
6
Server side
Map
Projection
• : # tiles
• encoding rate set
• per-tile representations
N
R = {R1, R2, ..., RQ}
Q
Silvia Rossi

7
System model
Client side
At each downloading time, the
client downloads:
• : chunk of framesTCt

Geometry-based QoE metric
8
To consider content characteristics and scene geometry,
we evaluate viewport quality on the sphere based on
distortion on panoramic content
VPi
VPi

Distortion of viewport
8
VPi• : viewport with center in
x
y
z
VPi
i = (✓i, i)
• : distortion function on the sphereD(✓, )

8
✓i
x
y
z
i
VPi
VPi• : viewport with center in i = (✓i, i)

8
• : surface of viewportSi
Di =
1
SVi
Z ✓i+ ✓v
2
✓i
✓v
2
Z i+ v
2
i
v
2
D(✓, ) sin d✓d
✓i
x
y
z
i
VPi
VPi• : viewport with center in i = (✓i, i)

8
X
n2VPi
Dn(rn)Sn↵i
n• : distortion of block encoded with=
1
SVPi
X
n2VPi
Dn(rn)Sn↵i
n
✓i
x
y
z
i
VPi
Di =
1
SVi
Z ✓i+ ✓v
2
✓i
✓v
2
Z i+ v
2
i
v
2
D(✓, ) sin d✓d
VPi
n

8
X
n2VPi
Dn(rn)Sn↵i
1
SVPi
X
n2VPi
Dn(rn)Sn↵i
n
✓i
x
y
z
i
VPi
Di =
1
SVi
Z ✓i+ ✓v
2
✓i
✓v
2
Z i+ v
2
i
v
2
D(✓, ) sin d✓d
VPi
Dn(rn)Dn(rn)
n

8
X
n2VPi
Dn(rn)Sn↵i
1
SVPi
X
n2VPi
Dn(rn)Sn↵i
n
n 2 i
n
↵n,i• : % of block VPi
Di(r) =
NX
n=1
Dn(rn)↵n,i
bSn,i
bSn,i =
Sbn
SVi
•
✓i
x
y
z
i
VPi
Di =
1
SVi
Z ✓i+ ✓v
2
✓i
✓v
2
Z i+ v
2
i
v
2
D(✓, ) sin d✓d
VPi
Dn(rn)

8
X
n2VPi
Dn(rn)Sn↵i
1
SVPi
X
n2VPi
Dn(rn)Sn↵i
nn
↵n,i• : % of block
Di(r) =
NX
n=1
Dn(rn)↵n,i
bSn,i
bSn,i =
Sbn
SVi
•
✓i
x
y
z
i
VPi
Di =
1
SVi
Z ✓i+ ✓v
2
✓i
✓v
2
Z i+ v
2
i
v
2
D(✓, ) sin d✓d
VPi
Dn(rn) ↵ = 1
↵ = 0.4n 2 iVPi

Navigation-Aware Adaptive Logic
9
D(r) =
TX
t=1
IX
i=1
NX
n=1
Dn(rn) bSn,i↵n,ipt,i.•
Di(r)

9
D(r) =
TX
t=1
IX
i=1
NX
n=1
Di(r)
• Popularity-weighted geometry-based
distortion

9
D(r) =
TX
t=1
IX
i=1
NX
n=1
Dn(rn) bSn,i↵n,ipt,i.• Popularity-weighted geometry-based
distortion
Di(r)

9
D(r) =
TX
t=1
IX
i=1
NX
n=1
Dn(rn) bSn,i↵n,ipt,i.• Popularity-weighted geometry-based
distortion
Di(r)
D(r) =
TX
t=2
IX
i=1
Dt,i(r)pt,i• Navigation-smoothness

9
D(r) =
TX
t=2
IX
i=1
Dt,i(r)pt,i
D(r) =
TX
t=1
IX
i=1
NX
n=1
Dn(rn) bSn,i↵n,ipt,i.
• : encoded rate of blockrn n
• : estimated channel capacity
distortion
• Navigation-smoothness
min
r
Duser(r) = D(r) + D(r)
s.t.
NX
n=1
rn  C
C

• : encoded rate of block nrn
•• Navigation-smoothness
distortion
min
r
s.t.
NX
n=1
rn  C
D(r) =
TX
t=2
IX
i=1
Dt,i(r)pt,i
D(r) =
TX
t=1
IX
i=1
NX
n=1
• : estimated channel capacityC
8
min
,y
X
t
X
i
"
X
n
X
q
xn,q n,q
cSn↵n,i +
X
j2N (i)
yi,jpt 1,j
#
pt,i
s.t.
X
q
n,q = 1, 8n 2 [1, N],
X
n
X
q
✓
bn
xn,q an
cn
◆
n,q  C 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
!
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
9

distortion
min
r
s.t.
NX
n=1
rn  C
D(r) =
TX
t=2
IX
i=1
Dt,i(r)pt,i
D(r) =
TX
t=1
IX
i=1
NX
n=1
8
min
,y
X
t
X
i
"
X
n
X
q
xn,q n,q
cSn↵n,i +
X
j2N (i)
yi,jpt 1,j
#
pt,i
s.t.
X
q
n,q = 1, 8n 2 [1, N],
X
n
X
q
✓
bn
xn,q an
cn
◆
n,q  C 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
!
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
Binary variable decision
9

distortion
min
r
s.t.
NX
n=1
rn  C
D(r) =
TX
t=2
IX
i=1
Dt,i(r)pt,i
D(r) =
TX
t=1
IX
i=1
NX
n=1
8
min
,y
X
t
X
i
"
X
n
X
q
xn,q n,q
cSn↵n,i +
X
j2N (i)
yi,jpt 1,j
#
pt,i
s.t.
X
q
n,q = 1, 8n 2 [1, N],
X
n
X
q
✓
bn
xn,q an
cn
◆
n,q  C 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
yi,j
X
n
X
q
xn,q n,q
cSn(↵n,i ↵n,j)
!
8t 2 [1, T], 8i 2 [1, I], 8j 2 N(i), 8n 2 [1, N]
Binary variable decision Auxiliary variable
9

10
Simulations: Settings
“Rollercoaster” “Timelapse NY”

•Resolution: 3840x2048
•Frame-rate: 30fps
•Fast-moving
•One main FoA
•Slow-moving
•More main FoAs
10

•Fast-moving
•One main FoA
•Slow-moving
•More main FoAs
10
• Chunk duration : 2s
• Tile sizes : [256, 512, 680] pixels
• 15 per tile-representations
• Heatmap evaluates via software

Simulations: Results
Video ”Rollercoaster“ with 𝞴 = 1 and 100 users
11

12
Simulations: Results
Video ”Timelapse NY“ with 𝞴 = 1 and 100 users

13
Conclusion
• Novel navigation-aware strategies for 360º video adaptive
streaming
• Problem formulation based on geometry and users'
interactivity
• Proposed ILP solving method
• Comparison with baseline methods
Future works
• Prediction of users' navigation paths
• Tile size, downloading time, storage optimization

Thank you for your attention!
➡ Questions ?

Navigation-aware adaptive streaming strategies for omnidirectional video

Recommended

Recommended

More Related Content

Similar to Navigation-aware adaptive streaming strategies for omnidirectional video

Similar to Navigation-aware adaptive streaming strategies for omnidirectional video (20)

Recently uploaded

Recently uploaded (20)

Navigation-aware adaptive streaming strategies for omnidirectional video