Presentation about a project where we developed a video player which displays a tapestry used to summarize a video

1. VIDEO SUMMARIZATION USING TAPESTRY
MULTIMEDIA SYSTEMS DESIGN PROJECT
TEAM:
• PRANAV GHATE
• RENU HIREMATH
1

2. PROBLEM STATEMENT
§ To implement a video summarization algorithm that produces a “summary image” summarizing the
video content
§ Furthermore, given a video and an audio file, you are required to design a user interface that can
§ display the video/audio stream in synchronization
§ given the “summary image”, allows a user to interact with this summary image to appropriately jump to the
correct location in the video in order provide an effective visual browsing interface.
§ To display the “summary image” at different levels of hierarchy
2

3. ALGORITHM USED FOR KEY FRAMES DETECTION
3
§ Color Histogram Differences between Consecutive Frames
§ Threshold calculated based on mean and standard deviation of difference values. Different scaling factors
give different results.
§ Calculates the color uniformity of each frame and checks if the amount of color in the next frame is similar or
not.
§ Removes a large number of false positives which were a result of a more naive absolute pixel difference
method.

4. ALGORITHM USED FOR KEY FRAMES DETECTION…
§ Edge Detection
§ Perform edge detection using the Sobel Operator
§ Count the number of pixels which are considered as part of the edges
§ When a drastic change in the number of pixels is seen, it is considered to be a different shot, and hence the
latest frame is picked up as a key frame
4

5. ALGORITHM USED FOR TAPESTRY CREATION
§ Seam Carving performed on all key frames before creating the tapestry
§ Criteria for frame selection for the different level of hierarchies
§ Histogram
§ Edge Detection
§ Different sizes are used to display the frames to highlight the zoom action
§ Every frame in the tapestry is clickable and starts playing the video from that frame
5

6. ALGORITHM USED FOR TAPESTRY CREATION…
§ Level1
§ All the key frames are of the same size
§ Level2
§ Parameters
§ List1 : Frames in Level1
§ List2 : Frames around the frame we zoomed into
§ Pos : Position of the frame we zoomed into w.r.t. level1 tapestry
§ Level3
§ Parameters
§ List1 : Frames in Level1
§ List2 : Frames around the frame we zoomed into at Level1
§ List3 : Frames around the frame we zoomed into at Level2
§ Pos1 : Position of the frame we zoomed into w.r.t. level1 tapestry
§ Pos2 : Position of the frame we zoomed into w.r.t. level2 tapestry
6

7. SYNCING OF AUDIO AND VIDEO
§ JavaFX guaranteed a main thread running in the background for the audio and video to run
continuously
§ The JavaFX Audio and Clip libraries allowed us to divide the .wav files into frames
§ Video was streamed frame-by-frame from the .rgb file through BufferedImage and JavaFX ImageView
7

8. PROBLEMS FACED
§ Different threshold values were to used for the different videos
§ Parameters of Seam Carving were adjusted appropriately
8

9. DEV ENVIRONMENT DETAILS
§ NetBeans
§ Apache Commons IO and Media Libraries
§ JavaFX Libraries
§ Java JDK
9

10. SCREENSHOTS
10

11. VIDEO PLAYER
11

12. Indexed and Clickable Regions in Tapestry
12

13. LEVEL 1 TAPESTRY – USC VILLAGE
13

14. LEVEL 2 TAPESTRY – USC VILLAGE
14

15. LEVEL 3 TAPESTRY – USC VILLAGE
15

16. LEVEL 1 TAPESTRY – DISNEY
16

17. LEVEL 2 TAPESTRY – DISNEY
17

18. LEVEL 3 TAPESTRY – DISNEY
18

19. REFERENCES
§ “Video Tapestries with Continuous Temporal Zoom”, Connelly Barnes, Dan B Goldman, Eli
Shechtman, and Adam Finkelstein, ACM Transactions on Graphics (Proc. SIGGRAPH) 29(3), August
2010
§ “An Interactive Comic Book Presentation for Exploring Video”, John Boreczky , Andreas Girgensohn ,
Gene Golovchinsky , Shingo Uchihashi, Proceedings of the SIGCHI conference on Human Factors in
Computing Systems, p.185-192, April 01-
§ “Seam carving for content-aware image resizing” , Shai Avidan , Ariel Shamir, ACM Transactions on
Graphics (TOG), v.26 n.3, July 2007 06, 2000, The Hague, The Netherlands
19

20. THANK YOU!
20