SlideShare a Scribd company logo
1 of 89
Download to read offline
Photo Archive

Image Analysis
John Resig	

March 2014	

http://ejohn.org/research/
Ukiyo-e.org
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
Frick Art Reference Library
Photoarchive
• Frick Art Reference Library Photoarchive
established in 1920.	

• Collects photographs of works of art from around
the world, including sponsoring own photography. 	

• Traditionally image analysis has not been applied to
photo archives.	

• The Frick Photoarchive was interested in
exploring image analysis more after seeing

Ukiyo-e.org.
Test case:Anonymous
Italian Art Photographs
• Anonymous Italian Art photographs are
small part (~18,000) of over 1.2 million
images in the Frick Photoarchive.	

• Very few color photographs (makes most
image analysis difficult, good test case).	

• Analysis was performed by myself in
collaboration with the Frick Photoarchive.
Digital Anonymous
Italian Art
• Early digitization project.	

• At the time there was no in-house digital lab
at the Frick Photoarchive.	

• Digitization self-funded and executed by an
outside institution.	

• When new images were added they were
not “merged” -- too hard with no
attribution.
Image Analysis
• I wrote custom software to automate
matches using MatchEngine (a TinEye
service).	

• Extremely effective image matching
technology.	

• Matches same images and portions of
images, even in black-and-white.
NYARC 2014: Frick/Zeri Results
MatchEngine
image1.jpg
image2.jpg
image3.jpg
imageN.jpg
Upload Images to MatchEngine
.
.
.
.
.
.
MatchEngine
image1.jpg
image2.jpg
Retrieve Image Matches from MatchEngine
image9.jpg
Matches
image2.jpg
image1.jpg
Matches
Match	

Database
Archive (e.g. Frick Art Reference Library)
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Artwork
Confirming a Known Match
ArtworkArtwork
Discovering a New Match
5%
3%
92%
Artworks with no match Confirms a known match Matches a new work
Frick Artworks That Have A Match
Similar Images
Different photo, same work of art.
Similar Images
Different photo, slightly different cropping.
Similar Images
Different photo, dramatically different lighting.
Alternate Images
Partial Image vs. Full Image
Alternate Images
Partial Image vs. Much Larger Image
Alternate Images
Color vs. Black-and-White
Conservation
Conservation
Repairs and possibly removal of later additions.
Conservation
Conservation
Possibly just a drastic difference in lighting?
Conservation
Analysis even spots dramatic conservation work.
Copies
Copies
Copies
Copies
Copies
Copies
Digitization Errors
Arms with Folded Hands
Castello sforzesco, Milan.
Female Head

Gabinetto disegni e stampe
degli Uffizi, Florence.
Work doesn’t match
description, wrong cataloging.
Still Life with a Bottle, a Plate, a Mortar and
Pestle...

Virgin Enthroned Nursing Christ, Between
Two Saints

S. Biovenale, Orvieto.
Work doesn’t match description, wrong cataloging.
2%4%
4%
26%
31%
34%
Similar Image Wrong Image Alternate Image Ambiguous Image
Different Works Conservation
Types of New Matches Discovered
Measuring Image
Analysis Efficacy
Artwork
Confirming a Known Match
Artwork
Impossible to Confirm Match
Artworks With Multiple Photos
47%
33%
20%
Failed Match Successful Match No Possible Matches
# of Match Successes and Failures by Type
0
80
160
240
320
Same Similar Alternate Conservation 3D Negative
Success Failure
Similar Images
Similar Images
Alternate Images
Alternate Images
Likelihood of a match by

% of image overlap
0%
25%
50%
75%
100%
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Failed Alternate Matches Successful Alternate Matches Successful Other Matches
% of Image Overlapping
Likelihood of an ‘Alternate Image’
match by % of image overlap
9%
91%
Failure Success
86%
14%
Failure Success
Under 30% Overlap Over 30% Overlap
Conservation
3D Works
3D Works
Negative Images
Overall MatchEngine Quality
62%
38%
Failure Success
88%
12%
Failure Success
All Potential Matches No: Under 30% Overlap, 3D, or Negative
Frederico Zeri
Foundation
• University of Bologna	

• 290,000 photos of artworks from national
archives, private archives, museums, etc.
Zeri 15thC Italian Art
• Digitized 15th century Italian art	

• Persumably some overlap with the Frick’s
Anonymous Italian Art archive
Image-to-Artwork Ratio
0
10000
20000
30000
40000
Frick Zeri
17,929
14,284
38,511
18,548
Images Artworks
3%
36%
61%
Artworks with no match Confirms a known match Matches a new work
Zeri Artworks That Have A Match
Image Analysis Matches
0
5500
11000
16500
22000
Frick Zeri
738815
20,051
1,053
Same Artwork Different Artwork
Image Detail
Image Detail
Same Piece, Similar Style
Same Piece, Similar Style
Similar/Copies
NYARC 2014: Frick/Zeri Results
What Happens When
We Combine Archives?
• Discover new relationships with the other
archive	

• Discover new relationships inside our own
archive
Frick Photoarchive
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Zeri Archive
Artwork
Image
Image
Image
Artwork
Image
Image
Image
Frick-Zeri Image Matches
18,548

Images
38,511 Images
1,117 Images 1,442 Images
Frick-Zeri Artwork Matches
14,284

Artworks
17,929 Artworks
855 Artworks (6%) 938 Artworks (5%)
MergeVerification
• Previously, 77 artworks were manually
matched by a researcher at the Zeri
foundation	

• Can use to verify the quality of the matches
MergeVerification
16%
84%
Same Artwork Failed to Match
Lighting Differences
Tiny Details
Merging Works!
• 84% success confirming known
relationships (with a small sample)	

• Only 6% of computer-discovered matches
were also found by a human.	

• (55 human matches vs. 938 computer
matches)
Graph Analysis
Frick 420
420
Zeri 15836
42091
42090
Frick 417
417
42092
?
NYARC 2014: Frick/Zeri Results
Finding new Links
• Found 14 new Frick links via Zeri	

• Found 236 new Zeri links via Frick
Frick 347
347
Zeri 12227
33526 33527
Frick 348
348
33525 33528
?
NYARC 2014: Frick/Zeri Results
Frick 8131
Zeri 23402
57136
Frick 8132
8132a 8132
57129
57134
Zeri 23405
5713057137
57138
8131a 8131
?
NYARC 2014: Frick/Zeri Results
NYARC 2014: Frick/Zeri Results
Conclusion
• It becomes possible to merge photo archives,
even though they’re in different langauges.	

• New relationships are discovered in the
process.	

• The synergy of combining multiple archives
reveals even more information.	

• All of this is completely automated!
Contact
• John Resig	

• jeresig@gmail.com	

• http://ejohn.org/research/	

• http://ukiyo-e.org/

More Related Content

More from jeresig

Does Coding Every Day Matter?
Does Coding Every Day Matter?Does Coding Every Day Matter?
Does Coding Every Day Matter?jeresig
 
Accidentally Becoming a Digital Librarian
Accidentally Becoming a Digital LibrarianAccidentally Becoming a Digital Librarian
Accidentally Becoming a Digital Librarianjeresig
 
2014: John's Favorite Thing (Neo4j)
2014: John's Favorite Thing (Neo4j)2014: John's Favorite Thing (Neo4j)
2014: John's Favorite Thing (Neo4j)jeresig
 
Computer Vision as Art Historical Investigation
Computer Vision as Art Historical InvestigationComputer Vision as Art Historical Investigation
Computer Vision as Art Historical Investigationjeresig
 
Hacking Art History
Hacking Art HistoryHacking Art History
Hacking Art Historyjeresig
 
Using JS to teach JS at Khan Academy
Using JS to teach JS at Khan AcademyUsing JS to teach JS at Khan Academy
Using JS to teach JS at Khan Academyjeresig
 
Applying Computer Vision to Art History
Applying Computer Vision to Art HistoryApplying Computer Vision to Art History
Applying Computer Vision to Art Historyjeresig
 
Applying Computer Vision to Art History
Applying Computer Vision to Art HistoryApplying Computer Vision to Art History
Applying Computer Vision to Art Historyjeresig
 
Introduction to jQuery (Ajax Exp 2006)
Introduction to jQuery (Ajax Exp 2006)Introduction to jQuery (Ajax Exp 2006)
Introduction to jQuery (Ajax Exp 2006)jeresig
 
jQuery Recommendations to the W3C (2011)
jQuery Recommendations to the W3C (2011)jQuery Recommendations to the W3C (2011)
jQuery Recommendations to the W3C (2011)jeresig
 
jQuery Open Source Process (RIT 2011)
jQuery Open Source Process (RIT 2011)jQuery Open Source Process (RIT 2011)
jQuery Open Source Process (RIT 2011)jeresig
 
jQuery Open Source Process (Knight Foundation 2011)
jQuery Open Source Process (Knight Foundation 2011)jQuery Open Source Process (Knight Foundation 2011)
jQuery Open Source Process (Knight Foundation 2011)jeresig
 
jQuery Mobile
jQuery MobilejQuery Mobile
jQuery Mobilejeresig
 
jQuery Open Source (Fronteer 2011)
jQuery Open Source (Fronteer 2011)jQuery Open Source (Fronteer 2011)
jQuery Open Source (Fronteer 2011)jeresig
 
Holistic JavaScript Performance
Holistic JavaScript PerformanceHolistic JavaScript Performance
Holistic JavaScript Performancejeresig
 
New Features Coming in Browsers (RIT '09)
New Features Coming in Browsers (RIT '09)New Features Coming in Browsers (RIT '09)
New Features Coming in Browsers (RIT '09)jeresig
 
Introduction to jQuery (Ajax Exp 2007)
Introduction to jQuery (Ajax Exp 2007)Introduction to jQuery (Ajax Exp 2007)
Introduction to jQuery (Ajax Exp 2007)jeresig
 
Advanced jQuery (Ajax Exp 2007)
Advanced jQuery (Ajax Exp 2007)Advanced jQuery (Ajax Exp 2007)
Advanced jQuery (Ajax Exp 2007)jeresig
 
JavaScript Library Overview (Ajax Exp West 2007)
JavaScript Library Overview (Ajax Exp West 2007)JavaScript Library Overview (Ajax Exp West 2007)
JavaScript Library Overview (Ajax Exp West 2007)jeresig
 
Meta Programming with JavaScript
Meta Programming with JavaScriptMeta Programming with JavaScript
Meta Programming with JavaScriptjeresig
 

More from jeresig (20)

Does Coding Every Day Matter?
Does Coding Every Day Matter?Does Coding Every Day Matter?
Does Coding Every Day Matter?
 
Accidentally Becoming a Digital Librarian
Accidentally Becoming a Digital LibrarianAccidentally Becoming a Digital Librarian
Accidentally Becoming a Digital Librarian
 
2014: John's Favorite Thing (Neo4j)
2014: John's Favorite Thing (Neo4j)2014: John's Favorite Thing (Neo4j)
2014: John's Favorite Thing (Neo4j)
 
Computer Vision as Art Historical Investigation
Computer Vision as Art Historical InvestigationComputer Vision as Art Historical Investigation
Computer Vision as Art Historical Investigation
 
Hacking Art History
Hacking Art HistoryHacking Art History
Hacking Art History
 
Using JS to teach JS at Khan Academy
Using JS to teach JS at Khan AcademyUsing JS to teach JS at Khan Academy
Using JS to teach JS at Khan Academy
 
Applying Computer Vision to Art History
Applying Computer Vision to Art HistoryApplying Computer Vision to Art History
Applying Computer Vision to Art History
 
Applying Computer Vision to Art History
Applying Computer Vision to Art HistoryApplying Computer Vision to Art History
Applying Computer Vision to Art History
 
Introduction to jQuery (Ajax Exp 2006)
Introduction to jQuery (Ajax Exp 2006)Introduction to jQuery (Ajax Exp 2006)
Introduction to jQuery (Ajax Exp 2006)
 
jQuery Recommendations to the W3C (2011)
jQuery Recommendations to the W3C (2011)jQuery Recommendations to the W3C (2011)
jQuery Recommendations to the W3C (2011)
 
jQuery Open Source Process (RIT 2011)
jQuery Open Source Process (RIT 2011)jQuery Open Source Process (RIT 2011)
jQuery Open Source Process (RIT 2011)
 
jQuery Open Source Process (Knight Foundation 2011)
jQuery Open Source Process (Knight Foundation 2011)jQuery Open Source Process (Knight Foundation 2011)
jQuery Open Source Process (Knight Foundation 2011)
 
jQuery Mobile
jQuery MobilejQuery Mobile
jQuery Mobile
 
jQuery Open Source (Fronteer 2011)
jQuery Open Source (Fronteer 2011)jQuery Open Source (Fronteer 2011)
jQuery Open Source (Fronteer 2011)
 
Holistic JavaScript Performance
Holistic JavaScript PerformanceHolistic JavaScript Performance
Holistic JavaScript Performance
 
New Features Coming in Browsers (RIT '09)
New Features Coming in Browsers (RIT '09)New Features Coming in Browsers (RIT '09)
New Features Coming in Browsers (RIT '09)
 
Introduction to jQuery (Ajax Exp 2007)
Introduction to jQuery (Ajax Exp 2007)Introduction to jQuery (Ajax Exp 2007)
Introduction to jQuery (Ajax Exp 2007)
 
Advanced jQuery (Ajax Exp 2007)
Advanced jQuery (Ajax Exp 2007)Advanced jQuery (Ajax Exp 2007)
Advanced jQuery (Ajax Exp 2007)
 
JavaScript Library Overview (Ajax Exp West 2007)
JavaScript Library Overview (Ajax Exp West 2007)JavaScript Library Overview (Ajax Exp West 2007)
JavaScript Library Overview (Ajax Exp West 2007)
 
Meta Programming with JavaScript
Meta Programming with JavaScriptMeta Programming with JavaScript
Meta Programming with JavaScript
 

Recently uploaded

its a wip forever because i said so yeuh
its a wip forever because i said so yeuhits a wip forever because i said so yeuh
its a wip forever because i said so yeuhangelicafronda7
 
The Art of Rain_The beauty of rain in paintings..ppsx
The Art of Rain_The beauty of rain in paintings..ppsxThe Art of Rain_The beauty of rain in paintings..ppsx
The Art of Rain_The beauty of rain in paintings..ppsxguimera
 
Miss Illusion- beginning boards yayyyyyy
Miss Illusion- beginning boards yayyyyyyMiss Illusion- beginning boards yayyyyyy
Miss Illusion- beginning boards yayyyyyycwang48
 
Fishy - a short comic by Petra van Berkum
Fishy - a short comic by Petra van BerkumFishy - a short comic by Petra van Berkum
Fishy - a short comic by Petra van Berkumberkumpje1
 
Deondre' O'Bannon -Photography Portfolio
Deondre' O'Bannon -Photography PortfolioDeondre' O'Bannon -Photography Portfolio
Deondre' O'Bannon -Photography Portfoliodxobannon
 
Man Rayed - Messing with the Mess Master
Man Rayed - Messing with the Mess MasterMan Rayed - Messing with the Mess Master
Man Rayed - Messing with the Mess MasterProfessorNordell
 
Digital Marketing Creative Portfolio - Xandra Somera
Digital Marketing Creative Portfolio - Xandra SomeraDigital Marketing Creative Portfolio - Xandra Somera
Digital Marketing Creative Portfolio - Xandra SomeraXandra26
 
Smudge Animated Short Thumbnails Version 2
Smudge Animated Short Thumbnails Version 2Smudge Animated Short Thumbnails Version 2
Smudge Animated Short Thumbnails Version 2micahhansonart
 
Flowering lilacs for celebrate spring 17
Flowering lilacs for celebrate spring 17Flowering lilacs for celebrate spring 17
Flowering lilacs for celebrate spring 17sandamichaela *
 
Smudge Animated Short Storyboards Sequence C
Smudge Animated Short Storyboards Sequence CSmudge Animated Short Storyboards Sequence C
Smudge Animated Short Storyboards Sequence Cmicahhansonart
 
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATION
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATIONMeal Planning, Nutrition, and Digestion / TREATING CONSTIPATION
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATIONbilalpakweb
 
A selection of short panel comics by Petra van Berkum
A selection of short panel comics by Petra van BerkumA selection of short panel comics by Petra van Berkum
A selection of short panel comics by Petra van Berkumberkumpje1
 
Adventures in Soup Storyboard Clickthrough
Adventures in Soup Storyboard ClickthroughAdventures in Soup Storyboard Clickthrough
Adventures in Soup Storyboard ClickthroughLillyKocurek
 
Jurassic Falls StoryBoards Click Through
Jurassic Falls StoryBoards Click ThroughJurassic Falls StoryBoards Click Through
Jurassic Falls StoryBoards Click ThroughLillyKocurek
 
Alex Matus - Professional Best Photographer
Alex Matus - Professional Best PhotographerAlex Matus - Professional Best Photographer
Alex Matus - Professional Best PhotographerAlex Matus Photography
 
Recycle Ann Arbor Brand Guide Presentation
Recycle Ann Arbor Brand Guide PresentationRecycle Ann Arbor Brand Guide Presentation
Recycle Ann Arbor Brand Guide Presentationmakaiodm
 
The Beach - a short visual story by Petra van Berkum
The Beach - a short visual story by Petra van BerkumThe Beach - a short visual story by Petra van Berkum
The Beach - a short visual story by Petra van Berkumberkumpje1
 

Recently uploaded (20)

its a wip forever because i said so yeuh
its a wip forever because i said so yeuhits a wip forever because i said so yeuh
its a wip forever because i said so yeuh
 
The Art of Rain_The beauty of rain in paintings..ppsx
The Art of Rain_The beauty of rain in paintings..ppsxThe Art of Rain_The beauty of rain in paintings..ppsx
The Art of Rain_The beauty of rain in paintings..ppsx
 
Shadow Creation Service | CPH Graphics Media
Shadow Creation Service | CPH Graphics MediaShadow Creation Service | CPH Graphics Media
Shadow Creation Service | CPH Graphics Media
 
SEC v Burns .
SEC v Burns                                            .SEC v Burns                                            .
SEC v Burns .
 
Miss Illusion- beginning boards yayyyyyy
Miss Illusion- beginning boards yayyyyyyMiss Illusion- beginning boards yayyyyyy
Miss Illusion- beginning boards yayyyyyy
 
Fishy - a short comic by Petra van Berkum
Fishy - a short comic by Petra van BerkumFishy - a short comic by Petra van Berkum
Fishy - a short comic by Petra van Berkum
 
Deondre' O'Bannon -Photography Portfolio
Deondre' O'Bannon -Photography PortfolioDeondre' O'Bannon -Photography Portfolio
Deondre' O'Bannon -Photography Portfolio
 
Man Rayed - Messing with the Mess Master
Man Rayed - Messing with the Mess MasterMan Rayed - Messing with the Mess Master
Man Rayed - Messing with the Mess Master
 
Digital Marketing Creative Portfolio - Xandra Somera
Digital Marketing Creative Portfolio - Xandra SomeraDigital Marketing Creative Portfolio - Xandra Somera
Digital Marketing Creative Portfolio - Xandra Somera
 
Smudge Animated Short Thumbnails Version 2
Smudge Animated Short Thumbnails Version 2Smudge Animated Short Thumbnails Version 2
Smudge Animated Short Thumbnails Version 2
 
Flowering lilacs for celebrate spring 17
Flowering lilacs for celebrate spring 17Flowering lilacs for celebrate spring 17
Flowering lilacs for celebrate spring 17
 
Smudge Animated Short Storyboards Sequence C
Smudge Animated Short Storyboards Sequence CSmudge Animated Short Storyboards Sequence C
Smudge Animated Short Storyboards Sequence C
 
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATION
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATIONMeal Planning, Nutrition, and Digestion / TREATING CONSTIPATION
Meal Planning, Nutrition, and Digestion / TREATING CONSTIPATION
 
A selection of short panel comics by Petra van Berkum
A selection of short panel comics by Petra van BerkumA selection of short panel comics by Petra van Berkum
A selection of short panel comics by Petra van Berkum
 
Adventures in Soup Storyboard Clickthrough
Adventures in Soup Storyboard ClickthroughAdventures in Soup Storyboard Clickthrough
Adventures in Soup Storyboard Clickthrough
 
Jurassic Falls StoryBoards Click Through
Jurassic Falls StoryBoards Click ThroughJurassic Falls StoryBoards Click Through
Jurassic Falls StoryBoards Click Through
 
BELSOSI
BELSOSIBELSOSI
BELSOSI
 
Alex Matus - Professional Best Photographer
Alex Matus - Professional Best PhotographerAlex Matus - Professional Best Photographer
Alex Matus - Professional Best Photographer
 
Recycle Ann Arbor Brand Guide Presentation
Recycle Ann Arbor Brand Guide PresentationRecycle Ann Arbor Brand Guide Presentation
Recycle Ann Arbor Brand Guide Presentation
 
The Beach - a short visual story by Petra van Berkum
The Beach - a short visual story by Petra van BerkumThe Beach - a short visual story by Petra van Berkum
The Beach - a short visual story by Petra van Berkum
 

NYARC 2014: Frick/Zeri Results