SlideShare a Scribd company logo
1 of 17
Download to read offline
Zurich Universitiy
of Applied Sciences

Potential and Limitations of
Commercial Sentiment Detection Tools

Fatih Uzdilli
joint work with Mark Cieliebak and Oliver Dürr

03.12.2013 @ ESSEM’13
Zurich Universitiy
of Applied Sciences

About Me

Fatih Uzdilli
Institute of Applied Information Technology (InIT)
ZHAW, Winterthur, Switzerland
Email:
, more about me: home.zhaw.ch/~uzdi

Research Interest
Information Retrieval, Machine Learning, Sentiment Analysis
Background
Software Engineer, Social Media Monitoring, Search Technologies

03.12.2013

Fatih Uzdilli

2
Zurich Universitiy
of Applied Sciences

Abstract

Evaluation of 9 commercial sentiment tools on approx.
30'000 short texts.
Best commercial tools have accuracy of only 60%.

Combining all tools using Random Forest improved the
accuracy.

03.12.2013

Fatih Uzdilli

3
Zurich Universitiy
of Applied Sciences

Motivation

• Scientific results for sentiment detection:
«very good performance: > 80% accuracy»

C

• Blog posts about commercial tools:
«very poor quality, unusable»

D

03.12.2013

Fatih Uzdilli

4
Zurich Universitiy
of Applied Sciences

Motivation

• Scientific results for sentiment detection:
«very good performance: > 80% accuracy»

C

• Blog posts about commercial tools:
«very poor quality, unusable»

D

03.12.2013

Fatih Uzdilli

5
Zurich Universitiy
of Applied Sciences

How good is commercial
Sentiment Detection?

source: http://www.commute.com/images/schools_evaluation.jpg

03.12.2013

Is there potential for
improvement?

source: http://3.bp.blogspot.com/-u3acK_WjaLU/ULYv51mHEhI/
AAAAAAAAARY/ DIZqOfxuswc/s1600/IcebergQ1.jpg

Fatih Uzdilli

6
Zurich Universitiy
of Applied Sciences

Evaluation Setup

• 9 Commercial APIs

• 7 Public Text Corpora

– Stand-alone
– Free for this evaluation
– Arbitrary Text

– Single Statements
– Different Media Types
• Tweet, News, Review,
Speech Transcript

– Total: 28653 Texts

POSITIVE

NEGATIVE

OTHER
( neutral / mixed )
03.12.2013

Fatih Uzdilli

7
Zurich Universitiy
of Applied Sciences

Tool Accuracy

Avg.
Best Tool per Corpus

Accuracy

52%

Worst Tool per Corpus

0.7

61%

Average of All Tools

0.8

40%

0.6

0.5
0.4
0.3
0.2

03.12.2013

Fatih Uzdilli

8
Zurich Universitiy
of Applied Sciences

Tool Accuracy

Best Tool per Corpus
Average of All Tools
Worst Tool per Corpus
Overall Best Tool
Overall Worst Tool

0.8

Accuracy

0.7

Avg.
61%
52%
40%
59%
45%

0.6

0.5
0.4
0.3
0.2

03.12.2013

Fatih Uzdilli

9
Zurich Universitiy
of Applied Sciences

Further Findings

• Longer texts are hard to classify

• Corpus annotations might be erroneous

03.12.2013

Fatih Uzdilli

10
Zurich Universitiy
of Applied Sciences

Can a Meta-Classifier do better?
• 1st Approach: Majority Classifier
– Sentiment with most votes chosen
Illustration:
api1 api2 api3 api4 api5 api6 api7
Text 1
+
+
o
+
o
Text 2
Text 3
Text n

03.12.2013

o

+
o
o

+
+
+

+
o

+
-

Fatih Uzdilli

+
o

o

Majority
+

+
o

11
Zurich Universitiy
of Applied Sciences

Tool Accuracy

Best Tool per Corpus

0.8

Average of All Tools
Worst Tool per Corpus

Accuracy

0.7
0.6
0.5
0.4

0.3
0.2

03.12.2013

Fatih Uzdilli

12
Majority Classifier beats Average

Best Tool per Corpus
Average of All Tools
Worst Tool per Corpus
Majority Classifier

0.8
0.7

Accuracy

Zurich Universitiy
of Applied Sciences

0.6
0.5
0.4

0.3
0.2

03.12.2013

Fatih Uzdilli

13
Zurich Universitiy
of Applied Sciences

2nd Approach: Random-Forest
api1 api2 api3 … api n annotation
Text 1 +
+ … o
+
Train
Text 2
+
o … +
Train
Text 3
o
- … +
Train
Text 4 +
Train
o
+ …
+
Text 5
Text 6
Text 7
Text 8
Text 9
03.12.2013

+
+
+
+
o

o
o
+
-

+
o
+
o
+

…
…
…
…
…

o
o
o
Fatih Uzdilli

o
Train
Train
o
unknown Predict
unknown Predict
unknown Predict

Random
Forest
Classifier

+
+

o
14
Zurich Universitiy
of Applied Sciences

Before Random Forest

Best Tool per Corpus
Average of All Tools
Worst Tool per Corpus
Majority Classifier

0.8

Accuracy

0.7
0.6
0.5
0.4

0.3
0.2

03.12.2013

Fatih Uzdilli

15
Random Forest Beats Best Single Tool

Best Tool per Corpus
Average of All Tools
Worst Tool per Corpus
Majority Classifier
Random Forest Classifier

0.8
0.7

Accuracy

Zurich Universitiy
of Applied Sciences

0.6
0.5
0.4

0.3
0.2

03.12.2013

Fatih Uzdilli

16
Zurich Universitiy
of Applied Sciences

Summary

• Best Tool: 59% Accuracy
• Random Forest combination: Up to 9% improvement

<=9%

03.12.2013

Fatih Uzdilli

17

More Related Content

Similar to Potential and Limitations of Commercial Sentiment Detection Tools

Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdf
Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdfGuest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdf
Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdfallamodainternationa
 
ICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionAlessandro Bozzon
 
Social Media for Scientific Research
Social Media for Scientific ResearchSocial Media for Scientific Research
Social Media for Scientific ResearchRené Schneider
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceCarole Goble
 
Better Software, Better Research
Better Software, Better ResearchBetter Software, Better Research
Better Software, Better ResearchCarole Goble
 
Infrastructures for Open, Digital Science
Infrastructures for Open, Digital ScienceInfrastructures for Open, Digital Science
Infrastructures for Open, Digital ScienceCarl-Christian Buhr
 
Altmetrics - Measuring the Buzz
Altmetrics - Measuring the BuzzAltmetrics - Measuring the Buzz
Altmetrics - Measuring the BuzzBruce Antelman
 
The OpenCon Intro to Open Data
The OpenCon Intro to Open DataThe OpenCon Intro to Open Data
The OpenCon Intro to Open DataRoss Mounce
 
Technology has all the right answers - but we have to start thinking about wh...
Technology has all the right answers - but we have to start thinking about wh...Technology has all the right answers - but we have to start thinking about wh...
Technology has all the right answers - but we have to start thinking about wh...Andy Tattersall
 
CILIP Conference 2019 - Digital innovation - Andy Tattersall
CILIP Conference 2019 - Digital innovation - Andy TattersallCILIP Conference 2019 - Digital innovation - Andy Tattersall
CILIP Conference 2019 - Digital innovation - Andy TattersallCILIP
 
Dart ord the citizen's persepctive-20141107
Dart ord the citizen's persepctive-20141107Dart ord the citizen's persepctive-20141107
Dart ord the citizen's persepctive-20141107Andre Golliez
 
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...Online Research Coming of age - Brownbag Presentation at Universitty of Preto...
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...Holger Lütters
 
Better software, better service, better research: The Software Sustainabilit...
Better software, better service, better research: The Software Sustainabilit...Better software, better service, better research: The Software Sustainabilit...
Better software, better service, better research: The Software Sustainabilit...Carole Goble
 
The data science revolution in insurance
The data science revolution in insuranceThe data science revolution in insurance
The data science revolution in insuranceStefano Perfetti
 
Software Analytics: The Dark Side and the Test Side
Software Analytics: The Dark Side and the Test SideSoftware Analytics: The Dark Side and the Test Side
Software Analytics: The Dark Side and the Test SideAndy Zaidman
 
Social media cafe ResearchGate
Social media cafe ResearchGateSocial media cafe ResearchGate
Social media cafe ResearchGateHugo Besemer
 
Reproducibility Analytics Lab
Reproducibility Analytics Lab Reproducibility Analytics Lab
Reproducibility Analytics Lab Verena139
 
Crowdsourcing - an overview
Crowdsourcing - an overviewCrowdsourcing - an overview
Crowdsourcing - an overviewMirko Presser
 

Similar to Potential and Limitations of Commercial Sentiment Detection Tools (20)

Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdf
Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdfGuest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdf
Guest post #17 Dr Daniel Turner Can a computer do qualitative analy.pdf
 
ICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo SessionICWE 2013 - Slides From The Poster And Demo Session
ICWE 2013 - Slides From The Poster And Demo Session
 
Social Media for Scientific Research
Social Media for Scientific ResearchSocial Media for Scientific Research
Social Media for Scientific Research
 
Software Sustainability: Better Software Better Science
Software Sustainability: Better Software Better ScienceSoftware Sustainability: Better Software Better Science
Software Sustainability: Better Software Better Science
 
Better Software, Better Research
Better Software, Better ResearchBetter Software, Better Research
Better Software, Better Research
 
Infrastructures for Open, Digital Science
Infrastructures for Open, Digital ScienceInfrastructures for Open, Digital Science
Infrastructures for Open, Digital Science
 
Future Technological Practices: Medical Librarians’ Skills and Information St...
Future Technological Practices: Medical Librarians’ Skills and Information St...Future Technological Practices: Medical Librarians’ Skills and Information St...
Future Technological Practices: Medical Librarians’ Skills and Information St...
 
Altmetrics - Measuring the Buzz
Altmetrics - Measuring the BuzzAltmetrics - Measuring the Buzz
Altmetrics - Measuring the Buzz
 
The OpenCon Intro to Open Data
The OpenCon Intro to Open DataThe OpenCon Intro to Open Data
The OpenCon Intro to Open Data
 
Technology has all the right answers - but we have to start thinking about wh...
Technology has all the right answers - but we have to start thinking about wh...Technology has all the right answers - but we have to start thinking about wh...
Technology has all the right answers - but we have to start thinking about wh...
 
CILIP Conference 2019 - Digital innovation - Andy Tattersall
CILIP Conference 2019 - Digital innovation - Andy TattersallCILIP Conference 2019 - Digital innovation - Andy Tattersall
CILIP Conference 2019 - Digital innovation - Andy Tattersall
 
Dart ord the citizen's persepctive-20141107
Dart ord the citizen's persepctive-20141107Dart ord the citizen's persepctive-20141107
Dart ord the citizen's persepctive-20141107
 
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...Online Research Coming of age - Brownbag Presentation at Universitty of Preto...
Online Research Coming of age - Brownbag Presentation at Universitty of Preto...
 
Open, Digital Science in Europe
Open, Digital Science in EuropeOpen, Digital Science in Europe
Open, Digital Science in Europe
 
Better software, better service, better research: The Software Sustainabilit...
Better software, better service, better research: The Software Sustainabilit...Better software, better service, better research: The Software Sustainabilit...
Better software, better service, better research: The Software Sustainabilit...
 
The data science revolution in insurance
The data science revolution in insuranceThe data science revolution in insurance
The data science revolution in insurance
 
Software Analytics: The Dark Side and the Test Side
Software Analytics: The Dark Side and the Test SideSoftware Analytics: The Dark Side and the Test Side
Software Analytics: The Dark Side and the Test Side
 
Social media cafe ResearchGate
Social media cafe ResearchGateSocial media cafe ResearchGate
Social media cafe ResearchGate
 
Reproducibility Analytics Lab
Reproducibility Analytics Lab Reproducibility Analytics Lab
Reproducibility Analytics Lab
 
Crowdsourcing - an overview
Crowdsourcing - an overviewCrowdsourcing - an overview
Crowdsourcing - an overview
 

Recently uploaded

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 

Recently uploaded (20)

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 

Potential and Limitations of Commercial Sentiment Detection Tools

  • 1. Zurich Universitiy of Applied Sciences Potential and Limitations of Commercial Sentiment Detection Tools Fatih Uzdilli joint work with Mark Cieliebak and Oliver Dürr 03.12.2013 @ ESSEM’13
  • 2. Zurich Universitiy of Applied Sciences About Me Fatih Uzdilli Institute of Applied Information Technology (InIT) ZHAW, Winterthur, Switzerland Email: , more about me: home.zhaw.ch/~uzdi Research Interest Information Retrieval, Machine Learning, Sentiment Analysis Background Software Engineer, Social Media Monitoring, Search Technologies 03.12.2013 Fatih Uzdilli 2
  • 3. Zurich Universitiy of Applied Sciences Abstract Evaluation of 9 commercial sentiment tools on approx. 30'000 short texts. Best commercial tools have accuracy of only 60%. Combining all tools using Random Forest improved the accuracy. 03.12.2013 Fatih Uzdilli 3
  • 4. Zurich Universitiy of Applied Sciences Motivation • Scientific results for sentiment detection: «very good performance: > 80% accuracy» C • Blog posts about commercial tools: «very poor quality, unusable» D 03.12.2013 Fatih Uzdilli 4
  • 5. Zurich Universitiy of Applied Sciences Motivation • Scientific results for sentiment detection: «very good performance: > 80% accuracy» C • Blog posts about commercial tools: «very poor quality, unusable» D 03.12.2013 Fatih Uzdilli 5
  • 6. Zurich Universitiy of Applied Sciences How good is commercial Sentiment Detection? source: http://www.commute.com/images/schools_evaluation.jpg 03.12.2013 Is there potential for improvement? source: http://3.bp.blogspot.com/-u3acK_WjaLU/ULYv51mHEhI/ AAAAAAAAARY/ DIZqOfxuswc/s1600/IcebergQ1.jpg Fatih Uzdilli 6
  • 7. Zurich Universitiy of Applied Sciences Evaluation Setup • 9 Commercial APIs • 7 Public Text Corpora – Stand-alone – Free for this evaluation – Arbitrary Text – Single Statements – Different Media Types • Tweet, News, Review, Speech Transcript – Total: 28653 Texts POSITIVE NEGATIVE OTHER ( neutral / mixed ) 03.12.2013 Fatih Uzdilli 7
  • 8. Zurich Universitiy of Applied Sciences Tool Accuracy Avg. Best Tool per Corpus Accuracy 52% Worst Tool per Corpus 0.7 61% Average of All Tools 0.8 40% 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 8
  • 9. Zurich Universitiy of Applied Sciences Tool Accuracy Best Tool per Corpus Average of All Tools Worst Tool per Corpus Overall Best Tool Overall Worst Tool 0.8 Accuracy 0.7 Avg. 61% 52% 40% 59% 45% 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 9
  • 10. Zurich Universitiy of Applied Sciences Further Findings • Longer texts are hard to classify • Corpus annotations might be erroneous 03.12.2013 Fatih Uzdilli 10
  • 11. Zurich Universitiy of Applied Sciences Can a Meta-Classifier do better? • 1st Approach: Majority Classifier – Sentiment with most votes chosen Illustration: api1 api2 api3 api4 api5 api6 api7 Text 1 + + o + o Text 2 Text 3 Text n 03.12.2013 o + o o + + + + o + - Fatih Uzdilli + o o Majority + + o 11
  • 12. Zurich Universitiy of Applied Sciences Tool Accuracy Best Tool per Corpus 0.8 Average of All Tools Worst Tool per Corpus Accuracy 0.7 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 12
  • 13. Majority Classifier beats Average Best Tool per Corpus Average of All Tools Worst Tool per Corpus Majority Classifier 0.8 0.7 Accuracy Zurich Universitiy of Applied Sciences 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 13
  • 14. Zurich Universitiy of Applied Sciences 2nd Approach: Random-Forest api1 api2 api3 … api n annotation Text 1 + + … o + Train Text 2 + o … + Train Text 3 o - … + Train Text 4 + Train o + … + Text 5 Text 6 Text 7 Text 8 Text 9 03.12.2013 + + + + o o o + - + o + o + … … … … … o o o Fatih Uzdilli o Train Train o unknown Predict unknown Predict unknown Predict Random Forest Classifier + + o 14
  • 15. Zurich Universitiy of Applied Sciences Before Random Forest Best Tool per Corpus Average of All Tools Worst Tool per Corpus Majority Classifier 0.8 Accuracy 0.7 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 15
  • 16. Random Forest Beats Best Single Tool Best Tool per Corpus Average of All Tools Worst Tool per Corpus Majority Classifier Random Forest Classifier 0.8 0.7 Accuracy Zurich Universitiy of Applied Sciences 0.6 0.5 0.4 0.3 0.2 03.12.2013 Fatih Uzdilli 16
  • 17. Zurich Universitiy of Applied Sciences Summary • Best Tool: 59% Accuracy • Random Forest combination: Up to 9% improvement <=9% 03.12.2013 Fatih Uzdilli 17