SlideShare a Scribd company logo
1 of 20
Download to read offline
How can crowdsourcing
and machine learning
improve speech
technology?
Joao Freitas, Daniela BragaCSW Global London
April 14th 2016
April 2016 definedcrowd2
How many of you have tried speech recognition?
April 2016 definedcrowd3
Speech Technology is everywhere
April 2016 definedcrowd4
And it starts to understands you…
April 2016 definedcrowd5
What it takes to get there
Large amounts of data Deep Learning
3000+ hours speech recordings + transcription
200+ words with pronunciations
0.5M natural language variants + semantic annotation
Language and Product dependent!
April 2016 definedcrowd6
DefinedCrowd landscape
We serve the data needs for AI and
ML landscape.
We’re a SaaS company that collects
and enriches training data for AI,
combining crowdsourcing and ML.
April 2016 definedcrowd7
The world before DefinedCrowd
Louis, Speech Scientist
Wants to test if the
Chinese acoustic model
works for Mandarin
speakers in Singapore
User Goal
Hires:
• Few vendors
• 1PM
• 1 Dev
• 1 Chinese LE in-house
What does he do?
50 hours of raw speech with…
• Poor quality (~20% of garbage)
• Unknown sources
• Long wait
What does he get?
April 2016 definedcrowd8
The world after DefinedCrowd
Andy, Speech Scientist
Wants to test if the
Chinese acoustic model
works for Mandarin
speakers in Singapore
User Goal
Subscribes our
platform
What does he do?
50 hours of pure speech
with…
• High-quality
• 100% transparency
• 50% faster throughput
What does he get?
• Picks a template
• Adjusts settings and
picks the crowd
• Launches the job
• Collects the data
How does he do it?
April 2016 definedcrowd
9
Our platform – enterprise side
April 2016 definedcrowd
Unique crowd model
US: 200+
Brazil: 200+
Taiwan: 100+
Russia: 200+
Japan: 100+
Korea: 100+
Ukraine (100+)
Spain (100+)
Portugal (100+)
France (100+)
Germany (100+)
Denmark (50+)
Sweden (50+)
Finland (50+)
Netherlands (50+)
Italy (100+)
Greece (100+)
Czech Republic (100+)
Poland (100+)
Turkey (100+)
Belgium (50+)
Australia: 100+
New Zealand:50+
Mexico: 100+
Puerto Rico: 100+
Canada: 100+
China: 200+
Vietnam: 50+
Thailand: 50+
Malaysia: 50+
Singapore: 50+
India: 100+
 30+ countries
 100+ dialects
 3,000 crowd
April 2016 definedcrowd11
We know a lot about our crowd
Languages
& Dialect
User
Activity
Job
Performance
School &
Courses
Profile
Info
Other
Jobs
April 2016 definedcrowd12
Why
is Machine Learning
relevant for
Crowdsourcing?
April 2016 definedcrowd13
We learn from metadata to provide recommendations to customers and crowd members
How we use Machine Learning
April 2016 definedcrowd14
How we detect spam
Raw data
• Logging system
• Behavior measures
Data Processing
•Clean data
•Transform data
Feature Extraction
• Task-related measures
(e.g. average duration)
• Session Duration
• Execution peaks
• Consensus score
• Real-time audits
Classification &
Analysis
• Detect outliers/
anomalies
• Predict task / job
duration
April 2016 definedcrowd15
Example of Results I
April 2016 definedcrowd
Same results – Different perspective
April 2016 definedcrowd17
Another Dimension
April 2016 definedcrowd18
Quality in our platform
1. Combined score of Qualification Tests
2. Real-time Audits and Reviews
3. Majority Vote
4. Overall Majority
5. Worker Expertise
6. Task Subjectiveness
7. …
April 2016 definedcrowd19
Other predictions using Machine Learning
Best quality / budget tradeoff
Best match between job and crowd member
Expected quality
When will a job finish (even before it starts)
Quality Time
Cost
definedcrowd
Intelligent data for AI
contacts:
joao@definedcrowd.com
daniela@definedcrowd.com
mail@definedcrowd.com

More Related Content

Viewers also liked

Using the Power of the Crowd to Shape the Economy of the Future
Using the Power of the Crowd to Shape the Economy of the FutureUsing the Power of the Crowd to Shape the Economy of the Future
Using the Power of the Crowd to Shape the Economy of the FutureCrowdsourcing Week
 
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdsourcing Week
 
The Role of Design in Crowdsourcing
The Role of Design in CrowdsourcingThe Role of Design in Crowdsourcing
The Role of Design in CrowdsourcingCrowdsourcing Week
 
Crowdsourced Education and Skills Needed for the Future of Employability
Crowdsourced Education and Skills Needed for the Future of EmployabilityCrowdsourced Education and Skills Needed for the Future of Employability
Crowdsourced Education and Skills Needed for the Future of EmployabilityCrowdsourcing Week
 

Viewers also liked (6)

Using the Power of the Crowd to Shape the Economy of the Future
Using the Power of the Crowd to Shape the Economy of the FutureUsing the Power of the Crowd to Shape the Economy of the Future
Using the Power of the Crowd to Shape the Economy of the Future
 
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
 
The Role of Design in Crowdsourcing
The Role of Design in CrowdsourcingThe Role of Design in Crowdsourcing
The Role of Design in Crowdsourcing
 
The Internet of Value
The Internet of ValueThe Internet of Value
The Internet of Value
 
Power from the Crowd
Power from the CrowdPower from the Crowd
Power from the Crowd
 
Crowdsourced Education and Skills Needed for the Future of Employability
Crowdsourced Education and Skills Needed for the Future of EmployabilityCrowdsourced Education and Skills Needed for the Future of Employability
Crowdsourced Education and Skills Needed for the Future of Employability
 

Similar to How Can Crowdsourcing and Machine Learning Improve Speech Technology?

Wipro-TechnicalRound&HRInterviewQuestions (1).doc
Wipro-TechnicalRound&HRInterviewQuestions (1).docWipro-TechnicalRound&HRInterviewQuestions (1).doc
Wipro-TechnicalRound&HRInterviewQuestions (1).docAyushVerma584677
 
Kush stats alpha
Kush stats alpha Kush stats alpha
Kush stats alpha Ajay Ohri
 
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016Sentiment Analysis - State of the Art in Research and Industry - SDS 2016
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016Mark Cieliebak
 
You already have an LxP, you just don't know it
You already have an LxP, you just don't know itYou already have an LxP, you just don't know it
You already have an LxP, you just don't know itJames Wann
 
RFPs For a Modern Learning Ecosystem
RFPs For a Modern Learning EcosystemRFPs For a Modern Learning Ecosystem
RFPs For a Modern Learning EcosystemDavid Blake
 
Unicom Core Committee Advisory Panel Meet
Unicom Core Committee Advisory Panel MeetUnicom Core Committee Advisory Panel Meet
Unicom Core Committee Advisory Panel MeetUnicom Conference
 
3 Corporate Learning Trends to Watch in 2017
3 Corporate Learning Trends to Watch in 20173 Corporate Learning Trends to Watch in 2017
3 Corporate Learning Trends to Watch in 2017Axonify
 
Education2.0 or Education and Web2.0
Education2.0 or Education and Web2.0Education2.0 or Education and Web2.0
Education2.0 or Education and Web2.0Lukas Ritzel
 
easy ten for Silicon Valley Meets Russia 2014
easy ten for Silicon Valley Meets Russia 2014easy ten for Silicon Valley Meets Russia 2014
easy ten for Silicon Valley Meets Russia 2014Dmitry Zaryuta
 
FP adoption at REA
FP adoption at REAFP adoption at REA
FP adoption at REAkenbot
 
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...Simon Darling
 
SENTHIL LATEST JULY 15TH
SENTHIL LATEST JULY 15THSENTHIL LATEST JULY 15TH
SENTHIL LATEST JULY 15THLinkedin Coach
 
Discover leo (english)
Discover leo (english)Discover leo (english)
Discover leo (english)shah zeb
 
xAPI Webinar July 23rd - Rob Faulkner
xAPI Webinar July 23rd - Rob FaulknerxAPI Webinar July 23rd - Rob Faulkner
xAPI Webinar July 23rd - Rob FaulknerWebanywhere Ltd
 
xAPI: The Landscape
xAPI: The LandscapexAPI: The Landscape
xAPI: The LandscapeMegan Bowe
 

Similar to How Can Crowdsourcing and Machine Learning Improve Speech Technology? (20)

FOSS Workshop Profile
FOSS Workshop ProfileFOSS Workshop Profile
FOSS Workshop Profile
 
Wipro-TechnicalRound&HRInterviewQuestions (1).doc
Wipro-TechnicalRound&HRInterviewQuestions (1).docWipro-TechnicalRound&HRInterviewQuestions (1).doc
Wipro-TechnicalRound&HRInterviewQuestions (1).doc
 
Kush stats alpha
Kush stats alpha Kush stats alpha
Kush stats alpha
 
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016Sentiment Analysis - State of the Art in Research and Industry - SDS 2016
Sentiment Analysis - State of the Art in Research and Industry - SDS 2016
 
You already have an LxP, you just don't know it
You already have an LxP, you just don't know itYou already have an LxP, you just don't know it
You already have an LxP, you just don't know it
 
RFPs For a Modern Learning Ecosystem
RFPs For a Modern Learning EcosystemRFPs For a Modern Learning Ecosystem
RFPs For a Modern Learning Ecosystem
 
Unicom Core Committee Advisory Panel Meet
Unicom Core Committee Advisory Panel MeetUnicom Core Committee Advisory Panel Meet
Unicom Core Committee Advisory Panel Meet
 
3 Corporate Learning Trends to Watch in 2017
3 Corporate Learning Trends to Watch in 20173 Corporate Learning Trends to Watch in 2017
3 Corporate Learning Trends to Watch in 2017
 
Life @ EdApp
Life @ EdAppLife @ EdApp
Life @ EdApp
 
STRV Insights
STRV InsightsSTRV Insights
STRV Insights
 
Education2.0 or Education and Web2.0
Education2.0 or Education and Web2.0Education2.0 or Education and Web2.0
Education2.0 or Education and Web2.0
 
easy ten for Silicon Valley Meets Russia 2014
easy ten for Silicon Valley Meets Russia 2014easy ten for Silicon Valley Meets Russia 2014
easy ten for Silicon Valley Meets Russia 2014
 
FP adoption at REA
FP adoption at REAFP adoption at REA
FP adoption at REA
 
senthil.sqc
senthil.sqcsenthil.sqc
senthil.sqc
 
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...
HR Tech Applicant Tracking System ATS and Candidate Relationship Management (...
 
Plug Tree
Plug TreePlug Tree
Plug Tree
 
SENTHIL LATEST JULY 15TH
SENTHIL LATEST JULY 15THSENTHIL LATEST JULY 15TH
SENTHIL LATEST JULY 15TH
 
Discover leo (english)
Discover leo (english)Discover leo (english)
Discover leo (english)
 
xAPI Webinar July 23rd - Rob Faulkner
xAPI Webinar July 23rd - Rob FaulknerxAPI Webinar July 23rd - Rob Faulkner
xAPI Webinar July 23rd - Rob Faulkner
 
xAPI: The Landscape
xAPI: The LandscapexAPI: The Landscape
xAPI: The Landscape
 

More from Crowdsourcing Week

Crowdsourcing à la sbv IMPROVER: the challenge of being your own client
Crowdsourcing à la sbv IMPROVER: the challenge of being your own clientCrowdsourcing à la sbv IMPROVER: the challenge of being your own client
Crowdsourcing à la sbv IMPROVER: the challenge of being your own clientCrowdsourcing Week
 
Transforming the Global Payments Operation
Transforming the Global Payments OperationTransforming the Global Payments Operation
Transforming the Global Payments OperationCrowdsourcing Week
 
Crowdsourced to Outsourced: How online platforms are shaping the future of work
Crowdsourced to Outsourced: How online platforms are shaping the future of workCrowdsourced to Outsourced: How online platforms are shaping the future of work
Crowdsourced to Outsourced: How online platforms are shaping the future of workCrowdsourcing Week
 
Malasya's Experience in Crowd Labour and Sharing Economy
Malasya's Experience in Crowd Labour and Sharing EconomyMalasya's Experience in Crowd Labour and Sharing Economy
Malasya's Experience in Crowd Labour and Sharing EconomyCrowdsourcing Week
 
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...Crowdsourcing Week
 
Human Collective Intelligence: the future of corporate innovation
Human Collective Intelligence: the future of corporate innovationHuman Collective Intelligence: the future of corporate innovation
Human Collective Intelligence: the future of corporate innovationCrowdsourcing Week
 
9 Ways to Ruin Your Open Innovation Challenge
9 Ways to Ruin Your Open Innovation Challenge9 Ways to Ruin Your Open Innovation Challenge
9 Ways to Ruin Your Open Innovation ChallengeCrowdsourcing Week
 
Accelerating Hardware Development: Ideation and Engineering
Accelerating Hardware Development: Ideation and EngineeringAccelerating Hardware Development: Ideation and Engineering
Accelerating Hardware Development: Ideation and EngineeringCrowdsourcing Week
 
Attracting and Retaining Top Partners with a Best-in-Class Payments Experience
Attracting and Retaining Top Partners with a Best-in-Class Payments ExperienceAttracting and Retaining Top Partners with a Best-in-Class Payments Experience
Attracting and Retaining Top Partners with a Best-in-Class Payments ExperienceCrowdsourcing Week
 
Core + Crowd: Why (and how) crowdsourcing is about to become mainstream
Core + Crowd: Why (and how) crowdsourcing is about to become mainstreamCore + Crowd: Why (and how) crowdsourcing is about to become mainstream
Core + Crowd: Why (and how) crowdsourcing is about to become mainstreamCrowdsourcing Week
 
Smart and Secure Cities and Communities
Smart and Secure Cities and Communities Smart and Secure Cities and Communities
Smart and Secure Cities and Communities Crowdsourcing Week
 
How Successful Crowdsourcing Depends on asking 'Interesting Questions'
How Successful Crowdsourcing Depends on asking 'Interesting Questions'How Successful Crowdsourcing Depends on asking 'Interesting Questions'
How Successful Crowdsourcing Depends on asking 'Interesting Questions'Crowdsourcing Week
 
Contestant Centered Design: creative approaches to designing competitions
Contestant Centered Design: creative approaches to designing competitionsContestant Centered Design: creative approaches to designing competitions
Contestant Centered Design: creative approaches to designing competitionsCrowdsourcing Week
 
How Crypto can Monetize Crowdsourcing
How Crypto can Monetize CrowdsourcingHow Crypto can Monetize Crowdsourcing
How Crypto can Monetize CrowdsourcingCrowdsourcing Week
 
A New Report on the State of Open Innovation and What it Means For you
A New Report on the State of Open Innovation and What it Means For youA New Report on the State of Open Innovation and What it Means For you
A New Report on the State of Open Innovation and What it Means For youCrowdsourcing Week
 
Expert Operating System: Business On-Demand
Expert Operating System: Business On-DemandExpert Operating System: Business On-Demand
Expert Operating System: Business On-DemandCrowdsourcing Week
 
Crowdsourcing: Changing the Faces of Innovation at NASA
Crowdsourcing: Changing the Faces of Innovation at NASACrowdsourcing: Changing the Faces of Innovation at NASA
Crowdsourcing: Changing the Faces of Innovation at NASACrowdsourcing Week
 
Crowdfunding an ICO Without Getting In Trouble
Crowdfunding an ICO Without Getting In TroubleCrowdfunding an ICO Without Getting In Trouble
Crowdfunding an ICO Without Getting In TroubleCrowdsourcing Week
 

More from Crowdsourcing Week (20)

Crowdsourcing à la sbv IMPROVER: the challenge of being your own client
Crowdsourcing à la sbv IMPROVER: the challenge of being your own clientCrowdsourcing à la sbv IMPROVER: the challenge of being your own client
Crowdsourcing à la sbv IMPROVER: the challenge of being your own client
 
Transforming the Global Payments Operation
Transforming the Global Payments OperationTransforming the Global Payments Operation
Transforming the Global Payments Operation
 
Crowdsourced to Outsourced: How online platforms are shaping the future of work
Crowdsourced to Outsourced: How online platforms are shaping the future of workCrowdsourced to Outsourced: How online platforms are shaping the future of work
Crowdsourced to Outsourced: How online platforms are shaping the future of work
 
Malasya's Experience in Crowd Labour and Sharing Economy
Malasya's Experience in Crowd Labour and Sharing EconomyMalasya's Experience in Crowd Labour and Sharing Economy
Malasya's Experience in Crowd Labour and Sharing Economy
 
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...
LM Industries: Harnessing The Power of Crowdsourced Innovation to Build the F...
 
Human Collective Intelligence: the future of corporate innovation
Human Collective Intelligence: the future of corporate innovationHuman Collective Intelligence: the future of corporate innovation
Human Collective Intelligence: the future of corporate innovation
 
9 Ways to Ruin Your Open Innovation Challenge
9 Ways to Ruin Your Open Innovation Challenge9 Ways to Ruin Your Open Innovation Challenge
9 Ways to Ruin Your Open Innovation Challenge
 
Disruptive Crowdsourcing
Disruptive CrowdsourcingDisruptive Crowdsourcing
Disruptive Crowdsourcing
 
Accelerating Hardware Development: Ideation and Engineering
Accelerating Hardware Development: Ideation and EngineeringAccelerating Hardware Development: Ideation and Engineering
Accelerating Hardware Development: Ideation and Engineering
 
Attracting and Retaining Top Partners with a Best-in-Class Payments Experience
Attracting and Retaining Top Partners with a Best-in-Class Payments ExperienceAttracting and Retaining Top Partners with a Best-in-Class Payments Experience
Attracting and Retaining Top Partners with a Best-in-Class Payments Experience
 
Crowdsourcing Disaster Relief
Crowdsourcing Disaster ReliefCrowdsourcing Disaster Relief
Crowdsourcing Disaster Relief
 
Core + Crowd: Why (and how) crowdsourcing is about to become mainstream
Core + Crowd: Why (and how) crowdsourcing is about to become mainstreamCore + Crowd: Why (and how) crowdsourcing is about to become mainstream
Core + Crowd: Why (and how) crowdsourcing is about to become mainstream
 
Smart and Secure Cities and Communities
Smart and Secure Cities and Communities Smart and Secure Cities and Communities
Smart and Secure Cities and Communities
 
How Successful Crowdsourcing Depends on asking 'Interesting Questions'
How Successful Crowdsourcing Depends on asking 'Interesting Questions'How Successful Crowdsourcing Depends on asking 'Interesting Questions'
How Successful Crowdsourcing Depends on asking 'Interesting Questions'
 
Contestant Centered Design: creative approaches to designing competitions
Contestant Centered Design: creative approaches to designing competitionsContestant Centered Design: creative approaches to designing competitions
Contestant Centered Design: creative approaches to designing competitions
 
How Crypto can Monetize Crowdsourcing
How Crypto can Monetize CrowdsourcingHow Crypto can Monetize Crowdsourcing
How Crypto can Monetize Crowdsourcing
 
A New Report on the State of Open Innovation and What it Means For you
A New Report on the State of Open Innovation and What it Means For youA New Report on the State of Open Innovation and What it Means For you
A New Report on the State of Open Innovation and What it Means For you
 
Expert Operating System: Business On-Demand
Expert Operating System: Business On-DemandExpert Operating System: Business On-Demand
Expert Operating System: Business On-Demand
 
Crowdsourcing: Changing the Faces of Innovation at NASA
Crowdsourcing: Changing the Faces of Innovation at NASACrowdsourcing: Changing the Faces of Innovation at NASA
Crowdsourcing: Changing the Faces of Innovation at NASA
 
Crowdfunding an ICO Without Getting In Trouble
Crowdfunding an ICO Without Getting In TroubleCrowdfunding an ICO Without Getting In Trouble
Crowdfunding an ICO Without Getting In Trouble
 

Recently uploaded

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 

Recently uploaded (20)

COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 

How Can Crowdsourcing and Machine Learning Improve Speech Technology?

  • 1. How can crowdsourcing and machine learning improve speech technology? Joao Freitas, Daniela BragaCSW Global London April 14th 2016
  • 2. April 2016 definedcrowd2 How many of you have tried speech recognition?
  • 3. April 2016 definedcrowd3 Speech Technology is everywhere
  • 4. April 2016 definedcrowd4 And it starts to understands you…
  • 5. April 2016 definedcrowd5 What it takes to get there Large amounts of data Deep Learning 3000+ hours speech recordings + transcription 200+ words with pronunciations 0.5M natural language variants + semantic annotation Language and Product dependent!
  • 6. April 2016 definedcrowd6 DefinedCrowd landscape We serve the data needs for AI and ML landscape. We’re a SaaS company that collects and enriches training data for AI, combining crowdsourcing and ML.
  • 7. April 2016 definedcrowd7 The world before DefinedCrowd Louis, Speech Scientist Wants to test if the Chinese acoustic model works for Mandarin speakers in Singapore User Goal Hires: • Few vendors • 1PM • 1 Dev • 1 Chinese LE in-house What does he do? 50 hours of raw speech with… • Poor quality (~20% of garbage) • Unknown sources • Long wait What does he get?
  • 8. April 2016 definedcrowd8 The world after DefinedCrowd Andy, Speech Scientist Wants to test if the Chinese acoustic model works for Mandarin speakers in Singapore User Goal Subscribes our platform What does he do? 50 hours of pure speech with… • High-quality • 100% transparency • 50% faster throughput What does he get? • Picks a template • Adjusts settings and picks the crowd • Launches the job • Collects the data How does he do it?
  • 9. April 2016 definedcrowd 9 Our platform – enterprise side
  • 10. April 2016 definedcrowd Unique crowd model US: 200+ Brazil: 200+ Taiwan: 100+ Russia: 200+ Japan: 100+ Korea: 100+ Ukraine (100+) Spain (100+) Portugal (100+) France (100+) Germany (100+) Denmark (50+) Sweden (50+) Finland (50+) Netherlands (50+) Italy (100+) Greece (100+) Czech Republic (100+) Poland (100+) Turkey (100+) Belgium (50+) Australia: 100+ New Zealand:50+ Mexico: 100+ Puerto Rico: 100+ Canada: 100+ China: 200+ Vietnam: 50+ Thailand: 50+ Malaysia: 50+ Singapore: 50+ India: 100+  30+ countries  100+ dialects  3,000 crowd
  • 11. April 2016 definedcrowd11 We know a lot about our crowd Languages & Dialect User Activity Job Performance School & Courses Profile Info Other Jobs
  • 12. April 2016 definedcrowd12 Why is Machine Learning relevant for Crowdsourcing?
  • 13. April 2016 definedcrowd13 We learn from metadata to provide recommendations to customers and crowd members How we use Machine Learning
  • 14. April 2016 definedcrowd14 How we detect spam Raw data • Logging system • Behavior measures Data Processing •Clean data •Transform data Feature Extraction • Task-related measures (e.g. average duration) • Session Duration • Execution peaks • Consensus score • Real-time audits Classification & Analysis • Detect outliers/ anomalies • Predict task / job duration
  • 16. April 2016 definedcrowd Same results – Different perspective
  • 18. April 2016 definedcrowd18 Quality in our platform 1. Combined score of Qualification Tests 2. Real-time Audits and Reviews 3. Majority Vote 4. Overall Majority 5. Worker Expertise 6. Task Subjectiveness 7. …
  • 19. April 2016 definedcrowd19 Other predictions using Machine Learning Best quality / budget tradeoff Best match between job and crowd member Expected quality When will a job finish (even before it starts) Quality Time Cost
  • 20. definedcrowd Intelligent data for AI contacts: joao@definedcrowd.com daniela@definedcrowd.com mail@definedcrowd.com

Editor's Notes

  1. Does this sound familiar? How many of you have tried a speech recognition system? How many of you are still using it?
  2. Different dialects, age and gender balanced
  3. Being Speech and NLP our core expertize. But we’re moving fast to other industry verticals.
  4. Customer side
  5. Students have proven to be more reliable; work flexibility; managed crowd – we validate who they are
  6. School they attend Course they are enrolled City, country where they live Languages they speak Age Gender Dialect Performance in the job - Qualifications tests, RTAs, agreement scores Jobs they took Speed Activity in our platform