SlideShare a Scribd company logo
1 of 22
Download to read offline
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DISIT lab, IEEE SCI 2017, Freemont CA USA
Daniele Cenni, Paolo Nesi, Gianni Pantaleo, Imad Zaza
University of Florence, Department of Information Engineering,
DISIT Lab, http://www.disit.org ,
http://www.sii-mobility.org , http://www.km4city.org
paolo.nesi@unifi.it
Twitter Vigilance: a Multi-User platform for Cross-
Domain Twitter Data Analytics, NLP and Sentiment
Analysis
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Exploiting Social Media Data
• Mainly Natural Language (multiple), specific slang,
– E.g., Twitter with its # Hashtags, @ citations, etc.
– Most of the posts are scarcely geolocated
• Main Domain Analysis
– Social and market analysis
– Predictive model
– Early warning, anomaly detection
• Derived Metrics may be of many kind and have to be validate to
use them
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Prediction/Assessment
• Football game results as related to the volume of Tweets
• Number of votes on political elections,
via sentiment analysis, SA
• Size and inception of contagious diseases
• marketability of consumer goods
• public health seasonal flu
• box-office revenues for movies
• places to be visited, most visited
• number of people in locations like airports
• audience of TV programmes, political TV shows
• weather forecast information
• Appreciation of services
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Twitter Vigilance
• http://www.disit.org/tv
• http://www.disit.org/rttv
• Citizens as sensors to
– Assess sentiment on services,
events, …
– Response of consumers wrt…
– Early detection of critical
conditions
– Information channel
– Opinion leaders
– Communities
– Formation
– Predicting volume of visitors for
tuning the services
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Requirements
• Collecting Tweets
–on the basis of several criterial, searches
• Multiple users may have multiple searches and multiple purposes
(views on those searches)  minimization of searches
–With high reliable model exploiting Twitter Search and/or
Stream API
• Performing NLP and Sentiment Analysis
–Real time or daily
–Multiple languages
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
State of the art
DISIT lab, IEEE SCI 2017, Freemont CA USA
Service
Twitter
Metrics(e.g.#
oftweets,
retweetsover
time)
Sentiment
analysis
NLPAnalysis
API
availability
Usernetwork
analysis
Dataanalysis
basedon
geolocation
Realtime
Analytics
Fullfaceted
Search
Metricsfor
assessingrecall
efficiency
Minimization
ofsearchesto
Twitter
SAS N Y N Y N N Y N N na
Keyhole
Aggre-
gate
N N N
Aggre
-gate
Y N N N na
Tweetreach
Aggre-
gate
N N N
Aggre
-gate
Y N N N na
Brandwatch N N N N Y Y Y N N na
Followewonk N N N N Y Y Y N N na
Twitris N Y N N N Y Y N N na
OSoMe Y N N Y Y Y Y N N na
Twitter Vigilance Y Y Y Y Y N Y Y Y Y
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Twitter Vigilance Public Views
• TV: Twitter Vigilance main tool (http://disit.org/tv/),
collecting and analyzing tweets daily;
• RTTV: Real-time twitter Vigilance (http://disit.org/rttv/),
collecting and analyzing tweets in real time;
• TVSolr: Twitter Vigilance Advanced search
• (http://tvsolr.disit.org/), indexing tweets and faceted
search
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Architecture
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Twitter Vigilance Users can
• create and edit customized channels as a collection of searches
on API
– Per channel and per search
• crawls tweets, computes metrics, and shows results of Twitter
Data, as: volume metrics about tweets, retweets and user
statistics, NLP and Sentiment Analyses based metrics
• provides public access to metric results computed on channels
and search analysis
• Allows the researchers to download resulting metrics values
(through API service) over time for further analysis
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Several Channels
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
A Channel
DISIT lab, IEEE SCI 2017, Freemont CA USA
Its searches
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Twitter Syntax for Searches
• String substring: Caldo
• Hashtag: #Caldo,
• Citations: @CivilProtection, @paolonesi
• From users: From:@paolonesi
• Etc.
• ….ANDed and ORed
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Metrics’ Kinds
• Volume Metrics
– Number of TW, number of RTW
• User Metrics
– Number of distinct users
– Number of followers, following
• NLP and SA metrics
– Counting word, adjective, noun, verbs, ….
– Estimating SA, weighting with SentiWordNet (extended to Italian)
• High level metrics (compositing all the other metrics)
– Addition of metrics..
– Ratio among metrics, e.g.: num of TW/num of RTW,…
– Cumulated metrics over time, e.g.: number of TW in the last X days..
• All: (i) per day, per hour, etc. (ii) per channel, per search
• Recently: we added the possibility of using metrics as firing conditions for alerts and
bot on Twitter.
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Problem addressed
Strong Limitations of the Search API of Twitter
• minimizing the number of searches on the basis of the user requests:
– different users with their queries request tweets already requested by others
• Recovering of parent Tweets from Orphan reTweets taken in the
searching process
Analytics:
• High performance solution based on HDFS, Hadoop for NLP and SA,
exploiting MapReduce programming model
• Estimating the network of influencer
• Computing metrics and prediction in real time.
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Sentiment Analysis
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org Influence Network
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DISIT lab, IEEE SCI 2017, Freemont CA USA
Early Warning
Predictive models
Hot flows
Attendance at long lasting events: EXPO2015
Attendance at recurrent events: TV, footbal
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Efficiency in retrieval
DISIT lab, IEEE SCI 2017, Freemont CA USA
Posts
Volume
(Tweets +
Retweets)
Range
#
Recovered
Original
Tweets
# Missing
Original
Tweets
% Original
Tweets
Coverage
(CoTWO)
# Twitter
Search
API
requests
# Saturations
on Twitter
Search API
requests
% Saturations
on Twitter
Search API
requests (S%)
% Not-Saturated
Twitter Search
API requests (1-
S%)
< 10k 18571 2033 89,05% 124299 1 0,00% 100,00%
[ 10k, 50k
)
130051 13716 89,45% 399170 100 0,03% 99,97%
[ 50k, 100k
)
96171 10278 89,31% 123804 165 0,13% 99,87%
[ 100k,
500k )
997833 86755 91,31% 849062 1589 0,19% 99,81%
[ 500k, 1M
)
930646 61632 93,38% 439956 1998 0,45% 99,55%
[ 1M, 5M ) 6454463 439628 93,19% 2787485 31585 1,13% 98,87%
> 5M 14714124 899035 93,89% 4509184 64284 1,43% 98,57%
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Original Tweets coverage and Twitter Search API
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dependance on RTW/TW ratio
DISIT lab, IEEE SCI 2017, Freemont CA USA
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Conclusions
• Twitter Vigilance is now operative since 2 years with many institutional users:
ARPAT, LAMMA, UNIFI, CNR,..
• It presents an high efficiency in recovering twitter data despite to the
complexity and provided API.
• It has been used/validated with data coming from several scenarios and
domains
• for early warning and prediction in the domain of:
– social communication, hot in Tuscany, rain measures, etc.
– Disaster alerts: water bomb
– TV audience (X factor, etc.), large events as Expo 2015
• New version is providing direct metrics estimation which can be composed by
users, and resulting data can be downloaded
DISIT lab, IEEE SCI 2017, Freemont CA USA

More Related Content

What's hot

DISIT Lab overview: smart city, big data, semantic computing, cloud
DISIT Lab overview: smart city, big data, semantic computing, cloudDISIT Lab overview: smart city, big data, semantic computing, cloud
DISIT Lab overview: smart city, big data, semantic computing, cloudPaolo Nesi
 
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Paolo Nesi
 
Smart City and Open Data Projects and tools of DISIT Lab
Smart City and Open Data Projects and tools of DISIT LabSmart City and Open Data Projects and tools of DISIT Lab
Smart City and Open Data Projects and tools of DISIT LabPaolo Nesi
 
Km4City: Smart City HowTo and Overview, 2016
Km4City: Smart City HowTo and Overview, 2016Km4City: Smart City HowTo and Overview, 2016
Km4City: Smart City HowTo and Overview, 2016Paolo Nesi
 
Km4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of ServicesKm4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of ServicesPaolo Nesi
 
Smart City at DISIT Lab, step two after smart city for beginners
Smart City at DISIT Lab, step two after smart city for beginnersSmart City at DISIT Lab, step two after smart city for beginners
Smart City at DISIT Lab, step two after smart city for beginnersPaolo Nesi
 
Km4City Smart City API: an integrated support for mobility services
Km4City Smart City API: an integrated support for mobility servicesKm4City Smart City API: an integrated support for mobility services
Km4City Smart City API: an integrated support for mobility servicesPaolo Nesi
 
Functional Resonance Analysis Method based- Decision Support tool for Urban T...
Functional Resonance Analysis Method based- Decision Support tool for Urban T...Functional Resonance Analysis Method based- Decision Support tool for Urban T...
Functional Resonance Analysis Method based- Decision Support tool for Urban T...Paolo Nesi
 
Overview on Smart City: Smart City for Beginners
Overview on Smart City: Smart City for BeginnersOverview on Smart City: Smart City for Beginners
Overview on Smart City: Smart City for BeginnersPaolo Nesi
 
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services""Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services"diannepatricia
 
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...Paolo Nesi
 
Smart Cloud Engine and Solution based on Knowledge Base
Smart Cloud Engine and Solution based on Knowledge BaseSmart Cloud Engine and Solution based on Knowledge Base
Smart Cloud Engine and Solution based on Knowledge BasePaolo Nesi
 
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...Paolo Nesi
 
Smart Cities and Data Analytics: Challenges and Opportunities
Smart Cities and Data Analytics: Challenges and Opportunities Smart Cities and Data Analytics: Challenges and Opportunities
Smart Cities and Data Analytics: Challenges and Opportunities PayamBarnaghi
 
Smart Cities….Smart Future
Smart Cities….Smart FutureSmart Cities….Smart Future
Smart Cities….Smart FuturePayamBarnaghi
 
The Future is Cyber-Healthcare
The Future is Cyber-Healthcare The Future is Cyber-Healthcare
The Future is Cyber-Healthcare PayamBarnaghi
 
What makes smart cities “Smart”?
What makes smart cities “Smart”? What makes smart cities “Smart”?
What makes smart cities “Smart”? PayamBarnaghi
 
How to make cities "smarter"?
How to make cities "smarter"?How to make cities "smarter"?
How to make cities "smarter"?PayamBarnaghi
 
Smart Cities: How are they different?
Smart Cities: How are they different? Smart Cities: How are they different?
Smart Cities: How are they different? PayamBarnaghi
 

What's hot (20)

DISIT Lab overview: smart city, big data, semantic computing, cloud
DISIT Lab overview: smart city, big data, semantic computing, cloudDISIT Lab overview: smart city, big data, semantic computing, cloud
DISIT Lab overview: smart city, big data, semantic computing, cloud
 
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
 
Smart City and Open Data Projects and tools of DISIT Lab
Smart City and Open Data Projects and tools of DISIT LabSmart City and Open Data Projects and tools of DISIT Lab
Smart City and Open Data Projects and tools of DISIT Lab
 
Km4City: Smart City HowTo and Overview, 2016
Km4City: Smart City HowTo and Overview, 2016Km4City: Smart City HowTo and Overview, 2016
Km4City: Smart City HowTo and Overview, 2016
 
Km4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of ServicesKm4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of Services
 
Smart City at DISIT Lab, step two after smart city for beginners
Smart City at DISIT Lab, step two after smart city for beginnersSmart City at DISIT Lab, step two after smart city for beginners
Smart City at DISIT Lab, step two after smart city for beginners
 
Km4City Smart City API: an integrated support for mobility services
Km4City Smart City API: an integrated support for mobility servicesKm4City Smart City API: an integrated support for mobility services
Km4City Smart City API: an integrated support for mobility services
 
Functional Resonance Analysis Method based- Decision Support tool for Urban T...
Functional Resonance Analysis Method based- Decision Support tool for Urban T...Functional Resonance Analysis Method based- Decision Support tool for Urban T...
Functional Resonance Analysis Method based- Decision Support tool for Urban T...
 
Overview on Smart City: Smart City for Beginners
Overview on Smart City: Smart City for BeginnersOverview on Smart City: Smart City for Beginners
Overview on Smart City: Smart City for Beginners
 
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services""Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
 
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...
Overview on Smart City, DISIT lab solution for beginners, 2015, Part 7: Distr...
 
Smart Cloud Engine and Solution based on Knowledge Base
Smart Cloud Engine and Solution based on Knowledge BaseSmart Cloud Engine and Solution based on Knowledge Base
Smart Cloud Engine and Solution based on Knowledge Base
 
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
 
Smart Cities and Data Analytics: Challenges and Opportunities
Smart Cities and Data Analytics: Challenges and Opportunities Smart Cities and Data Analytics: Challenges and Opportunities
Smart Cities and Data Analytics: Challenges and Opportunities
 
Smart Cities….Smart Future
Smart Cities….Smart FutureSmart Cities….Smart Future
Smart Cities….Smart Future
 
Long uglytestingdeck
Long uglytestingdeckLong uglytestingdeck
Long uglytestingdeck
 
The Future is Cyber-Healthcare
The Future is Cyber-Healthcare The Future is Cyber-Healthcare
The Future is Cyber-Healthcare
 
What makes smart cities “Smart”?
What makes smart cities “Smart”? What makes smart cities “Smart”?
What makes smart cities “Smart”?
 
How to make cities "smarter"?
How to make cities "smarter"?How to make cities "smarter"?
How to make cities "smarter"?
 
Smart Cities: How are they different?
Smart Cities: How are they different? Smart Cities: How are they different?
Smart Cities: How are they different?
 

Similar to Twitter Vigilance: a Multi-User platform for Cross-Domain Twitter Data Analytics, NLP and Sentiment Analysis

Graph Databases Lifecycle Methodology and Tool to Support Index/Store Versio...
Graph Databases Lifecycle Methodology  and Tool to Support Index/Store Versio...Graph Databases Lifecycle Methodology  and Tool to Support Index/Store Versio...
Graph Databases Lifecycle Methodology and Tool to Support Index/Store Versio...Paolo Nesi
 
Knowledge mining and Semantic Models: from Cloud to Smart City
Knowledge mining and Semantic Models: from Cloud to Smart CityKnowledge mining and Semantic Models: from Cloud to Smart City
Knowledge mining and Semantic Models: from Cloud to Smart CityPaolo Nesi
 
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...Paolo Nesi
 
Ontology Building vs Data Harvesting and Cleaning for Smart-city Services
Ontology Building vs Data Harvesting and Cleaning for Smart-city ServicesOntology Building vs Data Harvesting and Cleaning for Smart-city Services
Ontology Building vs Data Harvesting and Cleaning for Smart-city ServicesPaolo Nesi
 
DAI DATI INTELLIGENTI AI SERVIZI Smart City API Hackathon
DAI DATI INTELLIGENTI AI SERVIZI Smart City API HackathonDAI DATI INTELLIGENTI AI SERVIZI Smart City API Hackathon
DAI DATI INTELLIGENTI AI SERVIZI Smart City API HackathonPaolo Nesi
 
RESOLUTE: Resilience management guidelines and Operationalization applied to ...
RESOLUTE: Resilience management guidelines and Operationalization applied to ...RESOLUTE: Resilience management guidelines and Operationalization applied to ...
RESOLUTE: Resilience management guidelines and Operationalization applied to ...Paolo Nesi
 
Data extraction tools
Data extraction toolsData extraction tools
Data extraction toolsCristian Ruiz
 
Technologies for Enhancing Knowledge and Training, the future of e-learning t...
Technologies for Enhancing Knowledge and Training, the future of e-learning t...Technologies for Enhancing Knowledge and Training, the future of e-learning t...
Technologies for Enhancing Knowledge and Training, the future of e-learning t...Paolo Nesi
 
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...Paolo Nesi
 
Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View Paolo Nesi
 
DISIT lab Overview on Tourism and Training, June 2014
DISIT lab Overview on Tourism and Training, June 2014DISIT lab Overview on Tourism and Training, June 2014
DISIT lab Overview on Tourism and Training, June 2014Paolo Nesi
 
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...Paolo Nesi
 
RESOLUTE: Governing for Resilience – Implementation Challenges
RESOLUTE: Governing for Resilience – Implementation Challenges RESOLUTE: Governing for Resilience – Implementation Challenges
RESOLUTE: Governing for Resilience – Implementation Challenges Paolo Nesi
 
Open Urban Platform: Technical View 2018: Km4City
Open Urban Platform: Technical View 2018: Km4CityOpen Urban Platform: Technical View 2018: Km4City
Open Urban Platform: Technical View 2018: Km4CityPaolo Nesi
 
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...Paolo Nesi
 
SuanIct-Bigdata desktop-final
SuanIct-Bigdata desktop-finalSuanIct-Bigdata desktop-final
SuanIct-Bigdata desktop-finalstelligence
 
MBA-TU-Thailand:BigData for business startup.
MBA-TU-Thailand:BigData for business startup.MBA-TU-Thailand:BigData for business startup.
MBA-TU-Thailand:BigData for business startup.stelligence
 
JIMS Rohini IT Flash Monthly Newsletter - October Issue
JIMS Rohini IT Flash Monthly Newsletter  - October IssueJIMS Rohini IT Flash Monthly Newsletter  - October Issue
JIMS Rohini IT Flash Monthly Newsletter - October IssueJIMS Rohini Sector 5
 

Similar to Twitter Vigilance: a Multi-User platform for Cross-Domain Twitter Data Analytics, NLP and Sentiment Analysis (20)

Graph Databases Lifecycle Methodology and Tool to Support Index/Store Versio...
Graph Databases Lifecycle Methodology  and Tool to Support Index/Store Versio...Graph Databases Lifecycle Methodology  and Tool to Support Index/Store Versio...
Graph Databases Lifecycle Methodology and Tool to Support Index/Store Versio...
 
Knowledge mining and Semantic Models: from Cloud to Smart City
Knowledge mining and Semantic Models: from Cloud to Smart CityKnowledge mining and Semantic Models: from Cloud to Smart City
Knowledge mining and Semantic Models: from Cloud to Smart City
 
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...
NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...
 
Ontology Building vs Data Harvesting and Cleaning for Smart-city Services
Ontology Building vs Data Harvesting and Cleaning for Smart-city ServicesOntology Building vs Data Harvesting and Cleaning for Smart-city Services
Ontology Building vs Data Harvesting and Cleaning for Smart-city Services
 
DAI DATI INTELLIGENTI AI SERVIZI Smart City API Hackathon
DAI DATI INTELLIGENTI AI SERVIZI Smart City API HackathonDAI DATI INTELLIGENTI AI SERVIZI Smart City API Hackathon
DAI DATI INTELLIGENTI AI SERVIZI Smart City API Hackathon
 
RESOLUTE: Resilience management guidelines and Operationalization applied to ...
RESOLUTE: Resilience management guidelines and Operationalization applied to ...RESOLUTE: Resilience management guidelines and Operationalization applied to ...
RESOLUTE: Resilience management guidelines and Operationalization applied to ...
 
Data extraction tools
Data extraction toolsData extraction tools
Data extraction tools
 
Technologies for Enhancing Knowledge and Training, the future of e-learning t...
Technologies for Enhancing Knowledge and Training, the future of e-learning t...Technologies for Enhancing Knowledge and Training, the future of e-learning t...
Technologies for Enhancing Knowledge and Training, the future of e-learning t...
 
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
 
Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View
 
DISIT lab Overview on Tourism and Training, June 2014
DISIT lab Overview on Tourism and Training, June 2014DISIT lab Overview on Tourism and Training, June 2014
DISIT lab Overview on Tourism and Training, June 2014
 
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...
Rights Enforcement and Licensing Understanding for RDF Stores Aggregating Ope...
 
RESOLUTE: Governing for Resilience – Implementation Challenges
RESOLUTE: Governing for Resilience – Implementation Challenges RESOLUTE: Governing for Resilience – Implementation Challenges
RESOLUTE: Governing for Resilience – Implementation Challenges
 
Open Urban Platform: Technical View 2018: Km4City
Open Urban Platform: Technical View 2018: Km4CityOpen Urban Platform: Technical View 2018: Km4City
Open Urban Platform: Technical View 2018: Km4City
 
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
 
SuanIct-Bigdata desktop-final
SuanIct-Bigdata desktop-finalSuanIct-Bigdata desktop-final
SuanIct-Bigdata desktop-final
 
MBA-TU-Thailand:BigData for business startup.
MBA-TU-Thailand:BigData for business startup.MBA-TU-Thailand:BigData for business startup.
MBA-TU-Thailand:BigData for business startup.
 
Big Data: Big Issues for IP
Big Data: Big Issues for IPBig Data: Big Issues for IP
Big Data: Big Issues for IP
 
JIMS Rohini IT Flash Monthly Newsletter - October Issue
JIMS Rohini IT Flash Monthly Newsletter  - October IssueJIMS Rohini IT Flash Monthly Newsletter  - October Issue
JIMS Rohini IT Flash Monthly Newsletter - October Issue
 
Big Data, open data, IOT
Big Data, open data, IOTBig Data, open data, IOT
Big Data, open data, IOT
 

Recently uploaded

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 

Recently uploaded (17)

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 

Twitter Vigilance: a Multi-User platform for Cross-Domain Twitter Data Analytics, NLP and Sentiment Analysis

  • 1. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, IEEE SCI 2017, Freemont CA USA Daniele Cenni, Paolo Nesi, Gianni Pantaleo, Imad Zaza University of Florence, Department of Information Engineering, DISIT Lab, http://www.disit.org , http://www.sii-mobility.org , http://www.km4city.org paolo.nesi@unifi.it Twitter Vigilance: a Multi-User platform for Cross- Domain Twitter Data Analytics, NLP and Sentiment Analysis
  • 2. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Exploiting Social Media Data • Mainly Natural Language (multiple), specific slang, – E.g., Twitter with its # Hashtags, @ citations, etc. – Most of the posts are scarcely geolocated • Main Domain Analysis – Social and market analysis – Predictive model – Early warning, anomaly detection • Derived Metrics may be of many kind and have to be validate to use them DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 3. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Prediction/Assessment • Football game results as related to the volume of Tweets • Number of votes on political elections, via sentiment analysis, SA • Size and inception of contagious diseases • marketability of consumer goods • public health seasonal flu • box-office revenues for movies • places to be visited, most visited • number of people in locations like airports • audience of TV programmes, political TV shows • weather forecast information • Appreciation of services DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 4. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance • http://www.disit.org/tv • http://www.disit.org/rttv • Citizens as sensors to – Assess sentiment on services, events, … – Response of consumers wrt… – Early detection of critical conditions – Information channel – Opinion leaders – Communities – Formation – Predicting volume of visitors for tuning the services DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 5. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Requirements • Collecting Tweets –on the basis of several criterial, searches • Multiple users may have multiple searches and multiple purposes (views on those searches)  minimization of searches –With high reliable model exploiting Twitter Search and/or Stream API • Performing NLP and Sentiment Analysis –Real time or daily –Multiple languages DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 6. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org State of the art DISIT lab, IEEE SCI 2017, Freemont CA USA Service Twitter Metrics(e.g.# oftweets, retweetsover time) Sentiment analysis NLPAnalysis API availability Usernetwork analysis Dataanalysis basedon geolocation Realtime Analytics Fullfaceted Search Metricsfor assessingrecall efficiency Minimization ofsearchesto Twitter SAS N Y N Y N N Y N N na Keyhole Aggre- gate N N N Aggre -gate Y N N N na Tweetreach Aggre- gate N N N Aggre -gate Y N N N na Brandwatch N N N N Y Y Y N N na Followewonk N N N N Y Y Y N N na Twitris N Y N N N Y Y N N na OSoMe Y N N Y Y Y Y N N na Twitter Vigilance Y Y Y Y Y N Y Y Y Y
  • 7. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance Public Views • TV: Twitter Vigilance main tool (http://disit.org/tv/), collecting and analyzing tweets daily; • RTTV: Real-time twitter Vigilance (http://disit.org/rttv/), collecting and analyzing tweets in real time; • TVSolr: Twitter Vigilance Advanced search • (http://tvsolr.disit.org/), indexing tweets and faceted search DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 8. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Architecture DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 9. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance Users can • create and edit customized channels as a collection of searches on API – Per channel and per search • crawls tweets, computes metrics, and shows results of Twitter Data, as: volume metrics about tweets, retweets and user statistics, NLP and Sentiment Analyses based metrics • provides public access to metric results computed on channels and search analysis • Allows the researchers to download resulting metrics values (through API service) over time for further analysis DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 10. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Several Channels DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 11. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org A Channel DISIT lab, IEEE SCI 2017, Freemont CA USA Its searches
  • 12. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Syntax for Searches • String substring: Caldo • Hashtag: #Caldo, • Citations: @CivilProtection, @paolonesi • From users: From:@paolonesi • Etc. • ….ANDed and ORed DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 13. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Metrics’ Kinds • Volume Metrics – Number of TW, number of RTW • User Metrics – Number of distinct users – Number of followers, following • NLP and SA metrics – Counting word, adjective, noun, verbs, …. – Estimating SA, weighting with SentiWordNet (extended to Italian) • High level metrics (compositing all the other metrics) – Addition of metrics.. – Ratio among metrics, e.g.: num of TW/num of RTW,… – Cumulated metrics over time, e.g.: number of TW in the last X days.. • All: (i) per day, per hour, etc. (ii) per channel, per search • Recently: we added the possibility of using metrics as firing conditions for alerts and bot on Twitter. DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 14. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Problem addressed Strong Limitations of the Search API of Twitter • minimizing the number of searches on the basis of the user requests: – different users with their queries request tweets already requested by others • Recovering of parent Tweets from Orphan reTweets taken in the searching process Analytics: • High performance solution based on HDFS, Hadoop for NLP and SA, exploiting MapReduce programming model • Estimating the network of influencer • Computing metrics and prediction in real time. DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 15. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Sentiment Analysis DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 16. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 17. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Influence Network DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 18. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, IEEE SCI 2017, Freemont CA USA Early Warning Predictive models Hot flows Attendance at long lasting events: EXPO2015 Attendance at recurrent events: TV, footbal
  • 19. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Efficiency in retrieval DISIT lab, IEEE SCI 2017, Freemont CA USA Posts Volume (Tweets + Retweets) Range # Recovered Original Tweets # Missing Original Tweets % Original Tweets Coverage (CoTWO) # Twitter Search API requests # Saturations on Twitter Search API requests % Saturations on Twitter Search API requests (S%) % Not-Saturated Twitter Search API requests (1- S%) < 10k 18571 2033 89,05% 124299 1 0,00% 100,00% [ 10k, 50k ) 130051 13716 89,45% 399170 100 0,03% 99,97% [ 50k, 100k ) 96171 10278 89,31% 123804 165 0,13% 99,87% [ 100k, 500k ) 997833 86755 91,31% 849062 1589 0,19% 99,81% [ 500k, 1M ) 930646 61632 93,38% 439956 1998 0,45% 99,55% [ 1M, 5M ) 6454463 439628 93,19% 2787485 31585 1,13% 98,87% > 5M 14714124 899035 93,89% 4509184 64284 1,43% 98,57%
  • 20. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Original Tweets coverage and Twitter Search API DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 21. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Dependance on RTW/TW ratio DISIT lab, IEEE SCI 2017, Freemont CA USA
  • 22. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Conclusions • Twitter Vigilance is now operative since 2 years with many institutional users: ARPAT, LAMMA, UNIFI, CNR,.. • It presents an high efficiency in recovering twitter data despite to the complexity and provided API. • It has been used/validated with data coming from several scenarios and domains • for early warning and prediction in the domain of: – social communication, hot in Tuscany, rain measures, etc. – Disaster alerts: water bomb – TV audience (X factor, etc.), large events as Expo 2015 • New version is providing direct metrics estimation which can be composed by users, and resulting data can be downloaded DISIT lab, IEEE SCI 2017, Freemont CA USA