Data Mining The Social Web With APIs and Analysis

•

0 likes•839 views

E

Gary Short introduces data mining of the social web. He explains how to access public APIs to collect tweets mentioning brands like Apple, Google, and Microsoft. Short then preprocesses and stores the tweets in CouchDB for analysis and visualization. As an example, he analyzes around 6,000 tweets to look at lexical diversity by brand, popular words, top posters, and geocoded tweets. He is available for any questions.

Data
Mining
The
Social
Web

By
Gary
Short

Developer
Evangelist

Developer
Express

Introduc>on

•  Gary
Short

•  Developer
Evangelist

•  gary@garyshort.org

•  garys@devexpress.com

•  www.garyshort.org

•  @twiCer

What
About
You
Guys?

Why
Bother
Data
Mining
The
Social
Web?

How
do
we
Data
Mine
The
Social
Web?

Public
API

Data
Store

Analysis

Visualise

Beyond

Visualisa>on

My
Set
UP

Public
API

cUrl
to
ﬁle

Preprocessing

Store
to

CouchDB

Analysis
&

Visualisa>on

Today’s
Sample

•  ~6,000
tweets

•  Men>on

–  Apple

–  Google

–  MicrosoS

•  Doesn’t
include
last
night’s
announcement
J

Lexical
Diversity

Compe>>ve
Analysis

Top
10
Popular
Words

Top
10
Posters
by
Volume

Propor>on
Of
Geo
Coded
Tweets

Ques>ons

•  gary@garyshort.org

•  garys@devexpress.com

•  @twiCer

More Related Content

Similar to Data Mining The Social Web With APIs and Analysis

Digital strategy 11 april 2013.final

Digital strategy 11 april 2013.final

Digital strategy 11 april 2013.finalDave Hazlehurst

Tools for Entrepreneurs: Create. Collaborate. Communicate.

Tools for Entrepreneurs: Create. Collaborate. Communicate.

Tools for Entrepreneurs: Create. Collaborate. Communicate.Sara Rosso

Convergence of Social & Search - PubCon Vegas 2012

Convergence of Social & Search - PubCon Vegas 2012

Convergence of Social & Search - PubCon Vegas 2012David Wallace

Un:Think Introduction

Un:Think Introduction

Un:Think IntroductionVMware Un:Think

Blueprint for integrating big data analytics and bi

Blueprint for integrating big data analytics and bi

Blueprint for integrating big data analytics and biDataWorks Summit

State of CC Search (GS 2019)

State of CC Search (GS 2019)

State of CC Search (GS 2019)Jane Park

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for DrupalAndy Kucharski

Fast Delivery DevOps Israel

Fast Delivery DevOps Israel

Fast Delivery DevOps IsraelAdrian Cockcroft

Future of Web Apps (FOWA) 2009 Review

Future of Web Apps (FOWA) 2009 Review

Future of Web Apps (FOWA) 2009 ReviewGene Crawford

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for DrupalPromet Source

SEO EvolutioniAcquire

Managing your web content successfully

Managing your web content successfully

Managing your web content successfullyCampbell Wilson

DRUPAL SCALES. BIG TIME!

DRUPAL SCALES. BIG TIME!

DRUPAL SCALES. BIG TIME!Srijan Technologies

Harnessing the Power of Social Media

Harnessing the Power of Social Media

Harnessing the Power of Social MediaSheena Koshy

Mobile Apps for Events

Mobile Apps for Events

Mobile Apps for EventsJessica Levin

SEO and Influencer Marketing - Better Together

SEO and Influencer Marketing - Better Together

SEO and Influencer Marketing - Better TogethergShift

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the Difference

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the Difference

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the DifferenceAlterian

Digital Public Relations (PR): 46th Anvil Awards Briefing

Digital Public Relations (PR): 46th Anvil Awards Briefing

Digital Public Relations (PR): 46th Anvil Awards BriefingJanette Toral

Inbound Marketing - Presentation to Brisbane Web Design

Inbound Marketing - Presentation to Brisbane Web Design

Inbound Marketing - Presentation to Brisbane Web DesignMatter Solutions

More than the Sum of its parts, the API's whole

More than the Sum of its parts, the API's whole

More than the Sum of its parts, the API's whole3scale

Similar to Data Mining The Social Web With APIs and Analysis (20)

Digital strategy 11 april 2013.final

Digital strategy 11 april 2013.final

Digital strategy 11 april 2013.final

Tools for Entrepreneurs: Create. Collaborate. Communicate.

Tools for Entrepreneurs: Create. Collaborate. Communicate.

Tools for Entrepreneurs: Create. Collaborate. Communicate.

Convergence of Social & Search - PubCon Vegas 2012

Convergence of Social & Search - PubCon Vegas 2012

Convergence of Social & Search - PubCon Vegas 2012

Un:Think Introduction

Un:Think Introduction

Un:Think Introduction

Blueprint for integrating big data analytics and bi

Blueprint for integrating big data analytics and bi

Blueprint for integrating big data analytics and bi

State of CC Search (GS 2019)

State of CC Search (GS 2019)

State of CC Search (GS 2019)

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

Fast Delivery DevOps Israel

Fast Delivery DevOps Israel

Fast Delivery DevOps Israel

Future of Web Apps (FOWA) 2009 Review

Future of Web Apps (FOWA) 2009 Review

Future of Web Apps (FOWA) 2009 Review

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

Front End page speed performance improvements for Drupal

SEO Evolution

Managing your web content successfully

Managing your web content successfully

Managing your web content successfully

DRUPAL SCALES. BIG TIME!

DRUPAL SCALES. BIG TIME!

DRUPAL SCALES. BIG TIME!

Harnessing the Power of Social Media

Harnessing the Power of Social Media

Harnessing the Power of Social Media

Mobile Apps for Events

Mobile Apps for Events

Mobile Apps for Events

SEO and Influencer Marketing - Better Together

SEO and Influencer Marketing - Better Together

SEO and Influencer Marketing - Better Together

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the Difference

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the Difference

Web Metrics vs Web Behavioral Analytics and Why You Need to Know the Difference

Digital Public Relations (PR): 46th Anvil Awards Briefing

Digital Public Relations (PR): 46th Anvil Awards Briefing

Digital Public Relations (PR): 46th Anvil Awards Briefing

Inbound Marketing - Presentation to Brisbane Web Design

Inbound Marketing - Presentation to Brisbane Web Design

Inbound Marketing - Presentation to Brisbane Web Design

More than the Sum of its parts, the API's whole

More than the Sum of its parts, the API's whole

More than the Sum of its parts, the API's whole

More from ESUG

Workshop: Identifying concept inventories in agile programming

Workshop: Identifying concept inventories in agile programming

Workshop: Identifying concept inventories in agile programmingESUG

Technical documentation support in Pharo

Technical documentation support in Pharo

Technical documentation support in PharoESUG

The Pharo Debugger and Debugging tools: Advances and Roadmap

The Pharo Debugger and Debugging tools: Advances and Roadmap

The Pharo Debugger and Debugging tools: Advances and RoadmapESUG

Sequence: Pipeline modelling in Pharo

Sequence: Pipeline modelling in Pharo

Sequence: Pipeline modelling in PharoESUG

Migration process from monolithic to micro frontend architecture in mobile ap...

Migration process from monolithic to micro frontend architecture in mobile ap...

Migration process from monolithic to micro frontend architecture in mobile ap...ESUG

Analyzing Dart Language with Pharo: Report and early results

Analyzing Dart Language with Pharo: Report and early results

Analyzing Dart Language with Pharo: Report and early resultsESUG

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6ESUG

A Unit Test Metamodel for Test Generation

A Unit Test Metamodel for Test Generation

A Unit Test Metamodel for Test GenerationESUG

Creating Unit Tests Using Genetic Programming

Creating Unit Tests Using Genetic Programming

Creating Unit Tests Using Genetic ProgrammingESUG

Threaded-Execution and CPS Provide Smooth Switching Between Execution Modes

Threaded-Execution and CPS Provide Smooth Switching Between Execution Modes

Threaded-Execution and CPS Provide Smooth Switching Between Execution ModesESUG

Exploring GitHub Actions through EGAD: An Experience Report

Exploring GitHub Actions through EGAD: An Experience Report

Exploring GitHub Actions through EGAD: An Experience ReportESUG

Pharo: a reflective language A first systematic analysis of reflective APIs

Pharo: a reflective language A first systematic analysis of reflective APIs

Pharo: a reflective language A first systematic analysis of reflective APIsESUG

Garbage Collector Tuning

Garbage Collector Tuning

Garbage Collector TuningESUG

Improving Performance Through Object Lifetime Profiling: the DataFrame Case

Improving Performance Through Object Lifetime Profiling: the DataFrame Case

Improving Performance Through Object Lifetime Profiling: the DataFrame CaseESUG

Pharo DataFrame: Past, Present, and Future

Pharo DataFrame: Past, Present, and Future

Pharo DataFrame: Past, Present, and FutureESUG

thisContext in the Debugger

thisContext in the Debugger

thisContext in the DebuggerESUG

Websockets for Fencing Score

Websockets for Fencing Score

Websockets for Fencing ScoreESUG

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScript

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScript

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScriptESUG

Advanced Object- Oriented Design Mooc

Advanced Object- Oriented Design Mooc

Advanced Object- Oriented Design MoocESUG

A New Architecture Reconciling Refactorings and Transformations

A New Architecture Reconciling Refactorings and Transformations

A New Architecture Reconciling Refactorings and TransformationsESUG

More from ESUG (20)

Workshop: Identifying concept inventories in agile programming

Workshop: Identifying concept inventories in agile programming

Workshop: Identifying concept inventories in agile programming

Technical documentation support in Pharo

Technical documentation support in Pharo

Technical documentation support in Pharo

The Pharo Debugger and Debugging tools: Advances and Roadmap

The Pharo Debugger and Debugging tools: Advances and Roadmap

The Pharo Debugger and Debugging tools: Advances and Roadmap

Sequence: Pipeline modelling in Pharo

Sequence: Pipeline modelling in Pharo

Sequence: Pipeline modelling in Pharo

Migration process from monolithic to micro frontend architecture in mobile ap...

Migration process from monolithic to micro frontend architecture in mobile ap...

Migration process from monolithic to micro frontend architecture in mobile ap...

Analyzing Dart Language with Pharo: Report and early results

Analyzing Dart Language with Pharo: Report and early results

Analyzing Dart Language with Pharo: Report and early results

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6

Transpiling Pharo Classes to JS ECMAScript 5 versus ECMAScript 6

A Unit Test Metamodel for Test Generation

A Unit Test Metamodel for Test Generation

A Unit Test Metamodel for Test Generation

Creating Unit Tests Using Genetic Programming

Creating Unit Tests Using Genetic Programming

Creating Unit Tests Using Genetic Programming

Threaded-Execution and CPS Provide Smooth Switching Between Execution Modes

Threaded-Execution and CPS Provide Smooth Switching Between Execution Modes

Threaded-Execution and CPS Provide Smooth Switching Between Execution Modes

Exploring GitHub Actions through EGAD: An Experience Report

Exploring GitHub Actions through EGAD: An Experience Report

Exploring GitHub Actions through EGAD: An Experience Report

Pharo: a reflective language A first systematic analysis of reflective APIs

Pharo: a reflective language A first systematic analysis of reflective APIs

Pharo: a reflective language A first systematic analysis of reflective APIs

Garbage Collector Tuning

Garbage Collector Tuning

Garbage Collector Tuning

Improving Performance Through Object Lifetime Profiling: the DataFrame Case

Improving Performance Through Object Lifetime Profiling: the DataFrame Case

Improving Performance Through Object Lifetime Profiling: the DataFrame Case

Pharo DataFrame: Past, Present, and Future

Pharo DataFrame: Past, Present, and Future

Pharo DataFrame: Past, Present, and Future

thisContext in the Debugger

thisContext in the Debugger

thisContext in the Debugger

Websockets for Fencing Score

Websockets for Fencing Score

Websockets for Fencing Score

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScript

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScript

ShowUs: PharoJS.org Develop in Pharo, Run on JavaScript

Advanced Object- Oriented Design Mooc

Advanced Object- Oriented Design Mooc

Advanced Object- Oriented Design Mooc

A New Architecture Reconciling Refactorings and Transformations

A New Architecture Reconciling Refactorings and Transformations

A New Architecture Reconciling Refactorings and Transformations

Recently uploaded

Nell’iperspazio con Rocket: il Framework Web di Rust!

Nell’iperspazio con Rocket: il Framework Web di Rust!

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

Dev Dives: Streamline document processing with UiPath Studio Web

Dev Dives: Streamline document processing with UiPath Studio Web

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

Gen AI in Business - Global Trends Report 2024.pdf

Gen AI in Business - Global Trends Report 2024.pdf

Gen AI in Business - Global Trends Report 2024.pdfAddepto

Artificial intelligence in cctv survelliance.pptx

Artificial intelligence in cctv survelliance.pptx

Artificial intelligence in cctv survelliance.pptxhariprasad279825

Vertex AI Gemini Prompt Engineering Tips

Vertex AI Gemini Prompt Engineering Tips

Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi

Human Factors of XR: Using Human Factors to Design XR Systems

Human Factors of XR: Using Human Factors to Design XR Systems

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Scanning the Internet for External Cloud Exposures via SSL Certs

Scanning the Internet for External Cloud Exposures via SSL Certs

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Are Multi-Cloud and Serverless Good or Bad?

Are Multi-Cloud and Serverless Good or Bad?

Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson

DevEX - reference for building teams, processes, and platforms

DevEX - reference for building teams, processes, and platforms

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

Connect Wave/ connectwave Pitch Deck Presentation

Connect Wave/ connectwave Pitch Deck Presentation

Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation

Powerpoint exploring the locations used in television show Time Clash

Powerpoint exploring the locations used in television show Time Clash

Powerpoint exploring the locations used in television show Time Clashcharlottematthew16

DevoxxFR 2024 Reproducible Builds with Apache Maven

DevoxxFR 2024 Reproducible Builds with Apache Maven

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

Unleash Your Potential - Namagunga Girls Coding Club

Unleash Your Potential - Namagunga Girls Coding Club

Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar

Developer Data Modeling Mistakes: From Postgres to NoSQL

Developer Data Modeling Mistakes: From Postgres to NoSQL

Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3

Anypoint Exchange: It’s Not Just a Repo!

Anypoint Exchange: It’s Not Just a Repo!

Anypoint Exchange: It’s Not Just a Repo!Manik S Magar

CloudStudio User manual (basic edition):

CloudStudio User manual (basic edition):

CloudStudio User manual (basic edition):comworks

Ensuring Technical Readiness For Copilot in Microsoft 365

Ensuring Technical Readiness For Copilot in Microsoft 365

Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited

WordPress Websites for Engineers: Elevate Your Brandgvaughan

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

Recently uploaded (20)

Nell’iperspazio con Rocket: il Framework Web di Rust!

Nell’iperspazio con Rocket: il Framework Web di Rust!

Nell’iperspazio con Rocket: il Framework Web di Rust!

Dev Dives: Streamline document processing with UiPath Studio Web

Dev Dives: Streamline document processing with UiPath Studio Web

Dev Dives: Streamline document processing with UiPath Studio Web

Gen AI in Business - Global Trends Report 2024.pdf

Gen AI in Business - Global Trends Report 2024.pdf

Gen AI in Business - Global Trends Report 2024.pdf

Artificial intelligence in cctv survelliance.pptx

Artificial intelligence in cctv survelliance.pptx

Artificial intelligence in cctv survelliance.pptx

Vertex AI Gemini Prompt Engineering Tips

Vertex AI Gemini Prompt Engineering Tips

Vertex AI Gemini Prompt Engineering Tips

Human Factors of XR: Using Human Factors to Design XR Systems

Human Factors of XR: Using Human Factors to Design XR Systems

Human Factors of XR: Using Human Factors to Design XR Systems

Scanning the Internet for External Cloud Exposures via SSL Certs

Scanning the Internet for External Cloud Exposures via SSL Certs

Scanning the Internet for External Cloud Exposures via SSL Certs

Are Multi-Cloud and Serverless Good or Bad?

Are Multi-Cloud and Serverless Good or Bad?

Are Multi-Cloud and Serverless Good or Bad?

DevEX - reference for building teams, processes, and platforms

DevEX - reference for building teams, processes, and platforms

DevEX - reference for building teams, processes, and platforms

Connect Wave/ connectwave Pitch Deck Presentation

Connect Wave/ connectwave Pitch Deck Presentation

Connect Wave/ connectwave Pitch Deck Presentation

Powerpoint exploring the locations used in television show Time Clash

Powerpoint exploring the locations used in television show Time Clash

Powerpoint exploring the locations used in television show Time Clash

DevoxxFR 2024 Reproducible Builds with Apache Maven

DevoxxFR 2024 Reproducible Builds with Apache Maven

DevoxxFR 2024 Reproducible Builds with Apache Maven

Unleash Your Potential - Namagunga Girls Coding Club

Unleash Your Potential - Namagunga Girls Coding Club

Unleash Your Potential - Namagunga Girls Coding Club

Developer Data Modeling Mistakes: From Postgres to NoSQL

Developer Data Modeling Mistakes: From Postgres to NoSQL

Developer Data Modeling Mistakes: From Postgres to NoSQL

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx

Anypoint Exchange: It’s Not Just a Repo!

Anypoint Exchange: It’s Not Just a Repo!

Anypoint Exchange: It’s Not Just a Repo!

CloudStudio User manual (basic edition):

CloudStudio User manual (basic edition):

CloudStudio User manual (basic edition):

Ensuring Technical Readiness For Copilot in Microsoft 365

Ensuring Technical Readiness For Copilot in Microsoft 365

Ensuring Technical Readiness For Copilot in Microsoft 365

WordPress Websites for Engineers: Elevate Your Brand

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

Data Mining The Social Web With APIs and Analysis

1. Data Mining The Social Web By Gary Short Developer Evangelist Developer Express

2. Introduc>on •  Gary Short •  Developer Evangelist •  gary@garyshort.org •  garys@devexpress.com •  www.garyshort.org •  @twiCer

3. What About You Guys?

4. Why Bother Data Mining The Social Web?

5. How do we Data Mine The Social Web? Public API Data Store Analysis Visualise Beyond Visualisa>on

6. My Set UP Public API cUrl to ﬁle Preprocessing Store to CouchDB Analysis & Visualisa>on

7. Today’s Sample •  ~6,000 tweets •  Men>on –  Apple –  Google –  MicrosoS •  Doesn’t include last night’s announcement J

8. Lexical Diversity

9. Compe>>ve Analysis

10. Lexical Diversity by Brand

11. Top 10 Popular Words

12. Top 10 Posters by Volume

13. Propor>on Of Geo Coded Tweets

14.

15. Ques>ons •  gary@garyshort.org •  garys@devexpress.com •  @twiCer