SlideShare a Scribd company logo
1 of 45
Download to read offline
Big Data
Tap into Cloud Infrastructure with FME
March 18, 2014
Meet the presenters.
Don Murray
 President and Co-Founder
@DonAtSafe
Dean Hintz
 Senior Product Specialist
@DeanAtSafe
Ask us. And join the discussion.
Please submit using the
GoToWebinar panel.
We will follow-up with
unanswered questions.
Agenda.
 What is Big Data
 Big Data Challenges
 FME and Big Data
 FME Demos:
 Loading and Extracting from
MarkLogic
 Spatial Indexing and Loading to
DynamoDB
Big Data – Tap into Cloud Infrastructure with FME
What we do.
www.safe.com
Poll: Which version of FME
are you using?
New to FME?
 Get your bearings from our Getting
Started Page:
www.safe.com/fme/getting-started
 Learn from our crew in one of the
weekly FME Overview webinars:
safe.com/WeeklyIntro
What is Big Data?
Big Data and Cloud
Big Data needs big resources
 Big datastores
 Big processing power
 Big bandwidth
Cloud technology gives you this for fraction of
traditional cost!
Big Data and FME
 Big Data is a new data
“classification” for FME.
 Big Data is no different than
other data to FME
 FME Cloud is a natural fit for
data in the Cloud
FME makes it easy to leverage the power of Big Data.
Big Data and FME Support
Amazon S3
 Limitless internet based
storage
Amazon RDS
 See blog article on Amazon RDS (PostGIS)
Amazon DynamoDB
 NoSQL limitless database service
Amazon RedShift
 Petabyte scale database warehouse service.
Google BigQuery
 Superfast append only tables
MarkLogic
 Large XML based database
Poll: How are you currently
working with Big Data?
Big Data Challenges
 Loading Data
 Lacks spatial support
 Big Data Analysis
 Querying and Exporting
Data
Demo #1
 MarkLogic
Demo #2
 Limitless Spatial
Database
Why Demo FME with
MarkLogic and DynamoDB?
Different from other
databases supported by
FME.
What is ?
 NoSQL database – XML optimized
 Powerful search and analysis
 Native Spatial Support
 XML based data model (GML, XML, etc.)
 Deploy on Hadoop HDFS
FME and MarkLogic – A Natural Fit
 Convert data to XML/GML*
 Easily Load XML into MarkLogic with FME
 Process and convert XML results
 FME 2014: New schema based GML Writer
Demo #1a Loading MarkLogic
Convert GIS / CAD
data to GML (XML)
Compose REST request
to PUT to MarkLogic
database
1. Convert GIS / CAD data into Valid GML
2.Generate Key Fields
3. Build insert message
4. Execute PUT REST call
MarkLogic accepts any valid XML – just PUT it!
Loading GIS to MarkLogic
Loading GIS to MarkLogic with FME
What ​Big Data technology
are you most interested in?
Demo #1b Exporting from MarkLogic
GET Query to find
URI’s for features
of interest
GET Query using URI’s to
get feature XML/GML,
then
Conversion to format of
choice (CAD, GIS …)
/WFS
Exporting XML from MarkLogic
1. Query database via GET request
2. Parse search result and compose GET feature request
3. Extract attributes and geometry from result
4. Validate and Write XML Result
Exporting XML from MarkLogic
Search GET request:
http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago
Retrieval GET request:
http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c-
49735dd29356.xml
AIXM from MarkLogic via FMEServer
http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.fmw
?Element=airportCode&Value=CYVR
/AIXM
AIXM from MarkLogic via FMEServer
MarkLogic -> Anything
(JSON, KML, GML …)
MarkLogic to ArcGIS via FME Server:
1. Submit search to MarkLogic as described earlier
2. Extract attributes and geometry from result
3. Generate update ESRIJSON message from feature
4. Post update ESRIJSON to ArcGIS Server
MarkLogic / ArcGIS Integration
ArcGIS Server to MarkLogic
via FME Server
1. Retrive JSON data from ArcGIS Server
2. Generate output GML
3. Write data to MarkLogic via PUT REST call
ArcGIS Server to MarkLogic
Demo #2 – Limitless Spatial Database
DynamoDB
 NoSQL SSD-based database service
 No limit on size of Database
 Specify the needed performance
 Autoscale thru Dynamic DynamoDB
 Amazon EMR (Hadoop) integration
Demo # 2 – Index Strategy
Generate GeoHash Index
for each feature and
Write to
GeoHashSpatialIndex
Demo #2a – Vector, Raster, Lidar
Write small features
to DynamoDB
Write large features
to Amazon S3, link
to DynamoDB
Demo #2b – Geocoded Images
Generate Geohash record
of picture location
Write Image to S3, link
to DynamoDB
Demo #2c – Spatially Store Anything
Generate Geohash
index
Write Document to
S3 and Link to
DynamoDB
location
Demo #2d – Spatially Locate any
internet resource
Write URI Link to
DynamoDB
Generate Geohash
index
location
What data types are you
planning to store in Big Data?
Save the date.
Webinar: How to Automate Practically
Anything with FME Server (March 25th)
Webinar: How to Load Data into Google
Maps Engine (April 16th)
FME World Tour 2014 (April – June 2014)
FME International User Conference 2014
(20th Anniversary Celebration)
• June 10 – 13, 2014 in Vancouver, Canada
Free and fun to learn.
Online Courses - Live & Hands-On
 Feb 18-19: FME Desktop
Tutorials & Recorded Courses
Stay informed.
fmepedia.com/community
fmepedia.com/knowledge
@SafeSoftware
youtube.com/FMEChannel
blog.safe.com
Summary
Big Data = big new opportunities
FME great for working with Big Data
Cloud model is a natural fit for Big Data
This is just the beginning - more to come!
Hand raising has now
been enabled.
 If you’d like to ask a
question over the
air, please click the
hand icon and
ensure your audio
input is set up.
Thank you!
Sales
 info@safe.com
Support
 www.safe.com/support
 (604) 501-9985 ext. 278
Don Murray
 Don.murray@safe.com
Dean Hintz
 dean@safe.com

More Related Content

What's hot

Hadoop 2 cluster architecture
Hadoop 2 cluster architectureHadoop 2 cluster architecture
Hadoop 2 cluster architectureSandeep Patil
 
When OLAP Meets Real-Time, What Happens in eBay?
When OLAP Meets Real-Time, What Happens in eBay?When OLAP Meets Real-Time, What Happens in eBay?
When OLAP Meets Real-Time, What Happens in eBay?DataWorks Summit
 
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...huguk
 
Producing Standardized Data Using a Master FME Workspace
Producing Standardized Data Using a Master FME WorkspaceProducing Standardized Data Using a Master FME Workspace
Producing Standardized Data Using a Master FME WorkspaceSafe Software
 
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko KorndorfSpark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko KorndorfSpark Summit
 
Presto talk @ Global AI conference 2018 Boston
Presto talk @ Global AI conference 2018 BostonPresto talk @ Global AI conference 2018 Boston
Presto talk @ Global AI conference 2018 Bostonkbajda
 
Back to FME School - Day 1: Your Data and FME
Back to FME School - Day 1: Your Data and FMEBack to FME School - Day 1: Your Data and FME
Back to FME School - Day 1: Your Data and FMESafe Software
 
Introduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKIntroduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKKriangkrai Chaonithi
 
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...Amazon Web Services
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
 
Presto@Netflix Presto Meetup 03-19-15
Presto@Netflix Presto Meetup 03-19-15Presto@Netflix Presto Meetup 03-19-15
Presto@Netflix Presto Meetup 03-19-15Zhenxiao Luo
 
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...MongoDB
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData
 
Bring Satellite and Drone Imagery into your Data Science Workflows
Bring Satellite and Drone Imagery into your Data Science WorkflowsBring Satellite and Drone Imagery into your Data Science Workflows
Bring Satellite and Drone Imagery into your Data Science WorkflowsDatabricks
 
Data Warehousing with Spark Streaming at Zalando
Data Warehousing with Spark Streaming at ZalandoData Warehousing with Spark Streaming at Zalando
Data Warehousing with Spark Streaming at ZalandoDatabricks
 
Graph Processing with Apache TinkerPop
Graph Processing with Apache TinkerPopGraph Processing with Apache TinkerPop
Graph Processing with Apache TinkerPopJason Plurad
 
Presto Summit 2018 - 03 - Starburst CBO
Presto Summit 2018  - 03 - Starburst CBOPresto Summit 2018  - 03 - Starburst CBO
Presto Summit 2018 - 03 - Starburst CBOkbajda
 
Key Challenges in Cloud Computing and How Yahoo! is Approaching Them
Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemKey Challenges in Cloud Computing and How Yahoo! is Approaching Them
Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemYahoo Developer Network
 
Powering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationDatabricks
 

What's hot (20)

Hadoop 2 cluster architecture
Hadoop 2 cluster architectureHadoop 2 cluster architecture
Hadoop 2 cluster architecture
 
When OLAP Meets Real-Time, What Happens in eBay?
When OLAP Meets Real-Time, What Happens in eBay?When OLAP Meets Real-Time, What Happens in eBay?
When OLAP Meets Real-Time, What Happens in eBay?
 
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...
 
Producing Standardized Data Using a Master FME Workspace
Producing Standardized Data Using a Master FME WorkspaceProducing Standardized Data Using a Master FME Workspace
Producing Standardized Data Using a Master FME Workspace
 
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko KorndorfSpark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko Korndorf
 
Presto talk @ Global AI conference 2018 Boston
Presto talk @ Global AI conference 2018 BostonPresto talk @ Global AI conference 2018 Boston
Presto talk @ Global AI conference 2018 Boston
 
Back to FME School - Day 1: Your Data and FME
Back to FME School - Day 1: Your Data and FMEBack to FME School - Day 1: Your Data and FME
Back to FME School - Day 1: Your Data and FME
 
Introduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OKIntroduction to Data Engineer and Data Pipeline at Credit OK
Introduction to Data Engineer and Data Pipeline at Credit OK
 
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...
Lessons From Officeworks on Optimising Persistent Storage on AWS (Sponsored b...
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic DatasetsApache Iceberg - A Table Format for Hige Analytic Datasets
Apache Iceberg - A Table Format for Hige Analytic Datasets
 
Presto@Netflix Presto Meetup 03-19-15
Presto@Netflix Presto Meetup 03-19-15Presto@Netflix Presto Meetup 03-19-15
Presto@Netflix Presto Meetup 03-19-15
 
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...
Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Application to the Next Level...
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
 
Bring Satellite and Drone Imagery into your Data Science Workflows
Bring Satellite and Drone Imagery into your Data Science WorkflowsBring Satellite and Drone Imagery into your Data Science Workflows
Bring Satellite and Drone Imagery into your Data Science Workflows
 
Tailored for Spark
Tailored for SparkTailored for Spark
Tailored for Spark
 
Data Warehousing with Spark Streaming at Zalando
Data Warehousing with Spark Streaming at ZalandoData Warehousing with Spark Streaming at Zalando
Data Warehousing with Spark Streaming at Zalando
 
Graph Processing with Apache TinkerPop
Graph Processing with Apache TinkerPopGraph Processing with Apache TinkerPop
Graph Processing with Apache TinkerPop
 
Presto Summit 2018 - 03 - Starburst CBO
Presto Summit 2018  - 03 - Starburst CBOPresto Summit 2018  - 03 - Starburst CBO
Presto Summit 2018 - 03 - Starburst CBO
 
Key Challenges in Cloud Computing and How Yahoo! is Approaching Them
Key Challenges in Cloud Computing and How Yahoo! is Approaching ThemKey Challenges in Cloud Computing and How Yahoo! is Approaching Them
Key Challenges in Cloud Computing and How Yahoo! is Approaching Them
 
Powering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script Transformation
 

Similar to Big Data – Tap into Cloud Infrastructure with FME

Deep Dive into FME Desktop 2014
Deep Dive into FME Desktop 2014Deep Dive into FME Desktop 2014
Deep Dive into FME Desktop 2014Safe Software
 
Taste Test: FME 2011 & Beyond
Taste Test: FME 2011 & BeyondTaste Test: FME 2011 & Beyond
Taste Test: FME 2011 & BeyondSafe Software
 
FME, The Tool to Use When Standing Up a New Fiber Utility
FME, The Tool to Use When Standing Up a New Fiber UtilityFME, The Tool to Use When Standing Up a New Fiber Utility
FME, The Tool to Use When Standing Up a New Fiber UtilitySafe Software
 
FME World Tour 2015 - Around the World - Ken Bragg
FME World Tour 2015 - Around the World - Ken BraggFME World Tour 2015 - Around the World - Ken Bragg
FME World Tour 2015 - Around the World - Ken BraggIMGS
 
Improve Data Exchange in Intergraph Using FME
Improve Data Exchange in Intergraph Using FMEImprove Data Exchange in Intergraph Using FME
Improve Data Exchange in Intergraph Using FMESafe Software
 
Data in the Real World: FME AR and FME Data Express
Data in the Real World: FME AR and FME Data ExpressData in the Real World: FME AR and FME Data Express
Data in the Real World: FME AR and FME Data ExpressSafe Software
 
FME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationFME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationSafe Software
 
How to Exchange Data between CAD and GIS
How to Exchange Data between CAD and GISHow to Exchange Data between CAD and GIS
How to Exchange Data between CAD and GISSafe Software
 
How to Efficiently Transform Non-Spatial Data using FME
How to Efficiently Transform Non-Spatial Data using FMEHow to Efficiently Transform Non-Spatial Data using FME
How to Efficiently Transform Non-Spatial Data using FMESafe Software
 
How to Transform Data between AutoCAD and GIS
How to Transform Data between AutoCAD and GISHow to Transform Data between AutoCAD and GIS
How to Transform Data between AutoCAD and GISSafe Software
 
Still on IBM BigInsights? We have the right path for you
Still on IBM BigInsights? We have the right path for youStill on IBM BigInsights? We have the right path for you
Still on IBM BigInsights? We have the right path for youModusOptimum
 
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FMEEfficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FMESafe Software
 
Deep Dive into FME Desktop 2018
Deep Dive into FME Desktop 2018Deep Dive into FME Desktop 2018
Deep Dive into FME Desktop 2018Safe Software
 
Automating Enterprise Workflows with FME Server
 Automating Enterprise Workflows with FME Server Automating Enterprise Workflows with FME Server
Automating Enterprise Workflows with FME ServerSafe Software
 
Migration DB2 to EDB - Project Experience
 Migration DB2 to EDB - Project Experience Migration DB2 to EDB - Project Experience
Migration DB2 to EDB - Project ExperienceEDB
 
Data Integration Solutions for Airports
Data Integration Solutions for AirportsData Integration Solutions for Airports
Data Integration Solutions for AirportsSafe Software
 
FME Around the World
FME Around the WorldFME Around the World
FME Around the WorldSafe Software
 
FME User Stories from Around the World
FME User Stories from Around the WorldFME User Stories from Around the World
FME User Stories from Around the WorldSafe Software
 

Similar to Big Data – Tap into Cloud Infrastructure with FME (20)

Deep Dive into FME Desktop 2014
Deep Dive into FME Desktop 2014Deep Dive into FME Desktop 2014
Deep Dive into FME Desktop 2014
 
Taste Test: FME 2011 & Beyond
Taste Test: FME 2011 & BeyondTaste Test: FME 2011 & Beyond
Taste Test: FME 2011 & Beyond
 
FME, The Tool to Use When Standing Up a New Fiber Utility
FME, The Tool to Use When Standing Up a New Fiber UtilityFME, The Tool to Use When Standing Up a New Fiber Utility
FME, The Tool to Use When Standing Up a New Fiber Utility
 
FME World Tour 2015 - Around the World - Ken Bragg
FME World Tour 2015 - Around the World - Ken BraggFME World Tour 2015 - Around the World - Ken Bragg
FME World Tour 2015 - Around the World - Ken Bragg
 
Improve Data Exchange in Intergraph Using FME
Improve Data Exchange in Intergraph Using FMEImprove Data Exchange in Intergraph Using FME
Improve Data Exchange in Intergraph Using FME
 
Data in the Real World: FME AR and FME Data Express
Data in the Real World: FME AR and FME Data ExpressData in the Real World: FME AR and FME Data Express
Data in the Real World: FME AR and FME Data Express
 
FME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling InnovationFME 2022.0: Driving Data Decisions, Fueling Innovation
FME 2022.0: Driving Data Decisions, Fueling Innovation
 
Ibm db2 big sql
Ibm db2 big sqlIbm db2 big sql
Ibm db2 big sql
 
How to Exchange Data between CAD and GIS
How to Exchange Data between CAD and GISHow to Exchange Data between CAD and GIS
How to Exchange Data between CAD and GIS
 
How to Efficiently Transform Non-Spatial Data using FME
How to Efficiently Transform Non-Spatial Data using FMEHow to Efficiently Transform Non-Spatial Data using FME
How to Efficiently Transform Non-Spatial Data using FME
 
How to Transform Data between AutoCAD and GIS
How to Transform Data between AutoCAD and GISHow to Transform Data between AutoCAD and GIS
How to Transform Data between AutoCAD and GIS
 
Still on IBM BigInsights? We have the right path for you
Still on IBM BigInsights? We have the right path for youStill on IBM BigInsights? We have the right path for you
Still on IBM BigInsights? We have the right path for you
 
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FMEEfficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
Efficiently Implementing INSPIRE & Creating INSPIRE Mashups with FME
 
Deep Dive into FME Desktop 2018
Deep Dive into FME Desktop 2018Deep Dive into FME Desktop 2018
Deep Dive into FME Desktop 2018
 
Automating Enterprise Workflows with FME Server
 Automating Enterprise Workflows with FME Server Automating Enterprise Workflows with FME Server
Automating Enterprise Workflows with FME Server
 
Migration DB2 to EDB - Project Experience
 Migration DB2 to EDB - Project Experience Migration DB2 to EDB - Project Experience
Migration DB2 to EDB - Project Experience
 
Data Integration Solutions for Airports
Data Integration Solutions for AirportsData Integration Solutions for Airports
Data Integration Solutions for Airports
 
Unveiling FME 2018
Unveiling FME 2018Unveiling FME 2018
Unveiling FME 2018
 
FME Around the World
FME Around the WorldFME Around the World
FME Around the World
 
FME User Stories from Around the World
FME User Stories from Around the WorldFME User Stories from Around the World
FME User Stories from Around the World
 

More from Safe Software

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMESafe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Safe Software
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleSafe Software
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FMESafe Software
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Safe Software
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentSafe Software
 

More from Safe Software (20)

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Mastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FMEMastering DevOps-Driven Data Integration with FME
Mastering DevOps-Driven Data Integration with FME
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) People
 
Democratizing Transformer Creation in FME
Democratizing Transformer Creation in FMEDemocratizing Transformer Creation in FME
Democratizing Transformer Creation in FME
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernment
 

Recently uploaded

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 

Recently uploaded (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 

Big Data – Tap into Cloud Infrastructure with FME

  • 1. Big Data Tap into Cloud Infrastructure with FME March 18, 2014
  • 2. Meet the presenters. Don Murray  President and Co-Founder @DonAtSafe Dean Hintz  Senior Product Specialist @DeanAtSafe
  • 3. Ask us. And join the discussion. Please submit using the GoToWebinar panel. We will follow-up with unanswered questions.
  • 4. Agenda.  What is Big Data  Big Data Challenges  FME and Big Data  FME Demos:  Loading and Extracting from MarkLogic  Spatial Indexing and Loading to DynamoDB
  • 7. Poll: Which version of FME are you using?
  • 8. New to FME?  Get your bearings from our Getting Started Page: www.safe.com/fme/getting-started  Learn from our crew in one of the weekly FME Overview webinars: safe.com/WeeklyIntro
  • 9. What is Big Data?
  • 10. Big Data and Cloud Big Data needs big resources  Big datastores  Big processing power  Big bandwidth Cloud technology gives you this for fraction of traditional cost!
  • 11. Big Data and FME  Big Data is a new data “classification” for FME.  Big Data is no different than other data to FME  FME Cloud is a natural fit for data in the Cloud FME makes it easy to leverage the power of Big Data.
  • 12. Big Data and FME Support Amazon S3  Limitless internet based storage Amazon RDS  See blog article on Amazon RDS (PostGIS) Amazon DynamoDB  NoSQL limitless database service Amazon RedShift  Petabyte scale database warehouse service. Google BigQuery  Superfast append only tables MarkLogic  Large XML based database
  • 13. Poll: How are you currently working with Big Data?
  • 14. Big Data Challenges  Loading Data  Lacks spatial support  Big Data Analysis  Querying and Exporting Data
  • 15. Demo #1  MarkLogic Demo #2  Limitless Spatial Database
  • 16. Why Demo FME with MarkLogic and DynamoDB? Different from other databases supported by FME.
  • 17. What is ?  NoSQL database – XML optimized  Powerful search and analysis  Native Spatial Support  XML based data model (GML, XML, etc.)  Deploy on Hadoop HDFS
  • 18. FME and MarkLogic – A Natural Fit  Convert data to XML/GML*  Easily Load XML into MarkLogic with FME  Process and convert XML results  FME 2014: New schema based GML Writer
  • 19. Demo #1a Loading MarkLogic Convert GIS / CAD data to GML (XML) Compose REST request to PUT to MarkLogic database
  • 20. 1. Convert GIS / CAD data into Valid GML 2.Generate Key Fields 3. Build insert message 4. Execute PUT REST call MarkLogic accepts any valid XML – just PUT it! Loading GIS to MarkLogic
  • 21. Loading GIS to MarkLogic with FME
  • 22. What ​Big Data technology are you most interested in?
  • 23. Demo #1b Exporting from MarkLogic GET Query to find URI’s for features of interest GET Query using URI’s to get feature XML/GML, then Conversion to format of choice (CAD, GIS …) /WFS
  • 24. Exporting XML from MarkLogic 1. Query database via GET request 2. Parse search result and compose GET feature request 3. Extract attributes and geometry from result 4. Validate and Write XML Result
  • 25. Exporting XML from MarkLogic Search GET request: http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago Retrieval GET request: http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c- 49735dd29356.xml
  • 26. AIXM from MarkLogic via FMEServer http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.fmw ?Element=airportCode&Value=CYVR /AIXM
  • 27. AIXM from MarkLogic via FMEServer
  • 29. MarkLogic to ArcGIS via FME Server: 1. Submit search to MarkLogic as described earlier 2. Extract attributes and geometry from result 3. Generate update ESRIJSON message from feature 4. Post update ESRIJSON to ArcGIS Server MarkLogic / ArcGIS Integration
  • 30. ArcGIS Server to MarkLogic via FME Server 1. Retrive JSON data from ArcGIS Server 2. Generate output GML 3. Write data to MarkLogic via PUT REST call
  • 31. ArcGIS Server to MarkLogic
  • 32. Demo #2 – Limitless Spatial Database
  • 33. DynamoDB  NoSQL SSD-based database service  No limit on size of Database  Specify the needed performance  Autoscale thru Dynamic DynamoDB  Amazon EMR (Hadoop) integration
  • 34. Demo # 2 – Index Strategy Generate GeoHash Index for each feature and Write to GeoHashSpatialIndex
  • 35. Demo #2a – Vector, Raster, Lidar Write small features to DynamoDB Write large features to Amazon S3, link to DynamoDB
  • 36. Demo #2b – Geocoded Images Generate Geohash record of picture location Write Image to S3, link to DynamoDB
  • 37. Demo #2c – Spatially Store Anything Generate Geohash index Write Document to S3 and Link to DynamoDB location
  • 38. Demo #2d – Spatially Locate any internet resource Write URI Link to DynamoDB Generate Geohash index location
  • 39. What data types are you planning to store in Big Data?
  • 40. Save the date. Webinar: How to Automate Practically Anything with FME Server (March 25th) Webinar: How to Load Data into Google Maps Engine (April 16th) FME World Tour 2014 (April – June 2014) FME International User Conference 2014 (20th Anniversary Celebration) • June 10 – 13, 2014 in Vancouver, Canada
  • 41. Free and fun to learn. Online Courses - Live & Hands-On  Feb 18-19: FME Desktop Tutorials & Recorded Courses
  • 43. Summary Big Data = big new opportunities FME great for working with Big Data Cloud model is a natural fit for Big Data This is just the beginning - more to come!
  • 44. Hand raising has now been enabled.  If you’d like to ask a question over the air, please click the hand icon and ensure your audio input is set up.
  • 45. Thank you! Sales  info@safe.com Support  www.safe.com/support  (604) 501-9985 ext. 278 Don Murray  Don.murray@safe.com Dean Hintz  dean@safe.com

Editor's Notes

  1. Video plays here - what is big dataFuzzy term sort of like “cloud”. What does big data look like?As a catch-all term, “big data” can be pretty nebulous, in the same way that the term “cloud” covers diverse technologies. Input data to big data systems could be chatter from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data, the list goes on. Are these all really the same thing? To clarify matters, the three Vs of volume, velocity and variety are commonly used to characterize different aspects of big data. They’re a helpful lens through which to view and understand the nature of the data and the software platforms available to exploit them. Most probably you will contend with each of the Vs to one degree or another.
  2. Big data holds all of it
  3. - on premise - cloud (amazon web services) - cloud (google) - cloud (other) - not currently using Big Data
  4. Loading DataConversion: big data not spatial friendly (CAD, GIS)Expensive to upload / downloadGeoreferencing and spatial indexingmost big data repositories have limited geospatialBig Data AnalysisQuerying and Exporting DataTricky to find and access stored dataNeed to generate appropriate keys on load
  5. Loading DataConversion: big data not spatial friendly (CAD, GIS)Expensive to upload / downloadGeoreferencing and spatial indexingmost big data repositories have limited geospatialBig Data AnalysisQuerying and Exporting DataTricky to find and access stored dataNeed to generate appropriate keys on load
  6. Loading DataConversion: big data not spatial friendly (CAD, GIS)Expensive to upload / downloadGeoreferencing and spatial indexingmost big data repositories have limited geospatialBig Data AnalysisQuerying and Exporting DataTricky to find and access stored dataNeed to generate appropriate keys on load
  7. Big data repository – scale as big as you wantNoSQL database – optimized for XML / GMLPowerful search and analysis (BI, semantic queries)Stores location, not just geohashXML based data model – rapid XML exportStore any documents: GML, XML (metadata)Deploy on Hadoop HDFS
  8. * As applicable (e.g. cant convert raster to gml!)FME2014’s new schema based GML writer which allows FME to convert almost any CAD / GIS or even BIM data to GML or CityGML. This makes FME a very powerful loader tool for MarkLogicFME - A Natural Fit to support MarkLogic:Converts almost any spatial data to GMLWrite almost any XML with XMLTemplaterLoading XML into MarkLogic is a simple HTTP PUT operation easily done with HTTPUploaderQuery, process and reconvert XML results
  9. Converting features to GML/XML usually involves a GeometryExtractor transformer or some combination of CoordinateExtractor and XMLTemplaterKey fields can be captured from the source data or use UUIDGenerator to generate unique IDs for URIs etc.Build insert message with XMLTemplaterExecute REST PUT call with HTTPUploader
  10. Converting features to GML/XML usually involves a GeometryExtractor transformer or some combination of CoordinateExtractor and XMLTemplaterKey fields can be captured from the source data or use UUIDGenerator to generate unique IDs for URIs etc.Build insert message with XMLTemplaterExecute REST PUT call with HTTPUploader<?xml version="1.0" encoding="UTF-8"?><xml><docID>{fme:get-attribute("_uuid")}</docID><docAuthor>{fme:get-attribute("user")}</docAuthor><modType>{fme:get-attribute("updateType")}</modType><UpdateDate>{fme:get-attribute("_timestamp")}</UpdateDate><filePath>{fme:get-attribute("filePath")}</filePath><comment>{fme:get-attribute("comment")}</comment><doc_xml>{fme:get-xml-attribute("_file_contents")}</doc_xml></xml>
  11. As simple as 1,2,3,4!
  12. - on premise - cloud (amazon web services) - cloud (google) - cloud (other) - not currently using Big Data
  13. * need bubble here for XML/WFS – maybe a circle with something like this in it:<gml:featureMember> <gn:NamedPlacegml:id=“abc.123"> <gn:geometry> <gml:Pointgml:id=“p.abc.123" srsName="EPSG:4258"><gml:pos>15.2 36.7</gml:pos> </gml:Point> </gn:geometry>…
  14. This workspace can support the retrieval of any type of XML/GML regardless of schema. The same query workspace can be used to retrieve AIXM, INSPIRE or any other type of XML/GML.StringConcatenator composes search GET request based on input parametersHTTPFetcher sends search GET request to MarkLogicXMLFlattener flattens the response so result.uri can be exposedSecond StringConcatenatorcomposes document GET request based on matching URISecond HTTPFetcher sends document retrieval GET request to MarkLogicXMLFragmenter pulls out the doc_xml from the MarkLogic responseXML writer outputs the XML as a file or streams it to the FMEServer client once workspace is publishedSearch GET request to find URI based on query:http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.ChicagoDocument Retrieval GET request based on URI:http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c-49735dd29356.xml
  15. For this demo the previous workspace was published to FME Server to make a feature service hosted by FMEServer on top of MarkLogic. The example here supports a simple REST based XML data stream.We could easily use this approach to build a FMEServer hosted WFS on top of MarkLogic.
  16. This demo shows Inspector reading AIXM5 GML directly from the GET query: http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.fmw?Element=airportCode&Value=CYVRThe query goes to FMEServer’s data streaming serviceFMEServer uses the URL parameters to run the published QueryMarkLogicDB.fmw workspace.QueryMarkLogicDB.fmw uses the values of Element and Value to build a search request and send that to MarkLogicQueryMarkLogicDB.fmw uses the URI from MarkLogic’s search result to compose and submit a document request to MarkLogicQueryMarkLogicDB.fmw extracts the feature XML from the MarkLogic’s document response and streams it back to the FMEServer client
  17. This just shows how FME can read XML from MarkLogic and use the GeometryReplacer to covert it to virtually any format FME supports
  18. Shows how FME can be used to integrateMarkLogic and ArcGIS Server.These are the steps to move data from MarkLogic to Arc Server Feature Service
  19. Shows how FME can be used to integrateMarkLogic and ArcGIS Server.These are the steps to move data from Arc Server Feature Service to MarkLogic. Note this workflow could be event driven, real time or as a scheduled update.
  20. Workspace showing data flow from ArcServer toMarkLogic. REST call to feature service retrieves the feature of interest.JSON is extracted and GeometryReplacer generates an FME geometry from it.GeometryExtractor renders the FME geometry as GMLGML is added to an XML update message and posted to MarkLogic
  21. Demo #2 Limitless Spatial Indexed Database:Geohash spatial indexStore Vector DataStore Raster DataStore Lidar DataStore geotagged images by locationStore and associate any document with a location
  22. - on premise - cloud (amazon web services) - cloud (google) - cloud (other) - not currently using Big Data