Facing challenges with data management in the cloud environment? Tackling issues like inefficient data management, integration difficulties, scalability constraints, and security concerns? You're not alone. In this Part 2 of our database webinar series, we'll offer you the tools to overcome these challenges, streamline workflows, boost performance, and implement industry-leading cloud data management practices -- fueling your organization to extract maximum value from your data.
In this hour-long interactive session, our experts will walk you through:
The different formats and benefits of using cloud-native databases with FME
Strategies for deploying FME in cloud environments to ensure seamless integration and scalability
Best practices for managing and scaling data effectively in a cloud environment
Exclusive tips on data processing and transformation specifically for cloud-native databases
To cap it off, our experts will host a live Q&A session to answer any questions you may have. Don't let this opportunity slip by to optimize your cloud-native data workflows. Register now and secure your place at the forefront of data management innovation. Your data's untapped potential awaits.
3. Agenda
1 Introduction
2 FME and Cloud Native Databases
3 Optimizing Workflow Efficiency
4 Deploying FME in Cloud Environments
5 Best Practices for Managing Big Data with FME
6 Real-world example migrating 150,000,000 records
7 Q&A Session
Agenda
4. Welcome to Livestorm.
A few ways to engage with us during the webinar:
Audio issues? Click this for 4 simple
troubleshooting steps.
5. How to download slides
1. Hover over the
slide deck in the
webinar room
2. Click this button
11. Differences between on-prem vs Cloud databases
Common frustrations people face with this topic are:
● Data Integration complexity
● Security
● Performance
● Lack of Best Practices
12. Poll:
What are the biggest
challenges you face with
working with cloud databases?
14. 29+
27K+
128
190
20K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
29+
29K+
128
140+
25K+
years of solving data
challenges
FME Community
members
countries with
FME customers
organizations worldwide
global partners with
FME services
200K+
users worldwide
Safe & FME
15. One platform, two technologies
FME Form FME Flow
Build and run data workflows Automate data workflows
FME Flow Hosted
Safe Software managed instance
fme.safe.com/platform
FME Enterprise Integration Platform
16. Number
of
supported
data
types
in
FME
1995 2000 2005 2010 2015 2020 2023…
10
100
300
500
GIS
CAD
Database
XML
Raster
3D
BIM
Web
Point
Cloud
Cloud
Big
Data
IOT
Gaming
BI
Indoor
Mapping
AR/VR
Generative
AI
Cloud
Native
Tabular
Unrivalled Data Support
18. Slide Title
Reading from
MongoDB
Goal Block Key
MongoDB Database Reading/Writing
Result
Performance
blocks
Data Medium Improved
workspace times
that avoid the
database
22. Techniques for maximizing
FME's performance with big
data
● Let the database do the work
● Proximity to data
● Be creative
● Let the database do the work
1 2 3
4 5 6
7 8 9
SQL
23. Slide Title
Reading from
Snowflake
Goal Block Key
Cloud Database Reading
Result
Performance Leverage the
database to
improve reading
performance
Faster you can
pull data down,
faster you can
work with it
25. Results
Source
Location
FME Location Bucket
Location
Method Total Time
US West US East N/A Snowflake Reader 49m 39s
US West US East US West SQL to S3 Bucket + CSV
Reader (300 files)
58m
US West US East US West SQL to S3 Bucket + CSV
Reader (8 files)
41m
Scenario
● Snowflake Reading
● 150 Million Records
26. Strategies to improve
workflow efficiency and
reduce processing time
● Can you avoid writing directly to
database?
● Does the database support other
import/export methods
27. Slide Title
Writing to
Snowflake
Goal Block Key
Cloud Database Writing
Result
Performance Leverage the
database to
improve Writing
performance
Allow Database
to process the
data most
efficiently to
reduce
downtime
29. Results
Source
Location
FME Location Bucket
Location
Method Total Time
US West US East N/A Snowflake Writer 4h 40m
US West US East US West CSV Writer + S3
Connector + SQL(8 files)
11m 20s
US West US East US West CSV Writer(GZ) + S3
Connector + SQL (300
files batched)
1m 22s(+8m on
database side for
processing)
Scenario
● Snowflake Writing
● 150 Million Records
30. Addressing security concerns when working with
FME and big data
● TLS/SSL Considerations
● Sensitive Data
● Where’s your Data lingering
33. Integration considerations for
different cloud platforms
● Cloud Storage like S3 Bucket, Azure
Storage, etc
● Performance of CPU/RAM/IO bandwidth
● Performance of Disks (SSD)
34. Ensuring seamless connectivity and scalability
● Scaling FME for more throughput
● Consider network topology
● Virus scanners, file scanners,
● How close are you to the data
35. Slide Title
Maximize
Performance
Of Workflow
Goal Block Key
Maximizing Data Throughput Using Automations
Result
Don’t
Understand all
the Systems.
What can we
change in the
Workflow?
Massive
Performance
Gains
37. FME Flow Results
Test
Source
Location
FME
Location
Bucket
Location
Target
Location
System Count | Engine Count | CPU(RAM) Total
Time
1 US East US West US West US West 1 System | 15 Engines | 8 (32G) CPU 2h 48m
View with Group ID
● 50 Million Records
● Oracle Cloud to S3 Bucket to Snowflake
2 US East US West US West US West 2 System | 20 Engines | 8 (32G) CPU/SYS 2h 27m
View with Group ID (add more engines)
3 US East US West US West US West 1 System | 10 Engines | 8 (32G) CPU 2h 30m
Materialized View with Group ID indexed
4 US East US East US West US West 1 System | 14 Engines | 16 (64G) CPU 0h 21m
Materialized View with Group ID indexed and FME Close to DB
5 US West US West US West US West 2 System | 20 Engines | 8 (32G) CPU/SYS 1h 28m
Materialized View with Group ID indexed and FME Close to DB RDS
39. Key recommendations for effectively managing
and scaling big data workflows
● Data medium
● Breaking up the workflow
● Leverage Flow/Server and CPU engines
● Consider Network topology/architecture
40. Data processing and transformation tips
● Data delivery method
● Simplify the data
● Reduce on-the-fly transformation
42. Slide Title
Oracle to
Snowflake
Goal Block Key
Oracle Cloud to Snowflake
Result
Oracle Cloud to
Snowflake, 2
Cloud native
databases
Think outside the
box with FME
Drastically
improved
performance
47. Resources
● Let the Database Do the
Work tutorial
● Let Your Database Do the
Work Webinar
48. Get our Ebook
Spatial Data for the
Enterprise
fme.ly/gzc
Guided learning experiences
at your fingertips
community.safe.com
/s/academy
FME Academy
49. Check out how-to’s & demos
in the knowledge base
community.safe.com
/s/knowledge-base
Knowledge Base Webinars
Upcoming & on-demand
webinars
safe.com/webinars
51. We’d love to help you get
started.
Get in touch with us at
info@safe.com
Experience the FME Accelerator
Contact Us
Unlock the power of your
data in only 90 minutes
Register for free at
fme.safe.com/accelerator