This document discusses how FME (Feature Manipulation Engine) software can be used to support Open Data initiatives and INSPIRE requirements by transforming data into common schemas. It describes the key steps in the harmonization process including evaluation, assembly, transformation, validation, and publication of data. Specific FME capabilities that help with schema mapping, metadata, validation, and publishing data as web services are also outlined. The document concludes with an example of how FME has been used to demonstrate transforming data to Open Data standards.
Using FME to support open data initiatives and INSPIRE
1. Using FME to support Open
Data initiatives and INSPIRE
Ciaran Kirk Tony Kent
Technical Director GIS Technician
6th September 2012
2.
3. The Challenge
You want to meet Open data requirements, but
your data is organized rather differently
?
4. What we do …
?
We build data bridges – to your SDI
5. Harmonization Core Concepts
Harmonization: implied requirement for building an
SDI
Disparate sources must be mapped to a common
destination data model
Core to the harmonization workflow is a process called
schema mapping.
Delivered by services based on open standards
6. INSPIRE Harmonization Principles
Typical stages:
1. Evaluation
2. Assembly
3. Transformation
4. Validation
5. Publication
Based on the Spatial ETL concept (Extract,
Transform and Load), as applied to INSPIRE SDI’s
7. Evaluation
Assess destination schema and data requirements
Assess source datasets and schema
Consider fundamental differences in representation,
resolution
Closely inspect actual representative datasets
8. Data Assembly
Assess the diversity of source data types:
vector, raster, CAD, GIS, database, text, XML, web,
3D, sensor and non-spatial
Review format and semantic translation needs
Decide how to perform necessary joins
ID joins, spatial relates, nearest neighbor, one to
many relationships
Goal is to build a data structure that corresponds
with your published standard
9. Metadata – Data about data
Describes data structures
• tables
• geometry types
• data types
• fields
Describes data content
• coordinate system
• extent
• modification date
• quality, ownership, etc.
14. Data Transformation - Schema
Reshape source data to match required
destination schema
Schema mapping
feature type
attribute name
new attribute creation
code lists
conditional value
mappings
15. Schema Mapping in FME
Feature Type Mapping in FME Workbench
Attribute Mapping in FME Workbench
16. FME Data Model Restructuring:
Attribute Names & Values
Value Mapping
18. Transformation: Geometry
Non-spatial to spatial
Geometry extraction (spatial to GML)
Representation transform: CAD drawing lines with
labels to GIS polygonal features with attributes
Coordinate System Reprojection (ED50 to ETRF89)
Simple to complex geometry
Source point and polygon data to multiple geometric
representations (city as point / area, river as line / area)
Generalization and interpolation
Highly granular national and regional datasets often require
thinning to be usable on pan-European scales
19. Validation
Schema validation i.e. INSPIRE (xsds)
Data integrity
Unique IDs
Geometric integrity (closed polygons)
Null values (nullable?)
Valid values: ranges and domain codes
Data gaps
Bounds
Network integrity
Custom validity rules specific to domain
Validation automation via FME Server upload
Ensure data quality throughout the data transformation process.
20. Publication
Produce INSPIRE compliant GML
Provide discovery, view or download services, for
WxS, GML and other desired formats
Publish with FME Server or integrate with your
geo web server of choice
Spatial Data Services
21. Publication with FME Server
Publish workspace to FME Server
Store the workspace in a central repository
Make your FME workspaces available to others –
over the web
Register the workspace with one or more services
(Data Streaming, Data Download, etc.)
21
23. FME Tools for INSPIRE
Format translation
Schema mapping
String and list manipulation
Data validation
Database load and extract
XML,GML,WFS: reading, validation,
publication
Web services: WFS, WMS, integration
with others
Metadata support
Enterprise services
with FME Server
24. Summary
FME can provide all the tools to help you support
your data sharing needs:
Integrate your data sources
Manage your meta data catalogues
Transform your data to standard schemas
Publish the data in the required formats
25. Thank You!
Questions?
For more information:
Ciaran Kirk – ckirk@imgs.ie
Tony Kent – tkent@imgs.ie
support@imgs.ie
Supports searchesAllows you to review content without having to read all the data firstDiscover what’s availableSupport data management and updatesprovides parameters for automating processingControl accessTrack ownership
Read metadata attributesXML reading, writing, updatingSchema readingReading from web sourcesIntegration with web servicesHarvest and validation capabilitiesCreation of your own web services