NextGEOSS Webinar - Cloud APIs

Cloud APIs
NextGEOSS Webinar, November 28th 2019
Hervé Caumont (Terradue)
Bjorn Backeberg (EGI.eu)

● Introduction on the use of Cloud APIs within NextGEOSS
○ Hervé Caumont (Terradue)
● EGI.eu Federated Cloud APIs for a uniﬁed UX in a distributed cloud
infrastructure
○ Bjorn Backeberg (EGI Foundation)
● NextGEOSS Cloud Bursting Service - Terradue support to application builders
for multi-cloud deployment of Pilot Applications
○ Hervé Caumont (Terradue)
AGENDA
Platform Integration

1
Hervé Caumont, Terradue
Introduction on the use of Cloud APIs
within NextGEOSS

User
Management
AAA
NextGEOSS
DataHub
Usage
Analytics
User
Management
SSO
Usage
Analytics
Catalog
Discovery
User
Management
AAA
Geospatial
User
Feedback
User
Management
SSO

Jupyter
Notebooks
NextGEOSS
DataHub
Usage
Analytics
Tasks
?
Your Service
!
Usage
Analytics
Compute
Geospatial
tagging
User
Management
AUTHZ
Geospatial
User
Feedback
User
Management
SSO

Cloud Integration API
● Offer
○ Make your data processing algorithms
scalable & portable across Cloud
Providers, and directly accessible in a
standard way
○ Get your web applications more robust
in exploiting data sources
○ Build applications based on
interoperable, distributed systems
○ Benefit from standard integration
layers and focus on your core
business
Infrastructure-as-a-Service for
application design and
integration
Standard APIs to stage
data, run jobs & and
package Apps
eScience tools to build
laboratory notebooks
and scalable workflows
Cloud Sandboxes
Application Frameworks

User Management API
● Offer
○ Make users of your application
authenticated via Single-Sign-On (SSO)
○ Make your API calls mediated via
Authentication, Authorization and
Accounting (AAA)
○ All of it in a federated environment
(including the NextGEOSS Platform Services!)
○ Get intelligent access control to:
■ monitor your “free access” resources usage,
■ enforce other data or service policies,
■ serve your auditing needs
○ Your application is uplifted to the
level of effective management &
security
We establish digital trust with
the identity providers (IdP) of
your user community
Your user community can join
your application without
changing their sign-in habits
You define the
access levels for
people working with
your application
authentication
authorisation

Data Discovery API
NextGEOSS DataHub is powered by
A unique access point to the
wide range of European Earth
observation data
Applications can remotely access
Standard APIs, to systematically
feed their data processing needs
Data Hub users can explore Earth
observation data, share comments,
suggestions and rate the contents
● Offer
○ Make use of the NextGEOSS Data Hub
standard API to fetch and consume
data resources within your applications
○ Get high flexibility and robustness for
your data sourcing process.
○ Tap into distributed data repositories
from one place, without the burden of
maintaining multiple data access
endpoints and protocols.

Geospatial User Feedback API
● Offer
○ Add user feedback capabilities to
resources in your community portal,
using recognized OGC standards
○ Get user experiences using your
resources
○ Help build a knowledge base and a
collaborative platform for your
community portal
○ Get your community portal more
interactive
○ Supplement the information you
already provide about your data
○ Get better community engagement and
an increase trust in your services
By simply adding a link or button,
retrieve previous feedback items
Widget integration
label
Add new elements or edit previous onesWidget functions
Widget
customization
Use CSS to easily change styles and
to hide/show desired information

Cloud Bursting API
● Offer
○ Make your Cloud-ready application available
as a “Cloud appliance” on the Platform
repository
■ Decide on your Cloud appliances access
conditions: only to your organisation, shared
with partner organisations, ...
○ Leverage the “deployers” previously setup for
you, according to your existing Cloud
provisioning arrangements
■ Your credited accounts for a Cloud Provider, …
○ Adapt your Cloud deployment strategies
as new opportunities come to you
Ensure Cloud provisioning
is performed according to
the credit lines in place
Get access to the deployer
service matching your Cloud
provider resources
Define configuration
and “deploy” !
Check your
application service
endpoints in
production, and run
operations
Cloud Controller APIs
Deployer APIs

User
Management
AAA
NextGEOSS
DataHub
Usage
Analytics
User
Management
SSO
Usage
Analytics
Catalog
Discovery
User
Management
AAA
Geospatial
User
Feedback
User
Management
SSO
Our focus
for today’s
Webinar

2
Bjorn Backeberg, EGI.eu
EGI.eu Federated Cloud APIs for a unified UX in a
distributed cloud infrastructure

What is the EGI FedCloud?
● Offer
○ Multi-cloud Infrastructure as a Service (IaaS)
○ Harmonized access to participating cloud sites
○ Hybrid Cloud including public and commercial
infrastructures
○ Technology agnostic, support OpenStack, OpenNebula
and Synnefo
● Key features
○ Single Sign-On
○ Virtual Appliance catalogue and unified GUI dashboard
○ Resource discovery
○ Customer Relationship Management
■ Centralised accounting
■ Service Level Agreement monitoring
Cloud Compute
Run Virtual Machines on demand similar to AWS EC2/EBS or GCP Compute Engine
EGI Services powered by the FedCloud
Cloud Container
Compute
Online Storage
Training
infrastructure
Applications on
demand
Notebooks

● The EGI CLoud Infrastructure
Platform (CLIP):
○ an abstract Cloud Management
stack subsystem
○ integrated with components of
the EGI Core Information
Management System
● CLIP: a thin layer of federation and
interoperability services around
local deployments and integrations of
Cloud Management stacks.
● Cloud Management stack deployments must provide at least one of these IaaS interaction ports
preferably using standardised APIs, specifically OCCI for VM management, and CDMI for
object storage.
The (original) Federation Model

Challenges with OCCI
● It turned out that it was difficult to impose OCCI API as the single IaaS interaction port
for providers and users
● OCCI lacked support in user tools and had a limited feature list
⇨ Necessitated manual management processes of the IaaS
⇨ Limited possibility for adopting cloud-native approaches where resources are managed
automatically via API calls
➠ EGI decision to allow providers to decide for themselves which API standard to
implement
● EGI focus shift towards
○ building tools to enhance the user experience → including common interfaces towards
a unified system
○ automate resource provisioning

The current access layers
IaaS Federated Access
Tools layer helps users of
the cloud to deal with the
heterogeneity in the IaaS
API and EGI Federation
services
Providers have their APIs
that can be used with EGI
Check-in accounts, opening
the door to automation of
cloud-native applications.
EGI Federated Cloud no
longer mandates a single
API for every provider.
OCCI still widely supported
but sites are moving to
native APIs (mainly
OpenStack!)

EGI Applications Database
Registry for virtual appliances (VA)
✓ a logical container of versioned image
file & metadata bundles
VA distribution medium
✓ distributing endorsed VAs to the
resource providers/sites
Resource providers and VO catalogue
✓ list of the VAs which are available by
each site/resource provider for every
VO
https://appdb.egi.eu/
The base URI of the RESTful API is: http://appdb.egi.eu/rest/1.0

EGI VMOps Dashboard
AppDB VMOps - User Friendly GUI
● Single Web dashboard to manage
VMs in the federation
○ Point-and-click solution to create
new VMs
● Powered by Infrastructure Manager
● Integrated with:
○ Single Sign-On, discovery, VM
catalogue, monitoring
APIs used for the Dashboard:
● Check-in OpenID Connect for federated AAI
● AppDB information system API (GraphQL), discover VA, providers, VOs
● ARGO monitoring information, to query the status of the providers
● OpenStack/OCCI/OpenNebula APIs, via IM (used by VMOps) / Terraform
● Create VMs using information from APIs (send jobs to cloud)

The EGI fedcloud today
EGI FedCloud consists of 23
distributed IaaS providers
Activities coordinated by the
EGI Cloud Task Force
Users from a broad
range of scientific
disciplines supported
>3,000 computing hours
>600,000 VMs deployed

● Expand the capabilities and capacity of the federated cloud:
○ GPGPU, High-memory and compute intensive VMs
○ Improve discovery
● Move towards API-based management of resources
○ EGI involved in SLA setup
○ Users deploy their own resources with the tools of their choice: VMOps, IaaS
Orchestration, direct API access, their own tooling
● Simplify providers integration
○ Centralise the operation of the federation features (e.g. connection to AppDB)
○ Liaise with commercial providers
Future plans

3
Hervé Caumont, Terradue
NextGEOSS Cloud Bursting Service
Terradue support to application builders for Cloud bursting
(multi-cloud deployment of Pilot Applications)

Cloud Bursting: some use cases
● Multiple deployment of a same
application
○ For different target user audiences
○ and/or for different production campaigns,
requiring to secure dedicated Cloud
resources over a given period of time
● Multiple deployment of different
versions of a same application
○ According to different funding context
■ e.g. a stakeholder funding a new evolution of
the application for its own usage only

Ellip Solutions, for EO Application Integration
● Integration environment for the test & validation of EO data processing components developed in
several programming language:
○ C/C++, Java, Python, R, Matlab, IDL
● Automated build & packaging, based on workflow code wrappers to ensure scalability
● Cloud bursting (deployer engine) to selected Production environment (Cloud providers)

Cloud APIs, for Hybrid Cloud capability
Openstack API
powered by libcloud
CloudFerro IaaS
EODC Cloud
powered by jclouds
Commercial Providers
OCCI,
& native cloud APIs
EGI Federated Cloud
OpenNebula Cloud Controller
Terradue
ONDA DIASSobloo DIAS
Openstack API
powered by libcloud
Openstack API
powered by libcloudOpenstack API -
powered by libcloud
CREODIAS

Cloud Bursting: some use cases
● Multiple deployment of a same
application
○ For different target user audiences
○ and/or for different production campaigns,
requiring to secure dedicated Cloud
resources over a given period of time
● Multiple deployment of different
versions of a same application
○ According to different funding context
■ e.g. a stakeholder funding a new evolution of
the application for its own usage only
● This is supported in NextGEOSS by
the “Ellip Launchpads” solution:
○ Currently in beta release
○ Deployments are still operated by the
Operations support team at Terradue
■ on behalf of application owners,
■ after agreement on the target Cloud
resources to use for the deployment in
production

Cloud Bursting service - Operations
● Deploy and operate the NextGEOSS Cloud
resources in support of “Application Provider”
needs:
○ After the setup and validation of your Cloud
Appliance on your Cloud Integration
environment
■ Based on the configured Cloud Platform
repositories per Pilot service
■ Based on the configured Cloud Platform
deployers per Pilot service
○ And for the authorised and selected Cloud
Production environments per Pilot
■ Appliance deployment on the ‘production’ ICT
resources (Cloud bursting)
● Production servers on EGI federated
Cloud using the allocated budget
● Or other IaaS as needed
■ Then start of the Operations:
● Partner is in control of the application
running (directly, or via the authorised
Community Portals)

SATCEN Pilot – Application deployed on AWS and Hetzner
Reference to the Cloud
appliance to be deployed
Application owner:
Operations Support:

Simple configuration fields
to define the “Deployers”

Submit request to the Launchpad service to
perform the deployment

Cloud Bursting service - State of play
● Resources for developers
○ NextGEOSS Integration Guide (v2)
○ Ellip Solutions user guide
■ How to join / My account
■ Ellip Core services
○ UPCOMING: Ellip Launchpads user guide
● Support activities
○ As part of the NextGEOSS support team
■ On the NextGEOSS Service Desk system
○ As part of the user training sessions
■ Hands-on workshops with project partners
■ Face to face trainings (Sept. 2018 & upcoming
July 2019)
■ Webinars (April 2018, Nov. 2019)
○ Dedicated support telcos (on-demand)
○ Support site for business partners
■ https://support.terradue.com
● Currently used for operations by
○ NextGEOSS Platform Services
■ Sentinel-2 Biopar vegetation indices Cloud appliance for
crops monitoring and food security policies, by VITO
■ Environmental predictors and RS-EBVs model Cloud
appliance for Biodiversity Habitat Monitoring, by WENR
■ CAMS Solar radiation gridded data Cloud appliance for
photovoltaic plants production monitoring, by ARMINES
■ Sentinel-1 Change Detection Cloud appliance for
humanitarian relief and border security, by SATCEN
■ Sentinel-1 Sea Ice concentration Cloud appliance for Cold
Regions monitoring and policy making, by NERSC
■ Sentinel-1 InSAR-based ground velocity mapping Cloud
appliance for Disaster Risk Assessment, by NOA
■ Sentinel-2 Leaf Area Index Cloud appliance, by ITC
■ UPCOMING:
■ Sentinel-2 Land Use Cloud appliance, by DEIMOS
■ Objects Drift Model, by CLS
■ Wetlands water management, by FSU Jena
○ Other deployments in operations
■ ESA GEP - Geohazards Exploitation Platform (30+)
■ ESA HEP - Hydrology Exploitation Platform (15+)
■ H2020 INTAROS - integrated Arctic Observing System
■ H2020 Co-ReSyF - Coastal Areas Research Platform
■ H2020 EcoPotential - Protected Areas EO data products

Cloud Bursting service
● Offer
○ Make your Cloud-ready application available
as a “Cloud appliance” on the Platform
repository
■ Decide on your Cloud appliances access
conditions: only to your organisation, shared
with partner organisations, ...
○ Leverage the “deployers” previously setup for
you, according to your existing Cloud
provisioning arrangements
■ Your credited accounts for a Cloud Provider, …
○ Adapt your Cloud deployment strategies
as new opportunities come to you
Ensure Cloud provisioning
is performed according to
the credit lines in place
Get access to the deployer
service matching your Cloud
provider resources
Define configuration
and “deploy” !
Check your
application service
endpoints in
production, and run
operations
Cloud Controller APIs
Deployer APIs

Data analysis systems built with our solutions
Run and monitor data processing jobs
OGC Web Processing Service (WPS)
interface for management of processing jobs
OGC Web Processing Service (WPS)
accessed from a Jupyter Notebook

Looking forward hearing from you!
NextGEOSS Webinar, November 28th 2019
Hervé Caumont (Terradue)
Bjorn Backeberg (EGI.eu)
Cloud APIs

NextGEOSS Webinar - Cloud APIs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to NextGEOSS Webinar - Cloud APIs

Similar to NextGEOSS Webinar - Cloud APIs (20)

More from terradue

More from terradue (20)

Recently uploaded

Recently uploaded (20)

NextGEOSS Webinar - Cloud APIs