SlideShare a Scribd company logo
1 of 36
Download to read offline
The HDF Group

Support for NPP/NPOESS
by The HDF Group
Mike Folk, Elena Pourmal, Peter Cao
The HDF Group
November 5, 2009

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

1

www.hdfgroup.org
Goal
To develop and adapt HDF5
software and provide HDF5
support for NPP/NPOESS.

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

2

www.hdfgroup.org
Areas of immediate need
• Ensure easy and intuitive data access and
use by diverse communities
• Enable producers and consumers to view
content, manage metadata, and convert data
to other formats.
• Provide high-quality, rapid-response support
of HDF5 for NPOESS users at all levels.

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

3

www.hdfgroup.org
2009-2009 Priorities
• Data accessibility and usability
• Developed library of high level APIs to support
NPP/NPOESS data management

• Tool development
• Modify h5dump to enable viewing of region
references and quality flags
• Modify HDFView to enhance viewing of
references and quality flags

• System maintenance
• User support
November 3-5, 2009

HDF/HDF-EOS Workshop XIII

4

www.hdfgroup.org
NPP/NPOESS Project Information
• Project Web site
• http://www.hdfgroup.org/projects/npoess/
• Project description
• Released software
• Documentation

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

5

www.hdfgroup.org
NPOESS Web site

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

6

www.hdfgroup.org
NPOESS Software FTP Site

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

7

www.hdfgroup.org
NPOESS Project Information
• Project Wiki site
• http://wiki.hdfgroup.org/NPOESS-Project
• Monthly progress reports, links to software
under development, presentations, etc.
• Requires a password
• Contact epourmal@hdfgroup.org

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

8

www.hdfgroup.org
HDF NPOESS Wiki

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

9

www.hdfgroup.org
The HDF Group

Library of High-Level APIs
to support NPOESS data

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

10

www.hdfgroup.org
NPOESS High-Level Library
• Library to support HDF5 region references and
packed bits in support of NPOESS quality flags
•
•
•
•

C APIs
Documentation
Tests
Examples

• Source available from
• ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/s
ource/

• Documentation is available from
• http://www.hdfgroup.org/projects/npoess/

• Still a prototype!
November 3-5, 2009

HDF/HDF-EOS Workshop XIII

11

www.hdfgroup.org
Documentation

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

12

www.hdfgroup.org
The HDF Group

h5dump
New Features to Support Region
References and Quality Flags

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

13

www.hdfgroup.org
h5dump and Region Reference Data
• New flag –R to display data pointed to by
region references
• Improved display of subsets
• An element of a dataset is specified by its
indices

• Available in HDF5 1.8.4, November 2009

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

14

www.hdfgroup.org
h5dump and Quality Flags
• New flag –M to display packed bit-field data
• Available only in the HDF5 NPOESS branch
• Will NOT be in HDF5 1.8.4 release
• Source and pre-built binaries can be found at
• ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

15

www.hdfgroup.org
The HDF Group

HDFView
New Features to Support
References and Quality Flags

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

16

www.hdfgroup.org
HDFView and references
• Aggregated granules: show the whole
dataset pointed to by an object reference
• Granules: show data pointed to by a dataset
region reference

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

17

www.hdfgroup.org
Objects Pointed to by Object References

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

18

www.hdfgroup.org
Data Pointed to by Region References

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

19

www.hdfgroup.org
HDFView and quality flags
• By default, integers with quality flags are
displayed as 8-bit decimal values
• Needed: a way to show values of specific
quality flag bit fields

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

20

www.hdfgroup.org
Default display: 8-bit decimal integers

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

21

www.hdfgroup.org
Displaying quality flags in 8-bit integers

• Use “Open As” on dataset with quality flags.
• Apply bitmask to choose individual quality
flags

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

22

www.hdfgroup.org
Examples of bit masks
• Applied mask in red at top-right, e.g., “By bitmask {0, 1}”

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

23

www.hdfgroup.org
Quality Flags
• Data can be displayed in decimal, binary, or hexadecimal

al
cim
de
ry
i na
b

d
xa
he
November 3-5, 2009

ci
e

al
m

HDF/HDF-EOS Workshop XIII

24

www.hdfgroup.org
HDFView plug-in to Show Quality Flags

• Implemented using “TreeView” plug-in
•
•
•
•

Open HDFView
Select “User Options”
Select NPOESS plug-in
Restart HDFView

• Available with
HDFView 2.5 patch3

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

25

www.hdfgroup.org
The HDF Group

Possible Future Work

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

26

www.hdfgroup.org
Possible future work
•
•
•
•

Add new features
Enhance current features
System support
User support

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

27

www.hdfgroup.org
Maintenance
• Maintenance NPOESS/NPP features
•
•
•
•

High-Level Library
H5dump, other tools?
HDFView
Testing, documentation, releases

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

28

www.hdfgroup.org
New features
• Support Variable-Length datasets
• Ease access to information about quality flags
• Use definition of quality flags, which are in:
• The Common Data Format Control Book
• XML files as part of the CDFCB-X delivery

• Apply the structure of the XML to determine
whether a field is a packed quality flag byte
• Tell us what you need!

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

29

www.hdfgroup.org
Enhance current features
• Extend reference display options
• E.g., show properties of the referenced data

• Revise default TreeView and TableView to
improve NPOESS plug-in support
• Improve quality of testing, documentation, and
binary and source distribution
• Tell us what you need!

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

30

www.hdfgroup.org
NPOESS/NPP System Support
• Daily testing on systems critical to NPOESS/NPP
• Linux 32 and 64-bit
• Windows
• AIX 5.3
• Solaris 32 and 64-bit
• Regular testing of NPOESS high-Level library
• Linux 32 and 64-bit
• Windows
• Solaris 32 and 64-bit
• Porting to requested compilers
• Performance tuning for applications
• Improved support for C++
• Support for netCDF-4 access to NPOESS SDR and EDR
products

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

31

www.hdfgroup.org
NPOESS/NPP User Support
• Priority support to NPOESS/NPP users
• Send email to help@hdfgroup.org
• Add NPOESS to the subject field
• Example:
Subject: [NPOESS] Where is the plug-in?

• Response in less than 4 hours
• Continue gather requirements and enhance
HDF5 software as needed

• We want to hear from you!
November 3-5, 2009

HDF/HDF-EOS Workshop XIII

32

www.hdfgroup.org
Special Research Projects
• Improvements to HDF5 or HDF5 NPOESS
products with the data management portion
of the Raytheon Algorithm Development
Library (ADL)
• Implement OpeNDAP server for HDF5
NPOESS SDR and EDR products
• Develop NPOESS specialized version of the
HDF5 tool h5check
• Performance tuning of HDF-NPOESS
product files
November 3-5, 2009

HDF/HDF-EOS Workshop XIII

33

www.hdfgroup.org
Questions?

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

34

www.hdfgroup.org
The HDF Group

Questions/comments?

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

35

www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement
number NA08AANEG0128from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the author[s] and do not necessarily reflect
the views of the National Aeronautics and Space
Administration.

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

36

www.hdfgroup.org

More Related Content

What's hot

Hdf5 current future
Hdf5 current futureHdf5 current future
Hdf5 current futuremfolk
 
Building Your Data Streams for all the IoT
Building Your Data Streams for all the IoTBuilding Your Data Streams for all the IoT
Building Your Data Streams for all the IoTDevOps.com
 
Best practices and lessons learnt from Running Apache NiFi at Renault
Best practices and lessons learnt from Running Apache NiFi at RenaultBest practices and lessons learnt from Running Apache NiFi at Renault
Best practices and lessons learnt from Running Apache NiFi at RenaultDataWorks Summit
 
NiFi Best Practices for the Enterprise
NiFi Best Practices for the EnterpriseNiFi Best Practices for the Enterprise
NiFi Best Practices for the EnterpriseGregory Keys
 

What's hot (20)

Hdf5 current future
Hdf5 current futureHdf5 current future
Hdf5 current future
 
Status of HDF-EOS, Related Software and Tools
 Status of HDF-EOS, Related Software and Tools Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS, Related Software and Tools
 
HDF Updae
HDF UpdaeHDF Updae
HDF Updae
 
2011 ACSI Survey Summary
2011 ACSI Survey Summary2011 ACSI Survey Summary
2011 ACSI Survey Summary
 
Support for NPP/NPOESS by The HDF Group
Support for NPP/NPOESS by The HDF GroupSupport for NPP/NPOESS by The HDF Group
Support for NPP/NPOESS by The HDF Group
 
HDF-Java Overview
HDF-Java OverviewHDF-Java Overview
HDF-Java Overview
 
Adding new servicees for HDF in THREDDS Data Server (TDS)
Adding new servicees for HDF in THREDDS Data Server (TDS)Adding new servicees for HDF in THREDDS Data Server (TDS)
Adding new servicees for HDF in THREDDS Data Server (TDS)
 
HDF Project Update
HDF Project UpdateHDF Project Update
HDF Project Update
 
HDF OPeNDAP update
HDF OPeNDAP updateHDF OPeNDAP update
HDF OPeNDAP update
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF and ENVI Services Engine
HDF and ENVI Services EngineHDF and ENVI Services Engine
HDF and ENVI Services Engine
 
Building Your Data Streams for all the IoT
Building Your Data Streams for all the IoTBuilding Your Data Streams for all the IoT
Building Your Data Streams for all the IoT
 
HDF Studio
HDF StudioHDF Studio
HDF Studio
 
Using IDL with Suomi NPP VIIRS Data
Using IDL with Suomi NPP VIIRS DataUsing IDL with Suomi NPP VIIRS Data
Using IDL with Suomi NPP VIIRS Data
 
Transitioning from HDF4 to HDF5
Transitioning from HDF4 to HDF5Transitioning from HDF4 to HDF5
Transitioning from HDF4 to HDF5
 
HDF Workshop SlideShare Experiment
HDF Workshop SlideShare ExperimentHDF Workshop SlideShare Experiment
HDF Workshop SlideShare Experiment
 
HDF OPeNDAP Project Update and Demo
HDF OPeNDAP Project Update and DemoHDF OPeNDAP Project Update and Demo
HDF OPeNDAP Project Update and Demo
 
Best practices and lessons learnt from Running Apache NiFi at Renault
Best practices and lessons learnt from Running Apache NiFi at RenaultBest practices and lessons learnt from Running Apache NiFi at Renault
Best practices and lessons learnt from Running Apache NiFi at Renault
 
Make CERES products user-friendly
Make CERES products user-friendlyMake CERES products user-friendly
Make CERES products user-friendly
 
NiFi Best Practices for the Enterprise
NiFi Best Practices for the EnterpriseNiFi Best Practices for the Enterprise
NiFi Best Practices for the Enterprise
 

Viewers also liked

Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...The HDF-EOS Tools and Information Center
 
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...The HDF-EOS Tools and Information Center
 

Viewers also liked (20)

EOSDIS Survey Overview
EOSDIS Survey OverviewEOSDIS Survey Overview
EOSDIS Survey Overview
 
MATLAB, netCDF, and OPeNDAP
MATLAB, netCDF, and OPeNDAPMATLAB, netCDF, and OPeNDAP
MATLAB, netCDF, and OPeNDAP
 
IDL 8.0 & Update on IDL Development for HDF
IDL 8.0 & Update on IDL Development for HDFIDL 8.0 & Update on IDL Development for HDF
IDL 8.0 & Update on IDL Development for HDF
 
Introduction to HDF5
Introduction to HDF5Introduction to HDF5
Introduction to HDF5
 
Earth Science Data and Information System (ESDIS) Project Update
Earth Science Data and Information System (ESDIS) Project UpdateEarth Science Data and Information System (ESDIS) Project Update
Earth Science Data and Information System (ESDIS) Project Update
 
EOSDIS Status
EOSDIS StatusEOSDIS Status
EOSDIS Status
 
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...
 
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout MapsEnsuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
 
HDF-VFS
HDF-VFSHDF-VFS
HDF-VFS
 
Status of HDF-EOS, Related Software, and Tools
Status of HDF-EOS, Related Software, and ToolsStatus of HDF-EOS, Related Software, and Tools
Status of HDF-EOS, Related Software, and Tools
 
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...
Application of HDF/HDF-EOS data to atmospheric and climate sciences at Univer...
 
Easy Remote Access Via OPeNDAP
Easy Remote Access Via OPeNDAPEasy Remote Access Via OPeNDAP
Easy Remote Access Via OPeNDAP
 
Bridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data ProductsBridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data Products
 
HDF5 Tools
HDF5 ToolsHDF5 Tools
HDF5 Tools
 
iRODS: Interoperability in Data Management
iRODS: Interoperability in Data ManagementiRODS: Interoperability in Data Management
iRODS: Interoperability in Data Management
 
HDF-Java Products
HDF-Java ProductsHDF-Java Products
HDF-Java Products
 
Easy Access of NASA HDF data via OPeNDAP
Easy Access of NASA HDF data via OPeNDAPEasy Access of NASA HDF data via OPeNDAP
Easy Access of NASA HDF data via OPeNDAP
 
HDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSSHDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSS
 
HDF Tools Updates and Discussions
HDF Tools Updates and DiscussionsHDF Tools Updates and Discussions
HDF Tools Updates and Discussions
 
Support for NPP/NPOESS/JPSS by The HDF Group
 Support for NPP/NPOESS/JPSS by The HDF Group Support for NPP/NPOESS/JPSS by The HDF Group
Support for NPP/NPOESS/JPSS by The HDF Group
 

Similar to Support for NPP/NPOESS by The HDF Group

Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsInteroperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsThe HDF-EOS Tools and Information Center
 

Similar to Support for NPP/NPOESS by The HDF Group (19)

The New HDF-EOS WebSite - How it can help you
The New HDF-EOS WebSite - How it can help youThe New HDF-EOS WebSite - How it can help you
The New HDF-EOS WebSite - How it can help you
 
Introduction to HDF5
Introduction to HDF5Introduction to HDF5
Introduction to HDF5
 
Parallel HDF5 Developments
Parallel HDF5 DevelopmentsParallel HDF5 Developments
Parallel HDF5 Developments
 
HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)
 
Hdf5 intro
Hdf5 introHdf5 intro
Hdf5 intro
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
Introduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming ModelsIntroduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming Models
 
HDF Project Update
HDF Project UpdateHDF Project Update
HDF Project Update
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsInteroperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
 
HDF5 OPeNDAP project update and demo
HDF5 OPeNDAP project update and demoHDF5 OPeNDAP project update and demo
HDF5 OPeNDAP project update and demo
 
HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?
 
HDF4 Mapping Project Update
HDF4 Mapping Project UpdateHDF4 Mapping Project Update
HDF4 Mapping Project Update
 
Data Interoperability
Data InteroperabilityData Interoperability
Data Interoperability
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF5 Software Process
HDF5 Software ProcessHDF5 Software Process
HDF5 Software Process
 
HDF Tools Tutorial
HDF Tools TutorialHDF Tools Tutorial
HDF Tools Tutorial
 
HDF5 Tools Updates
HDF5 Tools UpdatesHDF5 Tools Updates
HDF5 Tools Updates
 

More from The HDF-EOS Tools and Information Center

STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
 

More from The HDF-EOS Tools and Information Center (20)

Cloud-Optimized HDF5 Files
Cloud-Optimized HDF5 FilesCloud-Optimized HDF5 Files
Cloud-Optimized HDF5 Files
 
Accessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDSAccessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDS
 
The State of HDF
The State of HDFThe State of HDF
The State of HDF
 
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance FeaturesHighly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
 
Creating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 FilesCreating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 Files
 
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance DiscussionHDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
 
Hyrax: Serving Data from S3
Hyrax: Serving Data from S3Hyrax: Serving Data from S3
Hyrax: Serving Data from S3
 
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLABAccessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
 
HDF - Current status and Future Directions
HDF - Current status and Future DirectionsHDF - Current status and Future Directions
HDF - Current status and Future Directions
 
HDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and FutureHDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and Future
 
HDF - Current status and Future Directions
HDF - Current status and Future Directions HDF - Current status and Future Directions
HDF - Current status and Future Directions
 
H5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only LibraryH5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only Library
 
MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10
 
HDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDFHDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDF
 
HDF5 <-> Zarr
HDF5 <-> ZarrHDF5 <-> Zarr
HDF5 <-> Zarr
 
HDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server FeaturesHDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server Features
 
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
 
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
 
HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020
 
Leveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software TestingLeveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software Testing
 

Recently uploaded

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 

Recently uploaded (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 

Support for NPP/NPOESS by The HDF Group

  • 1. The HDF Group Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal, Peter Cao The HDF Group November 5, 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 1 www.hdfgroup.org
  • 2. Goal To develop and adapt HDF5 software and provide HDF5 support for NPP/NPOESS. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 2 www.hdfgroup.org
  • 3. Areas of immediate need • Ensure easy and intuitive data access and use by diverse communities • Enable producers and consumers to view content, manage metadata, and convert data to other formats. • Provide high-quality, rapid-response support of HDF5 for NPOESS users at all levels. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 3 www.hdfgroup.org
  • 4. 2009-2009 Priorities • Data accessibility and usability • Developed library of high level APIs to support NPP/NPOESS data management • Tool development • Modify h5dump to enable viewing of region references and quality flags • Modify HDFView to enhance viewing of references and quality flags • System maintenance • User support November 3-5, 2009 HDF/HDF-EOS Workshop XIII 4 www.hdfgroup.org
  • 5. NPP/NPOESS Project Information • Project Web site • http://www.hdfgroup.org/projects/npoess/ • Project description • Released software • Documentation November 3-5, 2009 HDF/HDF-EOS Workshop XIII 5 www.hdfgroup.org
  • 6. NPOESS Web site November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6 www.hdfgroup.org
  • 7. NPOESS Software FTP Site November 3-5, 2009 HDF/HDF-EOS Workshop XIII 7 www.hdfgroup.org
  • 8. NPOESS Project Information • Project Wiki site • http://wiki.hdfgroup.org/NPOESS-Project • Monthly progress reports, links to software under development, presentations, etc. • Requires a password • Contact epourmal@hdfgroup.org November 3-5, 2009 HDF/HDF-EOS Workshop XIII 8 www.hdfgroup.org
  • 9. HDF NPOESS Wiki November 3-5, 2009 HDF/HDF-EOS Workshop XIII 9 www.hdfgroup.org
  • 10. The HDF Group Library of High-Level APIs to support NPOESS data November 3-5, 2009 HDF/HDF-EOS Workshop XIII 10 www.hdfgroup.org
  • 11. NPOESS High-Level Library • Library to support HDF5 region references and packed bits in support of NPOESS quality flags • • • • C APIs Documentation Tests Examples • Source available from • ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/s ource/ • Documentation is available from • http://www.hdfgroup.org/projects/npoess/ • Still a prototype! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 11 www.hdfgroup.org
  • 12. Documentation November 3-5, 2009 HDF/HDF-EOS Workshop XIII 12 www.hdfgroup.org
  • 13. The HDF Group h5dump New Features to Support Region References and Quality Flags November 3-5, 2009 HDF/HDF-EOS Workshop XIII 13 www.hdfgroup.org
  • 14. h5dump and Region Reference Data • New flag –R to display data pointed to by region references • Improved display of subsets • An element of a dataset is specified by its indices • Available in HDF5 1.8.4, November 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 14 www.hdfgroup.org
  • 15. h5dump and Quality Flags • New flag –M to display packed bit-field data • Available only in the HDF5 NPOESS branch • Will NOT be in HDF5 1.8.4 release • Source and pre-built binaries can be found at • ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/NPOESS/ November 3-5, 2009 HDF/HDF-EOS Workshop XIII 15 www.hdfgroup.org
  • 16. The HDF Group HDFView New Features to Support References and Quality Flags November 3-5, 2009 HDF/HDF-EOS Workshop XIII 16 www.hdfgroup.org
  • 17. HDFView and references • Aggregated granules: show the whole dataset pointed to by an object reference • Granules: show data pointed to by a dataset region reference November 3-5, 2009 HDF/HDF-EOS Workshop XIII 17 www.hdfgroup.org
  • 18. Objects Pointed to by Object References November 3-5, 2009 HDF/HDF-EOS Workshop XIII 18 www.hdfgroup.org
  • 19. Data Pointed to by Region References November 3-5, 2009 HDF/HDF-EOS Workshop XIII 19 www.hdfgroup.org
  • 20. HDFView and quality flags • By default, integers with quality flags are displayed as 8-bit decimal values • Needed: a way to show values of specific quality flag bit fields November 3-5, 2009 HDF/HDF-EOS Workshop XIII 20 www.hdfgroup.org
  • 21. Default display: 8-bit decimal integers November 3-5, 2009 HDF/HDF-EOS Workshop XIII 21 www.hdfgroup.org
  • 22. Displaying quality flags in 8-bit integers • Use “Open As” on dataset with quality flags. • Apply bitmask to choose individual quality flags November 3-5, 2009 HDF/HDF-EOS Workshop XIII 22 www.hdfgroup.org
  • 23. Examples of bit masks • Applied mask in red at top-right, e.g., “By bitmask {0, 1}” November 3-5, 2009 HDF/HDF-EOS Workshop XIII 23 www.hdfgroup.org
  • 24. Quality Flags • Data can be displayed in decimal, binary, or hexadecimal al cim de ry i na b d xa he November 3-5, 2009 ci e al m HDF/HDF-EOS Workshop XIII 24 www.hdfgroup.org
  • 25. HDFView plug-in to Show Quality Flags • Implemented using “TreeView” plug-in • • • • Open HDFView Select “User Options” Select NPOESS plug-in Restart HDFView • Available with HDFView 2.5 patch3 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 25 www.hdfgroup.org
  • 26. The HDF Group Possible Future Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 26 www.hdfgroup.org
  • 27. Possible future work • • • • Add new features Enhance current features System support User support November 3-5, 2009 HDF/HDF-EOS Workshop XIII 27 www.hdfgroup.org
  • 28. Maintenance • Maintenance NPOESS/NPP features • • • • High-Level Library H5dump, other tools? HDFView Testing, documentation, releases November 3-5, 2009 HDF/HDF-EOS Workshop XIII 28 www.hdfgroup.org
  • 29. New features • Support Variable-Length datasets • Ease access to information about quality flags • Use definition of quality flags, which are in: • The Common Data Format Control Book • XML files as part of the CDFCB-X delivery • Apply the structure of the XML to determine whether a field is a packed quality flag byte • Tell us what you need! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 29 www.hdfgroup.org
  • 30. Enhance current features • Extend reference display options • E.g., show properties of the referenced data • Revise default TreeView and TableView to improve NPOESS plug-in support • Improve quality of testing, documentation, and binary and source distribution • Tell us what you need! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 30 www.hdfgroup.org
  • 31. NPOESS/NPP System Support • Daily testing on systems critical to NPOESS/NPP • Linux 32 and 64-bit • Windows • AIX 5.3 • Solaris 32 and 64-bit • Regular testing of NPOESS high-Level library • Linux 32 and 64-bit • Windows • Solaris 32 and 64-bit • Porting to requested compilers • Performance tuning for applications • Improved support for C++ • Support for netCDF-4 access to NPOESS SDR and EDR products November 3-5, 2009 HDF/HDF-EOS Workshop XIII 31 www.hdfgroup.org
  • 32. NPOESS/NPP User Support • Priority support to NPOESS/NPP users • Send email to help@hdfgroup.org • Add NPOESS to the subject field • Example: Subject: [NPOESS] Where is the plug-in? • Response in less than 4 hours • Continue gather requirements and enhance HDF5 software as needed • We want to hear from you! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 32 www.hdfgroup.org
  • 33. Special Research Projects • Improvements to HDF5 or HDF5 NPOESS products with the data management portion of the Raytheon Algorithm Development Library (ADL) • Implement OpeNDAP server for HDF5 NPOESS SDR and EDR products • Develop NPOESS specialized version of the HDF5 tool h5check • Performance tuning of HDF-NPOESS product files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 33 www.hdfgroup.org
  • 34. Questions? November 3-5, 2009 HDF/HDF-EOS Workshop XIII 34 www.hdfgroup.org
  • 35. The HDF Group Questions/comments? November 3-5, 2009 HDF/HDF-EOS Workshop XIII 35 www.hdfgroup.org
  • 36. Acknowledgements This work was supported by cooperative agreement number NA08AANEG0128from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 36 www.hdfgroup.org

Editor's Notes

  1. A file format for managing