The document discusses NASA HDF/HDF-EOS data formats and how to work with them for Earth science data analysis. It begins with an overview of the complexity of NASA HDF data for new users. It then covers common HDF data formats like HDF4, HDF5, HDF-EOS2 and HDF-EOS5, explaining the differences and which APIs to use for each. It provides code examples for reading HDF-EOS data in C and Fortran. It also introduces tools that can help dump and extract HDF data in ASCII format for broader tool compatibility. Overall, the document aims to help new users understand and start working with NASA's HDF data products.

1. The HDF Group
NASA HDF/HDF-EOS Data for
Dummies (and Developers)
Joe Lee
The HDF Group
July 19, 2012
ESIP 2012 Summer Meeting
1
www.hdfgroup.org

2. How many dummies in this room?
I’ve never used
NASA HDF
data.
(What is HDF…?)
July 19, 2012
ESIP 2012 Summer Meeting
2
www.hdfgroup.org

3. I was and I am still a dummy
because NASA HDF is
complex!
Even after
5 years of work
at The HDF Group, I still have
something to
July 19, 2012
ESIP 2012 Summer Meeting
learn.
3
www.hdfgroup.org

4. I still remember the day one
as a NASA HDF dummy so I’d like to share
• common mistakes
• very useful tips
for handling NASA HDF data products
from the perspective of new users like you.
July 19, 2012
ESIP 2012 Summer Meeting
4
www.hdfgroup.org

5. Let’s feel the pain of a new user.
Analyze Drought in Madison,WI using NASA EOS data.
July 16 – Images of the Day – earthobservatory.org (6/25-7/10)
July 19, 2012
ESIP 2012 Summer Meeting
5
www.hdfgroup.org

6. Possible tasks for a new user
Compare the current drought period (6/25-7/10) data against
the average data of the same period in last 10 years in terms
of:
• Rainfall
• Ozone
• Normalized Difference Vegetation Index (NDVI)
• Wildfire
and
visualize the data!
Can a new user do it
easily with NASA satellite data?
July 19, 2012
ESIP 2012 Summer Meeting
6
www.hdfgroup.org

7. The answer will be no
because the user should master the skill of handling
many different NASA data products in HDF.
July 19, 2012
ESIP 2012 Summer Meeting
7
www.hdfgroup.org

8. Rainfall task example
A simple internet
search will
eventually lead to
GES DISC TRMM
web page that
distributes data in
HDF
.
July 19, 2012
ESIP 2012 Summer Meeting
8
www.hdfgroup.org

9. Same for Ozone, NDVI, and Wildfire
A simple internet search will eventually lead to various
NASA data centers web page that distributes data in
HDF.
HDF is simply
July 19, 2012
indispensable.
ESIP 2012 Summer Meeting
9
..
www.hdfgroup.org

10. So what is HDF?
I downloaded the .hdf file but no application can open.
PDF
HDF
It’s a file format for scientific data. It needs an application
like HDFView to open it.
July 19, 2012
ESIP 2012 Summer Meeting
10
www.hdfgroup.org

11. Why does NASA use HDF?
Why didn’t NASA use these well-known file formats?
• ASCII
• Spreadsheet (MS Excel)
• Database (MySQL, Oracle)
• Image (JPEG, GeoTIFF, GIF)
• XML
because HDF
is efficient data management
solution in terms of space (storage) and time
(performance).
July 19, 2012
ESIP 2012 Summer Meeting
11
www.hdfgroup.org

12. Is HDF also efficient for users?
Not necessarily because users may need to
learn
• new tools and APIs
• how to process the data correctly
Yet, there’s no “NASA HDF for Dummies” book.
Instead, help is
always available through
hdfgroup.org and hdfeos.org.
July 19, 2012
ESIP 2012 Summer Meeting
12
www.hdfgroup.org

13. What’s inside NASA HDF?
A typical NASA HDF file has
• attributes (metadata) describing the file contents itself
• coordinate variables (time, location – lat/lon/height)
• data variable (fire, water, air, land)
July 19, 2012
ESIP 2012 Summer Meeting
13
www.hdfgroup.org

14. NASA HDF has four faces
NASA data centers provide files in
• HDF4 (.h4, .hdf)
• HDF-EOS2 (he2, he)
• HDF5 (.h5)
• HDF-EOS5 (he5)
They are all different although they
share same HDF prefix.
July 19, 2012
ESIP 2012 Summer Meeting
14
www.hdfgroup.org

15. HDF4 vs HDF5
HDF4 is old. File starts with ^N^C^S^A.
HDF5 is new. File starts with 211HDF.
not compatible.
They are
You need different tools and APIs to
access them.
HDF4: Hopen()
HDF5: H5open()
July 19, 2012
ESIP 2012 Summer Meeting
15
www.hdfgroup.org

16. NASA HDF vs. HDF-EOS
HDF-
EOS is HDF + GCTP.
GCTP: General
Cartographic
Transformation Package Library
HDF: has latitude/longitude dataset.
HDF-EOS: has no latitude/longitude dataset.
July 19, 2012
ESIP 2012 Summer Meeting
16
www.hdfgroup.org

17. Why does HDF-EOS includes GCTP?
Save space for geo-location (map) information data
by storing only a few parameters
in the special
July 19, 2012
StructMetadata file attribute.
ESIP 2012 Summer Meeting
17
www.hdfgroup.org

18. HDF-EOS is based on HDF API (1)
so HDF APIs can read all HDF-EOS objects.
no easy way
However, there’s
to
figure out geo-location information with HDF
APIs alone.
exception: geographic projection / regular grid
July 19, 2012
ESIP 2012 Summer Meeting
18
www.hdfgroup.org

19. HDF-EOS API vs. HDF API
Thus, use HDF-EOS APIs for HDF-EOS products
instead of HDF APIs to resolve lat/lon.
HDF-EOS2: GDopen()
HDF4: Hopen()
HDF-EOS5:HE5GDopen()
HDF5:H5Open()
HDF-EOS APIs start with (HE5)
July 19, 2012
ESIP 2012 Summer Meeting
GD/SW.
19
www.hdfgroup.org

20. What are GD/SW in HDF-EOS APIs?
GD: Grid
Level 3 data
global
SW: Swath
Level 2 data
regional
HDF-EOS API is specific to Earth Science domain.
HDF API is for any domain such as finance and biology.
July 19, 2012
ESIP 2012 Summer Meeting
20
www.hdfgroup.org

21. HDF-EOS API is based on HDF API (2)
so HDF APIs can modify all HDF-EOS objects.
Data producers can add
new data using
HDF API later. HDF-EOS APIs
cannot
access such data.
July 19, 2012
ESIP 2012 Summer Meeting
21
www.hdfgroup.org

22. NASA HDF format summary
If the NASA file
ends with
.he2
.hdf, .h4
HDF4 API
.he5
HDF-EOS5 API
.h5
July 19, 2012
Use
HDF5 API
HDF-EOS2 API
ESIP 2012 Summer Meeting
22
www.hdfgroup.org

23. NASA HDF format review
Quiz: If file ends with .he2 but has some
additional objects that HDF-EOS2 cannot
access, what a user should do?
Answer: Open and read twice. First, with
HDF-EOS2 API. Second, with HDF4 API for
additional objects.
July 19, 2012
ESIP 2012 Summer Meeting
23
www.hdfgroup.org

24. So far, we talked about formats only
Show me the real
example.
July 19, 2012
ESIP 2012 Summer Meeting
24
www.hdfgroup.org

25. Read HDF-EOS2 Grid data in C
#include <HdfEosDef.h>
int main(int argc, char **argv)
{
/* Open 'MCD43C4.A2000049.005.2006271205532.hdf' using grid API */
if ((gridfile1 = GDopen("MCD43C4.A2000049.005.2006271205532.hdf", DFACC_RDONLY)) == -1) {
}
/* Open a grid named 'MCD_CMG_BRDF_0.05Deg' */
if ((grid1 = GDattach(gridfile1, "MCD_CMG_BRDF_0.05Deg")) == -1) {
}
/* Retrieve information about 'Nadir_Reflectance_Band1' datafield */
if ((GDfieldinfo(grid1, "Nadir_Reflectance_Band1", &datafield1rank, datafield1dimsize, &datafield1type,
datafield1dimname)) == -1) {
}
/* Allocate buffer for 'Nadir_Reflectance_Band1' */
if ((datafield1data = malloc(sizeof(int16) * 3600 * 7200)) == NULL) {
}
/* Read data from 'Nadir_Reflectance_Band1' */
if ((GDreadfield(grid1, "Nadir_Reflectance_Band1", NULL, NULL, NULL, datafield1data)) == -1) {
}
/* Close the grid named 'MCD_CMG_BRDF_0.05Deg' */
if ((GDdetach(grid1)) == -1) {
}
/* Close 'MCD43C4.A2000049.005.2006271205532.hdf' */
if ((GDclose(gridfile1)) == -1) {
}
return 0;
}
July 19, 2012
ESIP 2012 Summer Meeting
25
www.hdfgroup.org

26. Read HDF-EOS5 Swath data in FORTRAN
c Open 'OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5' using swath
c API
swathfile1 = he5_swopen
$ ('OMI-Aura_L2-OMAERO_2004m1001t0003-o01132_v003.he5',
$ HE5F_ACC_RDONLY)
c Open a swath named 'ColumnAmountAerosol'
swath1 = he5_swattach(swathfile1, 'ColumnAmountAerosol')
c Retrieve information about 'AerosolIndexUV' datafield
status = he5_swfldinfo(swath1, 'AerosolIndexUV', datafield1rank,
$ datafield1dimsize, datafield1type, datafield1dimname,
$ datafield1maxdimname)
c Select necessary range of elements
datafield1start(1) = 0
datafield1stride(1) = 1
datafield1edge(1) = 60
datafield1start(2) = 0
datafield1stride(2) = 1
datafield1edge(2) = 1644
c Read data from 'AerosolIndexUV'
status = he5_swrdfld(swath1, 'AerosolIndexUV', datafield1start,
$ datafield1stride, datafield1edge, datafield1data)
c Dump data from 'AerosolIndexUV'
do i = 1, 16
do j = 1, 6
write(*,*), datafield1data(j,i)
enddo
write(*,*)
enddo
July 19, 2012
ESIP 2012 Summer Meeting
26
www.hdfgroup.org

27. Complete Examples for Dummies
Complete and more examples:
http://hdfeos.org/examples/library.php
Don’t miss
how to read lat/lon:
http://hdfeos.org/examples/c_grid_lonlat.php
July 19, 2012
ESIP 2012 Summer Meeting
27
www.hdfgroup.org

28. HDF/HDF-EOS API look too difficult
Is there a simple API
like netCDF?
July 19, 2012
ESIP 2012 Summer Meeting
28
www.hdfgroup.org

29. H4CF Conversion Library APIs
// open the example HDF file
h4cf_open("geo.hdf");
// HDF file attributes are obtained in a C++
STL map object
const map<string, string> file_attrs =
h4cf_get_file_attrs();
// HDF file variables are obtained in a C++
STL list object
const list<var*> pvars = h4cf_get_vars();
// close the HDF file
h4cf_close();
July 19, 2012
ESIP 2012 Summer Meeting
29
www.hdfgroup.org

30. H4CF Conversion Library
• works on both HDF4 and HDF-EOS2.
• Variables (datasets) list
• Attributes (metadata) list
• All Vars & Attrs follow CF
conventions
• Develop your own CF application!
July 19, 2012
ESIP 2012 Summer Meeting
30
www.hdfgroup.org

31. What is CF conventions?
• Metadata conventions for earth science data.
• Sharing of files created with the NetCDF APIs,
but not specifically to netCDF.
• The CF conventions are now increasingly gaining
acceptance.
• URL: http://cf-pcmdi.llnl.gov/
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
31
www.hdfgroup.org

32. Don’t follow CF conventions blindly.
[1] CF: scale * value + offset
[2] MODIS: scale * (value – offset)
Scale = 0.01
Value = 5000
Offset = -15000.0
[1] 0.01 * 5000 + (-15000) = -14950
[2] 0.01 * (5000 – (-15000)) = 200
July 19, 2012
ESIP 2012 Summer Meeting
32
www.hdfgroup.org

33. -14950 Kelvin?
July 19, 2012
ESIP 2012 Summer Meeting
33
www.hdfgroup.org

34. MODIS Scale / Offset Correction
July 19, 2012
ESIP 2012 Summer Meeting
34
www.hdfgroup.org

35. Limitation – No Support Yet
MISR
(SOM/LAMAZ)
MERRA
AVHRR
LISTO
QuikSCAT/SeaWind
MOPITT L2
etc.
July 19, 2012
ESIP 2012 Summer Meeting
35
www.hdfgroup.org

36. Is H5CF Conversion Library available?
• Unfortunately, no.
• Compared to HDF4/HDF-EOS2,
HDF5/HDF-EOS5 products are not
diverse.
• HDF-EOS5 products follow NASA
AURA file guidelines and use only
geographic projections.
• New HDF5 products will follow CFconventions.
July 19, 2012
ESIP 2012 Summer Meeting
36
www.hdfgroup.org

37. How about programming dummies?
What is FORTRAN?
What is C?
What is C++?
…
July 19, 2012
ESIP 2012 Summer Meeting
37
www.hdfgroup.org

38. Why do I need APIs to read data?
Do you use PDF APIs to read PDF document?
• For most users, PDF reader application is
enough.
• Same for most HDF users. HDF reader
application is enough.
July 19, 2012
ESIP 2012 Summer Meeting
38
www.hdfgroup.org

39. Read HDF data in ASCII
Use generic HDF “dump” tools
• hdp for HDF-EOS2 and HDF4 files
• h5dump for HDF-EOS5 and HDF5 files
They are only good for attributes and data.
Cannot extract lat/lon from HDF-EOS Grid
July 19, 2012
ESIP 2012 Summer Meeting
39
www.hdfgroup.org

40. HDF-EOS2 Dumper Tool
Useful for tools that can’t handle some projections.
eos2
eos2
dump
ASCII
http://hdfeos.org/zoo
MATLAB
IDL
NCL
July 19, 2012
ESIP 2012 Summer Meeting
40
www.hdfgroup.org

41. Important Tool for Zoo Maintenance
July 19, 2012
ESIP 2012 Summer Meeting
41
www.hdfgroup.org

42. Why Tools Need HDF-EOS2 Dumper
HDF-EOS2
dumper
Lat / Lon
in
ASCII
Lat / Lon
LaRC MISR
SOM Projection
HDF-EOS2
July 19, 2012
MATLAB
IDL
NCL
ESIP 2012 Summer Meeting
42
www.hdfgroup.org

43. HDF-EOS2 Dumper Options
Usage: eos2dump options filename.hdf
What:
1=lat / 2 =lon
Format: m=5 values per line
July 19, 2012
ESIP 2012 Summer Meeting
43
www.hdfgroup.org

44. eos2dump Latitude Example
90
80
eos2dump –c filename.hdf > latitude.txt 70
60
50
40
What:
30
Format: one value per line (default)
20
10
0
-10
-20
-30
1
lat
July 19, 2012
ESIP 2012 Summer Meeting
44
www.hdfgroup.org

45. eos2dump Longitude Example
eos2dump –c2
m filename.hdf > longitude.txt
What: lon
Format: 5
values per line
MATLAB cannot read it.
180 175 170 165 160
155 150 145 140 135
130 125 120 115 110
July 19, 2012
ESIP 2012 Summer Meeting
45
www.hdfgroup.org

46. How about eos5dump?
• It’s not necessary because all HDF-EOS5
Grid use geographic projection.
• You can easily generate lat/lon using the
parameters stored in StructMetadata.
• E.g., 360 / x-dimension, 180 / y-dimension
July 19, 2012
ESIP 2012 Summer Meeting
46
www.hdfgroup.org

47. How about visualization?
0000111020230232
3234234299234234
2342343292343493
…
ASCII data is boring!
July 19, 2012
ESIP 2012 Summer Meeting
47
www.hdfgroup.org

48. HDFView
THE GUI Tool equivalent to Acrobat Reader.
• It can visualize data as image.
• It can even edit some data.
• Critical drawback: no
map!
• Why? Again, it’s a general purpose tool. Not
for Earth science users only.
July 19, 2012
ESIP 2012 Summer Meeting
48
www.hdfgroup.org

49. HDFView HDF-EOS Plug-in
An additional download for HDFView.
• It can visualize HDF-EOS2 and HDF-EOS5.
• It supports map.
July 19, 2012
ESIP 2012 Summer Meeting
49
www.hdfgroup.org

50. HDFView HDF-EOS Plug-in screenshot
Close but no cigar – it works only for HDF-EOS!
There are many generic HDF data products.
July 19, 2012
ESIP 2012 Summer Meeting
50
www.hdfgroup.org

51. Then, what should dummies do?
Any visualization tool
that works for both
HDF and HDF-EOS?
July 19, 2012
ESIP 2012 Summer Meeting
51
www.hdfgroup.org

52. Use a script language tool
such as
• NCL
• IDL
• MATLAB
with eos2dump tool
July 19, 2012
ESIP 2012 Summer Meeting
52
www.hdfgroup.org

53. Rich Dummy Poor Dummy
MATLAB and IDL licenses
are expensive!
July 19, 2012
ESIP 2012 Summer Meeting
53
www.hdfgroup.org

54. Solution for Poor Dummy
Try NCL.
• Free yet powerful.
• http://hdfeos.org/zoo page has complete codes
for visualizing most NASA data products in NCL.
• ncl-talk@ucar.edu mailing list will rock you.
July 19, 2012
ESIP 2012 Summer Meeting
54
www.hdfgroup.org

55. Solution for Rich Dummy
Visit Comprehensive
Examples page.
• http://hdfeos.org/zoo page has complete codes
for visualizing most NASA data products in IDL
and MATLAB.
July 19, 2012
ESIP 2012 Summer Meeting
55
www.hdfgroup.org

56. Poor and Lazy Dummy
I hate programming.
Any free visualization tool
that works for both
HDF/HDF-EOS?
July 19, 2012
ESIP 2012 Summer Meeting
56
www.hdfgroup.org

57. Recall: H4CF Conversion Library
• works on both HDF4 and HDF-EOS2.
• Variables (datasets) list
• Attributes (metadata) list
• All Vars & Attrs follow CF
conventions
• Develop your own CF
application!
July 19, 2012
ESIP 2012 Summer Meeting
57
www.hdfgroup.org

58. Solution: h4tonccf
• Usage: h4tonccf filename.hdf [filename.nc]
• generates either netcdf-3 or netcdf-4 file.
• follows netcdf
classic model.
• is based on H4CF Conversion Library.
Use free netcdf visualization tool like Panoply and IDV
after conversion.
July 19, 2012
ESIP 2012 Summer Meeting
58
www.hdfgroup.org

59. h4tonccf Demo
July 19, 2012
ESIP 2012 Summer Meeting
59
www.hdfgroup.org

60. Converted File Size Comparison
656M
Netcdf-3
128M
Netcdf-4
72M
HDF-EOS2
July 19, 2012
ESIP 2012 Summer Meeting
9X
60
www.hdfgroup.org

61. Same Limitation – No Support Yet
MISR
(SOM/LAMAZ)
MERRA
AVHRR
LISTO
QuikSCAT/SeaWind
MOPITT L2
etc.
July 19, 2012
ESIP 2012 Summer Meeting
61
www.hdfgroup.org

62. How about h5tonccf?
• Recall: no H5CF Conversion Library exists.
• The latest IDV and Panoply may open and
visualize HDF-EOS5 directly.
• However, no luck with HDF5.
July 19, 2012
ESIP 2012 Summer Meeting
62
www.hdfgroup.org

63. No hope for Poor and Lazy Dummy?
I hate programming.
I want free visualization tool
that works for both
HDF/HDF-EOS.
Am I hopeless?
July 19, 2012
ESIP 2012 Summer Meeting
63
www.hdfgroup.org

64. Solution for the hopeless: OPeNDAP
• Why OPeNDAP?
• What is HDF Handler?
• How important CF conventions?
July 19, 2012
ESIP 2012 Summer Meeting
64
www.hdfgroup.org

65. Access HDF via FTP
Aqua/Aura
Users
IDL
AIRS/OMI
FTP
Local HDF
Files
July 19, 2012
FTP
Clients
FTP
Servers
ESIP 2012 Summer Meeting
HDF Files
65
www.hdfgroup.org

66. OPeNDAP access
Typical remoteconcept problem
View Data
(User)
FTP/HTTP
Remote Data
(HDF4/5)
Visualization
Tools
tools
(e.g., GrADS)
(gradsdap)
Handler
(hdf4/5_handler)
Client Library
(libnc_dap)
Server
(Hyrax)
Data Access Protocol (DAP) via Internet
July 19, 2012
ESIP 2012 Summer Meeting
66
www.hdfgroup.org

67. Access remote data easily
July 19, 2012
ESIP 2012 Summer Meeting
67
www.hdfgroup.org

68. without worrying about data plans.
July 19, 2012
ESIP 2012 Summer Meeting
68
www.hdfgroup.org

69. Visualize remote data easily, too.
July 19, 2012
ESIP 2012 Summer Meeting
69
www.hdfgroup.org

70. HDF handler is key
Aqua/Aura
HDF
HANDLER
Users
IDV
Hyrax
NetCDF Java
AIRS/OMI
DAP
Visualization
Tools
July 19, 2012
OPeNDAP
Clients
OPeNDAP
Servers
ESIP 2012 Summer Meeting
HDF Files
70
www.hdfgroup.org

71. because if handler doesn’t follow CF
No Visualization Bonus!
July 19, 2012
ESIP 2012 Summer Meeting
71
www.hdfgroup.org

72. If handler gives up extra bonus
handler development is easier.
DAP
DAP Server
HDF / HDF-EOS
July 19, 2012
ESIP 2012 Summer Meeting
72
www.hdfgroup.org

73. What’s involved in extra bonus
Earth Scientist
Visualization
DAP Client
DAP
DAP Server
CF Conventions
HDF / HDF-EOS
July 19, 2012
ESIP 2012 Summer Meeting
73
www.hdfgroup.org

74. How handler developers feel
July 19, 2012
ESIP 2012 Summer Meeting
74
www.hdfgroup.org

75. Because we care all NASA products.
AURA
MISR
?
AQUA
TERRA
NPP
July 19, 2012
ESIP 2012 Summer Meeting
75
www.hdfgroup.org

76. Developers’ Dream
… all data are created equal,
that they shall be visualized
by all client tools easily…
July 19, 2012
ESIP 2012 Summer Meeting
76
www.hdfgroup.org

77. What if all people agreed…
development might have been much easier.
Earth Scientist
Visualization
DAP Client
DAP
DAP Server
Check once.
July 19, 2012
UDF (Imaginary)
ESIP 2012 Summer Meeting
77
www.hdfgroup.org

78. Lessons from handler development
Communicate and work together among
different groups.
Standardize everything from data products
to visualization tools.
Follow conventions and standards.
July 19, 2012
ESIP 2012 Summer Meeting
78
www.hdfgroup.org

79. Redesign of HDF5 handler
Goal: Support all NASA HDF5 products.
Modularize for future product support.
Follow CF
conventions robustly.
Product
July 19, 2012
ESIP 2012 Summer Meeting
79
www.hdfgroup.org

80. Why did we redesign?
In 2007,
HDF5 handler
AURA
Grid
was enough for
July 19, 2012
ESIP 2012 Summer Meeting
80
www.hdfgroup.org

81. Why did we redesign?
In 2011,
wasn’t enough for
AURA
Grid
July 19, 2012
AURA
Swath
ESIP 2012 Summer Meeting
AURA
ZA
81
GSSTF
www.hdfgroup.org

82. Why did we redesign?
In 2011,
wasn’t enough for
AURA
Grid
July 19, 2012
AURA
Swath
ESIP 2012 Summer Meeting
AURA
ZA
82
GSSTF
www.hdfgroup.org

83. and more… so we gave up!
ICESat-2
Aquarius
GOSAT
SMAP
July 19, 2012
ESIP 2012 Summer Meeting
83
www.hdfgroup.org

84. We rebuilt one that can scale up
July 19, 2012
ESIP 2012 Summer Meeting
84
www.hdfgroup.org

85. by modularizing the design
HDF5 handler
CF
Generic
HDFEOS5
HDF5
GOSAT
Aquarius
SMAP
with a
July 19, 2012
GRID
SWATH
ZA
big room for future products.
ESIP 2012 Summer Meeting
85
www.hdfgroup.org

86. Key updates on HDF5 handler
version
2.0.0
CF-convention
Support
Range of NASA
Data Products
Mulitiple
Grids/Swath/ZA
Configuration
Options
Support for
Future Products
July 19, 2012
1.5.x
Primitive
Robust
Limited
All
No
Yes
Static Build
Run-time
Rigid
Flexible
ESIP 2012 Summer Meeting
86
www.hdfgroup.org

87. HDF-EOS5 products supported
•
•
•
•
•
July 19, 2012
OMI Swath, Grid
MLS Swath
HIRDLS Swath, ZA
TES Swath, Grid
GSSTF
ESIP 2012 Summer Meeting
87
www.hdfgroup.org

88. HDF5 products supported
•
•
•
•
July 19, 2012
Aquarius Grid
MEaSUREs SeaWiFS DeepBlue
SBUV2 Ozone (O3) Profile
SMAP (Simulation)
ESIP 2012 Summer Meeting
88
www.hdfgroup.org

89. HIRDLS Zonal Average
July 19, 2012
ESIP 2012 Summer Meeting
89
www.hdfgroup.org

90. New TES Swath
July 19, 2012
ESIP 2012 Summer Meeting
90
www.hdfgroup.org

91. MEaSUREs GSSTF
July 19, 2012
ESIP 2012 Summer Meeting
91
www.hdfgroup.org

92. Aquarius Grid (Simulation)
July 19, 2012
ESIP 2012 Summer Meeting
92
www.hdfgroup.org

93. MEaSUREs SeaWiFS DeepBlue
July 19, 2012
ESIP 2012 Summer Meeting
93
www.hdfgroup.org

94. Ozone Profile (BUV)
July 19, 2012
ESIP 2012 Summer Meeting
94
www.hdfgroup.org

95. SMAP Swath (Simulation)
July 19, 2012
ESIP 2012 Summer Meeting
95
www.hdfgroup.org

96. SMAP Swath (Simulation)
July 19, 2012
ESIP 2012 Summer Meeting
96
www.hdfgroup.org

97. No CF magic for some products
Tool / CF Limitation
• GOSAT/acos: 1-D trajectory
• ICESat-2
• OMI L2G: Lat/Lon with 3-D
Java-NetCDF Client Limitation
• Dataset w/ long string
July 19, 2012
ESIP 2012 Summer Meeting
97
www.hdfgroup.org

98. It doesn’t mean handler failure
because
other tools like
can still access and
visualize data.
July 19, 2012
ESIP 2012 Summer Meeting
98
www.hdfgroup.org

99. MATLAB and IDL will work too.
HDF5 Examples under zoo will work.
• Visit http://hdfeos.org/zoo
• Replace local file access with OPeNDAP
URL access.
 MATLAB: Try new OPeNDAP functions.
• The rest will be same.
July 19, 2012
ESIP 2012 Summer Meeting
99
www.hdfgroup.org

100. Read if you can’t see
http://acdisc.sci.gsfc.nasa.gov/opendap/HDF-EOS5/Aura_OMI_Level3/OMTO3e.003/2012/OMI-Aura_L3OMTO3e_2012m0416_v0032012m0418t020608.he5.ascii?lon[1029:1:1029],lat[513:1:513],_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3[513:1:513][1029:1:1029]
2012m0418t020608.he5 lon,
Dataset: OMI-Aura_L3-OMTO3e_2012m0416_v003-
77.375 lat, 38.375
_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3.lon, 77.375
_HDFEOS_GRIDS_OMI_Column_Amount_O3_Data_Fields_ColumnAmountO3._HDFEOS_GRIDS_O
MI_Column_Amount_O3_Data_Fields_ColumnAmountO3[_HDFEOS_GRIDS_OMI_Column_Amount_
O3.lat=38.375],
O3_Data_Fields_ColumnAmount
July 19, 2012
309.6
ESIP 2012 Summer Meeting
100
www.hdfgroup.org

101. or hear from Siri?
July 19, 2012
ESIP 2012 Summer Meeting
101
www.hdfgroup.org

102. HDF4 OPeNDAP handler update
• Corrected the handling the scale/offset for
MODIS products.
• Added the support for Lambert azimuthal
equal-area (LAMAZ).
• Added the support for Space-oblique
Mercator projection (SOM).
• Other minor bug fixes.
July 19, 2012
ESIP 2012 Summer Meeting
102
www.hdfgroup.org

103. Key updates on HDF4 handler
version
3.9.4
Some MODIS
Scale/offset
LAMAZ
projection
support
SOM
projection
support
July 19, 2012
3.9.3
Wrong
Corrected
No
Yes
No
Yes
ESIP 2012 Summer Meeting
103
www.hdfgroup.org

104. MODIS scale / offset correction
July 19, 2012
ESIP 2012 Summer Meeting
104
www.hdfgroup.org

105. Support for LAMAZ projection
July 19, 2012
ESIP 2012 Summer Meeting
105
www.hdfgroup.org

106. Support for SOM projection
Dataset {
UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512];
Int32 SOMBlockDimU2[SOMBlockDimD1 = 180];
Float32 LatitudeU3[XDimD2 = 128][YDimD3 = 512];
Float32 LongitudeU4[XDimD2 = 128][YDimD3 = 512];
…
}
Dataset {
UInt16 Blue_Radiance_RDQIU1[SOMBlockDimD1 = 180][XDimD2 = 128][YDimD3 = 512];
Int32 SOMBlockDimU2[SOMBlockDimD1 = 180];
Float32 LatitudeU3[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512];
Float32 LongitudeU4[SOMBlockDim = 180][XDimD2 = 128][YDimD3 = 512];
…
}
July 19, 2012
ESIP 2012 Summer Meeting
106
www.hdfgroup.org

107. HDF-EOS2 products supported
GES DISC
• AIRS / MERRA / TOMS
LAADS / LP DAAC / NSIDC
• Many MODIS products
NSIDC
• AMSR_E / NISE / MODIS
LaRC
• MISR / MOPITT / some CERES (TRMM)
July 19, 2012
ESIP 2012 Summer Meeting
107
www.hdfgroup.org

108. HDF4 products supported
GES DISC
• TRMM L1B, L2B Swath
• TRMM L3 Grid 42B & 43B products
OBPG (Ocean Color)
• SeaWiFS / MODIST / MODISA
• CZCS / OCTS
July 19, 2012
ESIP 2012 Summer Meeting
108
www.hdfgroup.org

109. HDF4 products supported
CERES
•
•
•
•
•
•
•
July 19, 2012
CER_AVG
CER_ES4
CER_ISCCP-D2like-Day
CER_ISCCP-D2like-GEO
CER_SRBAVG
CER_SYN
CER_ZAVG
ESIP 2012 Summer Meeting
109
www.hdfgroup.org

110. HDF4 handler limitations
• HDF-EOS2 + additional HDF4 objects
• In some products, some additional HDF4
objects will be ignored.
not listed
• Other HDF4 products
in
previous slides are not tested and may
not be visualized by Java OPeNDAP
clients.
July 19, 2012
ESIP 2012 Summer Meeting
110
www.hdfgroup.org

111. Other limitations
CF conventions Limitation
• _FillValue inside latitude and longitude
Java-NetCDF Limitation
• Java Heap Memory
July 19, 2012
ESIP 2012 Summer Meeting
111
www.hdfgroup.org

112. Summary: we’ve updated handlers
HDFEOS2
HDF4
OPeNDAP
HDF
Handlers
HDF5
HDFEOS5
… and will continue.
July 19, 2012
ESIP 2012 Summer Meeting
112
www.hdfgroup.org

113. GES DISC Uses Handlers
July 19, 2012
ESIP 2012 Summer Meeting
113
www.hdfgroup.org

114. Please try HDF4/HDF5 handlers
New hdf5_handler 2.0.0 is released!
Source codes and RPMs:
http://opendap.org
Demo HDF5/HDF4 servers:
http://hdfeos.org/zoo/hdf4_handler/
http://hdfeos.org/zoo/hdf5_handler/
July 19, 2012
ESIP 2012 Summer Meeting
114
www.hdfgroup.org

115. and give us feedback.
July 19, 2012
ESIP 2012 Summer Meeting
115
www.hdfgroup.org

116. Got HDF?
I’m not a dummy
any more.
July 19, 2012
ESIP 2012 Summer Meeting
116
www.hdfgroup.org

117. Enjoy observing the Earth!
July 19, 2012
ESIP 2012 Summer Meeting
117
www.hdfgroup.org

118. Topics for advanced dummies
That’s it?
How about netCDF-4
interoperability?
July 19, 2012
ESIP 2012 Summer Meeting
118
www.hdfgroup.org

119. Clarification –netCDF format
• netCDF-3 format
• Simple self-describing data format based on
netCDF classic data model
• netCDF-4 format
• Uses HDF5 as a storage layer
• Exploits
• Compression, chunking, parallel-IO
• Group hierarchy, user-defined data types, etc.
• Supports both netCDF enhanced and netCDF
classic data models
• Interoperability with netCDF-4 format in this talk
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
119
www.hdfgroup.org

120. Clarification – netCDF packages
• netCDF software packages
• netCDF-C
• Support both netCDF-3 and netCDF-4 formats
• C++/Fortran Wrappers
• netCDF-Java
• Support both netCDF-3 and netCDF-4 formats
• The implementation of the Common Data Model
• netCDF version 4
• Generally mean the version 4 package of the
netCDF-C library
(Not only support netCDF-4 format)
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
120
www.hdfgroup.org

121. Why netCDF-4
• Big user community
• User-friendly data models
• Tools
• Home-grown and third-party visualization and
analysis tools
- ncdump, ncgen, IDV, Panoply, Ferret etc.
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
121
www.hdfgroup.org

122. NetCDF Classic Model
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
122
www.hdfgroup.org

123. NetCDF enhanced model
Apr. 17-19, 2012
HDF/HDF-EOS Workshop XV
123
www.hdfgroup.org

124. Conversion
Creating a new
netCDF-4 file
from
HDF5 file.
July 19, 2012
ESIP 2012 Summer Meeting
124
www.hdfgroup.org

125. eos52nc4
•
•
•
•
•
converts hdf-eos5 to netcdf-4.
follows netcdf enhanced model.
does not follow CF conventions.
Usage: eos52nc4 test.he5 test.nc
http://hdfeos.org/software/convert_hdfeos5.php
eos5
NETCDF-4
eos52nc4
July 19, 2012
ESIP 2012 Summer Meeting
125
www.hdfgroup.org

126. Augmentation
Making an HDF5 file look like a
netCDF-4 file.
July 19, 2012
ESIP 2012 Summer Meeting
126
www.hdfgroup.org

127. aug_eos5
• augments hdf-eos5 to netcdf-4.
• Both HDF-EOS5 and NetCDF4 can access it.
• Usage: aug_eos5 [-i|-f map] test.h5
• modifies the input file directly.
• does not follow CF conventions.
• http://hdfeos.org/software/aug_hdfeos5.php
eos5
NETCDF-4
aug_eos5
July 19, 2012
ESIP 2012 Summer Meeting
127
www.hdfgroup.org

128. Augmentation Concept
aug_eos5
HDF-EOS5
API
NETCDF-4
API
hdf-eos5
augmentation
netcdf4
hdf5
HDF5 API
July 19, 2012
ESIP 2012 Summer Meeting
128
www.hdfgroup.org

129. Why NetCDF-4 Doesn’t Work
[nLevels][nTimes]
hdf-eos5
July 19, 2012
ESIP 2012 Summer Meeting
NETCDF-4
API
129
www.hdfgroup.org

130. What aug_eos5 Does
[nLevels][nTimes]
hdf-eos5
July 19, 2012
ESIP 2012 Summer Meeting
NETCDF-4
API
130
www.hdfgroup.org

131. Options in aug_eos5
1. Default option
• -i option
2.-f map_file option
July 19, 2012
ESIP 2012 Summer Meeting
131
www.hdfgroup.org

132. Default option in aug_eos5
Grid
• XDim, YDim:
calculated
• Other dims: default fill value
Swath / ZA
• All dims: default fill value
July 19, 2012
ESIP 2012 Summer Meeting
132
www.hdfgroup.org

133. Better -i option in aug_eos5
• Grid
• XDim, YDim: calculated
• Other dims:
1,2,…,n
• Swath / ZA
• All dims:
July 19, 2012
1,2,…,n
ESIP 2012 Summer Meeting
133
www.hdfgroup.org

134. Best -f option in aug_eos5
• Needs an input map file.
• Input file specifies 3
rules:
1. What to be filled in
2. How to fill in
3. Where to get the values
• Most useful and flexible option
July 19, 2012
ESIP 2012 Summer Meeting
134
www.hdfgroup.org

135. aug_eos5 Input File Format
Each line in input file contains
1. How: 0,1,2
2. What: dim name
3. Where: data file or existing variable in file.
4. (Optional) Multiple grid/swath/za case only:
•
July 19, 2012
grid/swath/za name:<name>
ESIP 2012 Summer Meeting
135
www.hdfgroup.org

136. aug_eos5 Input File Example
Copy the data
Read file data.txt
How
0
1
2
What
Where
nTimes
Time
nLevels
data.txt
nCandidates
Fill 1,2,…, n
95% cases – Copy the data
July 19, 2012
ESIP 2012 Summer Meeting
136
www.hdfgroup.org

137. Multiple Swath Case
How
Multiple
What
Where
swath name: BrO
0 nTimes
Time
1 nLevels
data1.txt
2 nCandidates
swath name: BrO column
2 nTimes
1 nLevels
data2.txt
July 19, 2012
ESIP 2012 Summer Meeting
137
www.hdfgroup.org

138. aug_eos5 vs eos52nc4
aug_eos5
hdfeos-5
Yes
No
netcdf-4
July 19, 2012
eos52nc4
Yes
Yes
ESIP 2012 Summer Meeting
138
www.hdfgroup.org

139. HIRDLS/MLS Team Use aug_eos5
So You Can!
July 19, 2012
ESIP 2012 Summer Meeting
139
www.hdfgroup.org

140. The HDF Group
Thank you!
July 19, 2012
ESIP 2012 Summer Meeting
140
www.hdfgroup.org

141. Acknowledgements
This work was supported by Subcontract number
114820 under Raytheon Contract number
NNG10HP02C, funded by the National Aeronautics
and Space Administration (NASA) and by
cooperative agreement number NNX08AO77A from
the NASA. Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the authors and do not necessarily reflect
the views of Raytheon or the National Aeronautics
and Space Administration.
July 19, 2012
ESIP 2012 Summer Meeting
141
www.hdfgroup.org

142. The HDF Group
Questions/comments?
help@hdfgroup.org
July 19, 2012
ESIP 2012 Summer Meeting
142
www.hdfgroup.org