In EVDC, data are stored as binary files and only selected information from these files are extracted and inserted in the EVDC database, thus making it possible to search for and retrieve specific data files. 

The primary data format used is HDF (Hierarchical Data Format) and EVDC currently supports the HDF4/HDF5  and netCDF releases. Data files are formatted according to the GEOMS metadata standard.

Tools for reading the EVDC files

The simplest way to read a binary .hdf/.nc file is to use the program "ncdump" that is distributed as part of the HDF library. The program will return a simple ASCII dump of the HDF file content. However, the ASCII format is not very user friendly and becomes complicated and cumbersome for large and multidimensional HDF files. A better option is to download the "HDFview" program that is freely distributed through the HDF homepage. 

For EVDC data, the easiest solution is to use special GEOMS compatible tools that provide more functionality and are easy to integrate with existing data processing software. These tools are developed by NASA and made available below here, and through the Aura Validation Data Centre, AVDC.  

For reading the GEOMS files one can use the idlcr8ascii program. 

idlcr8ascii

idlcr8ascii reads GEOMS compliant HDF4/5 and netCDF files and provides output as ASCII formated data and metadata files. The program is written in IDL and also allows users to save file content in session memory in order to integrate the reading routine with other IDL code. The program is available as IDL-code, but also as a ".sav file" to be used with the IDL virtual machine. The latter option allows user without an IDL license to utilise the software. The software can be downloaded here:

Download the idlcr8ascii.pro file.

Download the idlcr8ascii.sav file.

Start IDL in the location where you have stored the idlcr8ascii program and the .hdf file: IDL > idlcr8ascii. If your X-11/Display connection is set properly, the program should be self explainatory. The output is two files; a .meta and a .data file. 

Contact nadirteam for further help on how to run these programs.

To create a hdf file from the .meta and .data files, see 

http://evdc.esa.int/tools/creating-hdf-files/

pycr8ascii

pycr8ascii is similar to idlcr8ascii and reads GEOMS compliant HDF4/5 and netCDF files and provides output as ASCII formated data and metadata files. The program is written in Python3.

To work locally with the code:

Clone the git repository in a Linux/Unix environment

git clone https://git.nilu.no/geoms/geoms_qa

cd geoms_qa

git checkout dev

 

Running:

python3 ./pycr8ascii.py -h

will print a short help message

 

usage: pycr8ascii.py [-h] [-f FILE_NAME] [-o OUT_DIR]

Read a GEOMS file in hdf5 or netCDF4 format and write .meta and .data files

 

optional arguments

  -h, --help            show this help message and exit

  -f FILE_NAME, --file FILE_NAME

                        GEOMS file in netCDF or HDF format (default: None)

  -o OUT_DIR, --outdir OUT_DIR

                        Directory to write output. Same as input directory if not given. (default: None)

Example of run:

./pycr8ascii.py -f test_data/balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.h5 -o test_data/ 

The result will look like:

balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.meta created!
balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.data created!