In 2021, EMADDC has developed a new processing system with several improvements from the previous system including improved corrections and data handling. The new system uses a large database for all data storage instead of operating file-based and per data-stream file handling. The database allows to blend all streams together and hence benefit from the increased data volume to create better corrections but also to mark duplicate observations due to geographical overlap between data streams.
In order to facilitate the increase in data volume the EMADDC team has developed new output data formats and decided to terminate the NetCDF format that was not in use by any of the users. As a result only raw text files (ASCII) and the WMO BUFR AMDAR version 7 format are provided to data users. Data will become available every 15 minutes where in the future a higher update rate is foreseen.
As per Q3 2022, EMADDC also outputs files in 5-minute windows where these files are processed 13 minutes and 23 minutes past the start of such window. These two delays have been chosen such that roughly 70% and 95% of all data is contained in a file respectively. EMADDC refers to these files as "fast" files. Fast files are found in the "fast" folder on EMADDC's FTP servers.
Note that all times references by EMADDC are in UTC!!
Since September 2024, EMADDC also provides the EMADDC Met Office Global data in NetCDF (v4) format. The regular EMADDC data is not available as NetCDF.
Below a description of the ASCII, BUFR and NetCDF formats used by EMADDC.
The ASCII file format is comma seperated and has extension csv and when distributed are compressed using gzip. Files contain the following parameters obs_id, date, time, Mid, lat, long, fl, wspd, wdir, temp, phase, roll angle, source, sic, wl_flag and qc_flag. A description of these parameters is found in the table below. Currently, EMADDC only provides output observations that have passed quality control and hence the qc_flag will always be 0 but this will change in the future.
To distinguish between derived Mode-S EHS observations and MRAR observations, two files are provided:
EMADDC_KNMI_<YMD>_<HM>_<YMD>_<YM>.csv.gz
(e.g., EMADDC_KNMI_20201204_1315_20201204_1329.csv.gz)
EMADDC_KNMI_MRAR_<YMD>_<HM>_<YMD>_<YM>.csv.gz
Both files are found in different folders, being ehs and mrar respectively on FTP. In case no MRAR file is present for a certain timestamp, this is likely the result of lack of data and not a processing error, unless the corresponding ehs file is also missing.
Important notice: the MRAR data output is not quality controlled and provides the wind and temperature as provided by the aircraft and as received by receivers.
The first time indicates the time of the first observation in the file and the second time indicates the time of last observation. Observations are sorted dy date and time (dtg).
In certain cases, for example due to quality control, certain parameters are invalid and hence are missing.
Also new in the ASCII files is a three line header (lines starting with a #) to indicate the software version used to produce the file, the correction method being used and a row indicating the parameter in each column. The latter should help processing the proper columns and to avoid mismatches:
- Software Version: Major Release.Minor Release._Machine Letter e.g, 2.2.135_p where 2.2 is the major version, 135 the minor version and p the server or environment system (Production). EMADDC major versions are always of the form x.y, and are only updated upon quality or other major changes.
- Process Time: Time at which the file was processed on EMADDC system (in UTC)
- Sequence Number: File sequence, 0 for first processed file and value is incremented upon subsequent runs (e.g., due to delayed input files)
- CorMethod: mhdgTasPhase (or any other string description)
- Offset: Offset value for each obs id. Actual DB obs_id is equal to offset + obs_id per observation.
- Columns header: obs_id,date,time,Mid,lat,lon,fl,wspd,wdir,temp,phase,ra,source,sic,wl_flag,qc_flag (this line no longer has a # prepended)
|
Description
|
Parameter
|
Units
|
Source
|
Remark
|
| OBSID |
obs_id |
|
EMADDC |
Add offset to get DB value
|
| Date |
date |
|
YYYYMMDD |
Radar / GPS Receiver |
| Time |
time |
|
HHMMSS |
Radar / GPS Receiver |
|
Anonymous Aircraft Mode-S ID ('M'+24-bit)
|
Mid
|
Mxxxxxx
|
BDS 2.X / Radar
|
|
|
Latitude
|
lat
|
deg (DDD.dddd)
|
Radar / ADS-B
|
|
|
Longitude
|
lon
|
deg (DDD.dddd)
|
Radar / ADS-B
|
|
|
Flight Level
|
fl
|
FL
|
Mode-C/S
|
|
|
Wind Speed
|
wspd
|
kts
|
EHS: Derived MRAR: BDS 4.4
|
(Caution: BUFR output of wspd is in m/s) |
|
Wind Direction
|
wdir
|
deg
|
EHS: Derived MRAR: BDS 4.4 |
|
|
Temperature
|
temp
|
Kelvin
|
EHS: Derived MRAR: BDS 4.4 |
|
|
Phase
|
phase
|
3, 5, 6
|
Calculated
|
See BUFR detailed phase of flight definition below
|
|
Roll Angle
|
ra
|
deg (- left turn)
|
BDS 5.0
|
|
|
Source
|
source
|
-
|
Filename
|
|
|
SIC (Station Identification Code)
|
sic
|
-
|
station ID
|
|
|
Whitelisted
|
wl_flag
|
-
|
Calculated
|
0 = wind and temp whitelisted
1 = wind not whitelisted temp whitelisted
2 = wind whitelisted temp not whitelisted
3 = wind not whitelisted temp not whitelisted
4 = whitelisting not performed
|
|
QC Flag
|
qc_flag
|
-
|
Calculated
|
For now qc_flag = 0
|
Example output fiile: EHS / MRAR
A dedicated ADD BUFR format is currently not available so the EMADDC team focused on providing the same parameters of the ASCII file in the BUFR file by expanding descriptors to the WMO AMDAR v7 format. The new EMADDC BUFR format contains the following parameters dtg, Mid, lat, long, fl, wspd, wdir, temp, phase, similar to version 1.6 but extended with descriptors to add roll angle quality, source, sic, wl_flag and qc_flag. A description of these parameters is found in the table below including the descriptors used to add the data to the files. Currently EMADDC provides only output observations that have passed quality control and hence the qc_flag will always be 0 but this will change in the future.
To distinguish between derived Mode-S EHS observations and MRAR observations, two files are provided:
EMADDC_KNMI_<YMD>_<HM>_<YMD>_<YM>.bufr.gz
(e.g., EMADDC_KNMI_20201204_1315_20201204_1329.bufr.gz)
EMADDC_KNMI_MRAR_<YMD>_<HM>_<YMD>_<YM>.bufr.gz
When distributed onto FTP, files are compressed using gzip to reduce bandwith usage.
Important notice: the MRAR data output is not quality controlled and provides the wind and temperature as provided by the aircraft and as received by the receivers. (EHS and MRAR files are found in different folders in the outbound folder on FTP).
Date and time in the filename follow the same convention as the ASCII files. The first time indicates the time of the first observation in the file and the second time indicates the time of last observation. Observations are sorted dy date and time (dtg).
AMRAR files use 148 for the dataSubCategory and the non-MRAR files use 147.
To provide the same header information a few more descriptors have been added per subset/observation. This is necessary as unfortunately the BUFR header does not contain descriptors for this purpose . To reduce file size, EMADDC now uses subsets which are currently maximized at 100 but will increase in the future to reduce file size even further.
The table below contains all descriptors in the WMO AMDAR v7 format that data is written to and the added descriptors as an extension. Hence, the used descriptors are 311010 extended with 25061, 1015, 1022, 1065, 33002. For code tables, see https://library.wmo.int/doc_num.php?explnum_id=4120 and https://confluence.ecmwf.int/display/ECC/WMO%3D33+element+table and is based on Master Table Version 33.
| |
|
|
Description
|
Parameter
|
Units
|
Remark
|
|
0
|
01
|
008
|
Aircraft registration number or other identification
|
Mid
|
|
anonymous mode-s 24-bit id (same Mid assigned to all obs from same aircraft)
|
|
3
|
01
|
011
|
Year, month, day
|
dtg
|
|
|
|
3
|
01
|
013
|
Hour, minute, second
|
dtg
|
|
|
|
3
|
01
|
021
|
Latitude/longitude (high accuracy)
|
lat, lon
|
deg
|
|
|
0
|
07
|
010
|
Flight Level (Pressure Altitude)
|
fl
|
m
|
|
|
0
|
08
|
009
|
Detailed phase of flight
|
phase
|
code
|
intentionally UNSTEADY is not supported by EMADDC as this would remove ASC/DESC information
Steadiness can be read from Aircraft roll angle quality instead
|
|
0
|
11
|
001
|
Wind direction
|
wdir
|
deg
|
wl_flag in associatedField (0, WL; 1, no-WL)
|
|
0
|
11
|
002
|
Wind speed
|
wspd
|
m/s
|
wl_flag in associatedField (0, WL; 1, no-WL, Caution: ASCII output of wspd is in knots)
|
|
0
|
02
|
064
|
Aircraft roll angle quality
|
ra
|
code
|
abs(ra) > 90 → 5, abs(ra) > 5 → 1, else 0
|
|
0
|
12
|
101
|
Temperature/air temperature
|
temp
|
K
|
wl_flag in associatedField (0, WL; 1, no-WL)
|
|
Extension:
|
|
0
|
01
|
015
|
Station or Site Name
|
sic
|
|
|
|
0
|
01
|
022
|
Name of Feature
|
correction
|
|
provides the name of corrections applied to produce the observations
|
|
0
|
01
|
065
|
ICAO Region Identifier
|
source
|
|
|
|
0
|
33
|
002
|
Quality Information
|
qc_flag
|
|
For now; qc_flag = 0
|
|
0
|
25
|
061
|
Software Version Number
|
version
|
|
repeated for each obs and only 12 chars available, e.g, 2.2.135_p where 2.2 is the Major Version, 135 refers to the Minor Version and p the machine/environment name
|
Example output file: EHS / MRAR
The NetCDF file format is formatted according to the netCDF4 format and has extension nc4 and when distributed is compressed using gzip. Files contain the following parameters obs_id, date, time, Mid, lat, long, fl, wspd, wdir, temp, phase, roll angle, source, sic, wl_flag and qc_flag. A description of these parameters is found in the table below. Currently, EMADDC only provides output observations that have passed quality control and hence the qc_flag will always be 0 but this might change in the future.
To distinguish between derived Mode-S EHS observations and MRAR observations, two files are provided:
EMADDC_KNMI_<YMD>_<HM>_<YMD>_<YM>.nc4.gz
(e.g., EMADDC_KNMI_20201204_1315_20201204_1329.csv.gz)
EMADDC_KNMI_MRAR_<YMD>_<HM>_<YMD>_<YM>.nc4.gz
Currently, EMADDC Met Office Global does not provide MRAR data yet. Hence, there is no MRAR data available on KDP yet.
Important notice: the MRAR data output is not quality controlled and provides the wind and temperature as provided by the aircraft and as received by receivers.
The files are named to reflect the time of the first observation and the time of the last observation. Observations within the files are organized by date and time (DTG). For some observations in a file, for example due to quality control, certain parameters are invalid and hence are missing (NaN).
Th NetCDF file contains meta-data as Global Attributes and contains the following information:
- Software Version: Major Release.Minor Release._Machine Letter e.g, 2.2.135_p where 2.2 is the major version, 135 the minor version and p the server or environment system (Production). EMADDC major versions are always of the form x.y, and are only updated upon quality or other major changes.
- Creation Date: Date and time at which the file was processed on EMADDC system (in UTC)
- Sequence Number: File sequence, 0 for first processed file and value is incremented upon subsequent runs (e.g., due to delayed input files)
- Title and Author information
|
Description
|
Parameter
|
Units
|
Source
|
Remark
|
| OBSID |
obs_id |
|
EMADDC |
|
| Date |
date |
|
YYYYMMDD |
Radar / GPS Receiver |
| Time |
time |
|
HHMMSS |
Radar / GPS Receiver |
|
Anonymous Aircraft Mode-S ID ('M'+24-bit)
|
Mid
|
Mxxxxxx
|
BDS 2.X / Radar
|
|
|
Latitude
|
lat
|
deg (DDD.dddd)
|
Radar / ADS-B
|
|
|
Longitude
|
lon
|
deg (DDD.dddd)
|
Radar / ADS-B
|
|
|
Flight Level
|
fl
|
m
|
Mode-C/S
|
Caution; FL is provided in meters |
|
Wind Speed
|
wspd
|
m/s
|
EHS: Derived MRAR: BDS 4.4
|
(Caution: output of wspd is in m/s) |
|
Wind Direction
|
wdir
|
deg
|
EHS: Derived MRAR: BDS 4.4 |
|
|
Temperature
|
temp
|
Kelvin
|
EHS: Derived MRAR: BDS 4.4 |
|
|
Phase
|
phase
|
3, 5, 6
|
Calculated
|
See BUFR detailed phase of flight definition below
|
|
Roll Angle
|
ra
|
deg (- left turn)
|
BDS 5.0
|
|
|
Source
|
source
|
-
|
Filename
|
|
|
SIC (Station Identification Code)
|
sic
|
-
|
station ID
|
|
|
Whitelisted
|
wl_flag
|
-
|
Calculated
|
0 = wind and temp whitelisted
1 = wind not whitelisted temp whitelisted
2 = wind whitelisted temp not whitelisted
3 = wind not whitelisted temp not whitelisted
4 = whitelisting not performed
|
|
QC Flag
|
qc_flag
|
-
|
Calculated
|
For now qc_flag = 0
|