Uploading data to EMADDC through SFTP

EMADDC offers partners that supply surveillance data the option to upload data for processing by the SFTP servers of KNMI EMADDC. How to use the EMADDC SFTP accounts appropriately is described here. Note that EMADDC only supports the SFTP protocol.

Besides the data push option (project partners uploading data) EMADDC can also be configured to download from FTP servers of the project partners (data pull). Please contact the EMADDC team if this is preferred.

Note that EMADDC is expected to have a NewPENS connection in 2024 to retrieve data from Air Traffic Service Providers in (near) real-time.

For supplying surveillance data to EMADDC please follow the steps below and take notice of the recommendations and instructions on the format:

EMADDC requires to check a sample file (via email) for correctness, validity and added value determination via the emaddc-test service. Please send an email to the EMADDC team emaddc@knmi.nl for further instructions.
Once the previous step is succesfull a Data Sharing Agreement will have to be signed before continuing with step 2. The EMADDC team will provide a dedicated FTP account on the DTA FTP server. EMADDC will then start processing the test data on the EMADDC DTA stream and will be available for test purposes via ftppro.knmi.nl
The final step will be to include the data into the operational (PRD) processing which implies uploading to the operational FTP server using the same FTP account but onto the production FTP server (ftpservice.knmi.nl). At this stage the data will be included in the operational EMADDC output.

Data files should be uploaded into a given sub-directory (*) provided by the EMADDC team but only to either the DTA or the PRD environment. If the data is NEW and has NOT yet been tested it can only be uploaded to the DTA (development-test-acceptation) environment: ftppro.knmi.nl.

After appropriate and succesfull testing of the data in the EMADDC DTA environment, the data supplier will be requested to stop uploading to the DTA (via FTPPRO) and switch to uploading of the data to the PRD (production) environment via ftpservice.knmi.nl.

EMADDC FTP Accounts

To ease use of both servers, the same SFTP credentials can be used for DTA and PRD and on earch server though, the same credentials can be used for both data upload (supplier) and download (user). It is not possible to change the password but a new password can be provided by the EMADDC team on request.

After login using an EMADDC data supplier SFTP account, data can be uploaded directly into the directory ("./" == " /inbound/<_USER_NAME_>"). There is no need to change the directory after login.

If a data supplier wishes to be able to download the EMADDC output the same SFTP credentials/account can be configured such that access is also provided to the /outbound/ directory contaning all EMADDC output. This can be requested by sending an email to the EMADDC team. Users having access to both inbound and outbound are considered EMADDC data supplier + data user type accounts. Please use the ftp-commando: "cd /outbound/" to get to the EMADDC output. And the command: "cd /inbound/<_USER_NAME_>" to return to the upload directory.

After login using an EMADDC data-user SFTP account the user enters the directory ("./" == " /outbound/") and this allows to download the EMADDC output only. It is not possible to enter other SFTP directories or to read the raw-input data provided by other EMADDC project partners.

Connecting with EMADDC FTP

How to use the KNMI EMADDC SFTP environments is described on this page and instructions for uploading specifically can be found on the bottom of this page.

The EMADDC team prefers data suppliers to compress receiver data before uploading to the EMADDC SFTP server. It is suggested to upload data with the *.gz (gzip/gunzip) extension.

The EMADDC system scans for new data files periodically in the /inbound/* directories. Note that it is important to upload files with a temporary extension (*.tmp). After uploading has finished the uploaded file shall be renamed to the final filename and extension (*.gz) to signal a successful and completed file upload. Not using the temporary extension will result in corrupted input data as EMADDC reads unfinished or uncompleted files that can not be processed by EMADDC. Once the new data files with given (final) extension appear they are ingested into the EMADDC processing and immediately removed from the SFTP server.

Data files shall be uploaded only once!

Please NEVER use an "rsync" of the last couple of hours (or days) to the KNMI EMADDC SFTP servers.

Please monitor the files that are uploaded to the EMADDC SFTP servers and refrain from syncing data continuously. The inbound directory on the SFTP server where the data is uploaded will contain the data files for a short period of time only and are removed after succesfull transfer to the EMADDC processing systems.

Please upload only data files that are known (registered) to the EMADDC system with known file-prefix and file extension as agreed upon with the EMADDC team. Please do not upload other files to the KNMI EMADDC SFTP servers.

Please do NOT upload outdated data files and only upload a file once (no mirror or rsync). Only If data transmission has halted or a breakdown in distribution of the data supplier for several days has occurred, is it allowed to upload upto 24 hours of older data file. Only the latest data upto approximately 1 hour old will be processed by EMADDC while the older data will be archived and could eventually be used in reprocessing studies.

Below some examples on how to implement the uploading of the files with temporary filenames.

SFTP client: UPLOAD files using temporary filename

# If needed: yum install sshpass 
export SSHPASS=..ftp-password..
export USER=ftp-user
export HOST=ftp-host
for f in $(find ./ -type f); do
    export TMPEXT=".tmp.$$"
    export DSTF=$RDIR/$(basename $f)
    echo "Uploading $f"
    sshpass -e sftp $USER@$HOST << EOFTP
put $f ${DSTF}${TMPEXT}
rename ${DSTF}${TMPEXT} ${DSTF}
EOFTP 
done

LFTP client: UPLOAD files using temporary filename

# ${HOME}/.netrc .. contains credentials; LFTP can use it

for f in $(find ./ -type f); do
    t="${f%.*}".tmp
    lftp  sftp://_USERNAME_@_FTP_.knmi.nl -c "put $f -o $t;mv $t $f;bye"
done 

# For LFTP versions > 4.6:

for f in $(find ./ -type f); do
    lftp  sftp://_USERNAME_@_FTP_.knmi.nl -c "set xfer:use-temp-file yes; set xfer:temp-file-name *.lftp;
    put $f; bye"
done

Important tips for SSH (SFTP) configuration

Since we (as EMADDC) do NOT facilitate usage of ssh-(public)-keys during the sFTP server login authentification you should configure your ftp-client machine not to use the ssh-public-keys in combination with the 2 FTP servers:
FTPPRO.knmi.nl and FTPSERVICE.knmi.nl

The sftp authentification for the EMADDC FTP accounts is done using a password only.
For scripting/coding purposes the password is not entered manualy but provided via an environment variable or configuration file, depending on FTP client.

If you are using LFTP client the password can be stored into ~/.netrc file.
If you prefer to user SFTP client you have to install sshpass to provide the password to the sftp client.
We recommend LFTP client.

Now how you can tell your ssh-(sftp-) client NOT to use ssh-public-keys and to use ONLY password authentification.

One-time work-around looks like:
 
sftp -o PreferredAuthentications=password -o PubkeyAuthentication=no <_USER_NAME_>@ftppro.knmi.nl
sftp -o PreferredAuthentications=password -o PubkeyAuthentication=no <_USER_NAME_>@ftpservice.knmi.nl

As a permanent solution you can edit ~/.ssh/config:
 
Host ftppro.knmi.nl
  User <_USER_NAME_>
  PreferredAuthentications password
  PubkeyAuthentication no
 
Host ftpservice.knmi.nl
  User <_USER_NAME_>
  PreferredAuthentications password
  PubkeyAuthentication no
 
sftp -v <_USER_NAME_>@ftppro.knmi.nl
sftp -v <_USER_NAME_>@ftpservice.knmi.nl
 
(Test with options -v , -vv, -vvv for detailed verbosity.)

Substitue <_USER_NAME_> with your emaddc ftp-account.

SFTP log-in issue due to incorrectly used ssh-keys

If you need to have multiple ssh-keys in the ~/.ssh/ directory it is wise to instruct the ssh-(sftp-) client
WHICH ssh-keys should be used with WHICH ssh-(sftp-) server.
If you don't instruct SSH which ssh-keys should be used with which server then your ssh client will try every ssh-key in the directory.

The (s)FTP servers of KNMI have a limit login-attempts. That includes failed ssh-keys trials as well.

What can happen that even a correct password (manually) entered will not be accepted and the server would NOT allow to log in.

There are legitimate reasons to use multiple separate ssh-keys for different ssh-(sftp-) servers.
So, if you have multiple ssh-keys in ~/.ssh/ directory it makes no sense to let the SSH-client to try incorrect (unregistered on server) ssh-keys on wrong server.

Therefore you can edit explicitly ~/.ssh/config and be specific with respect to the ssh-keys.

Host <ftp-server-1>
  IdentityFile ~/.ssh/id_rsa_for_ftp-server-1
  User <ftp-user-name-1>
 
Host <ftp-server-2>
  IdentityFile ~/.ssh/id_rsa_for_ftp-server-2
  User <ftp-user-name-2>