Submenu

Show or hide subpages

Connecting with EMADDC through SFTP

Access to EMADDC products (output data) is restricted to registered users of the EMADDC processing system only. See for further details the page Data Users.

The latest FTP environment for EMADDC supports the SFTP protocol only. FTP protocol is not used for security reasons as with FTP the non-encrypted ftp credentials and data are transmitted over open internet. Often SFTP is confused with a password-less SSH-public-KEY login onto an (S)FTP server which is not supported by the FTP server of KNMI.

For scripted or automated access to the EMADDC FTP servers with SFTP several options are being offered and it is recommended to use the LFTP client. It is widely available on most Linux (and other) platforms.

FTP Servers

For data exchange, KNMI operates two FTP services:

Production (PRD) system: ftpservice.knmi.nl
Development, Testing and Acceptance (DTA) system: ftppro.knmi.nl

Production delivers EMADDC's operational products which is steady, robust and actively 24/7 monitored. Our DTA systems run new software with fixes or improvements and likely new data streams. Once these changes or new datasets have been througly tested, they make its way to Production. New data stream are only available on DTA initailly!

Folder Structure

Both severs follow the same folder structure and files are named using the same convention. (Note that from the file itself, one cannot determine whether it was retrieved from DTA or PRD).

Data can be uploaded into: /inbound/<_USER_NAME_> (see this page for detailed instructions)

Data can be downloaded from: /outbound/ehs and /outbound/mrar

Where ehs contains derived ADD data and mrar for MRAR ADD data respectively. Be aware that MRAR data is currently not quality controlled.

Types of FTP Accounts

1) EMADDC Data Supplier + Data User

Can be used for data upload (raw data to EMADDC)
Can be used for data download (EMADDC products)
After login the user is directly in directory /inbound/<username> and upload their data
If this user wants to download EMADDC products - change directory to /outbound and download files from subdirectories (ehs,mrar,..)
The user cannot access /inbound/* directories with raw-data of other data-providers, only their own /inbound/<username> directory is accesible.

2) EMADDC Data User

Can ONLY be used for data download (EMADDC products)
After login the user is directly in directory /outbound and can download files from subdirectories (ehs,mrar,..)
The user cannot access /inbound/* directories with raw-data of other data-providers.

3) EMADDC Data Supplier

Can ONLY be used for data upload (raw data to EMADDC)
After login the user is directly in directory /inbound/<username> and upload their data
The user cannot access /inbound/* directories with raw-data of other data-providers and also cannot access /outbound directory with EMADDC products.

Connecting with EMADDC through SFTP

LFTP client: account credentials stored in $HOME/.netrc file

Assure .netrc file has proper rights:
```
chmod 600 ${HOME}/.netrc
```

Add KNMI FTP to .netrc file:

machine ftpservice.knmi.nl     login _USERNAME_    password _password-given-by-KNMI_
machine ftppro.knmi.nl     login _USERNAME_    password _password-given-by-KNMI_

lftp sftp://_USERNAME_@_FTP_.knmi.nl  -e "pwd; cd ./ehs/; ls; bye"

Download a file using:

lftp sftp://_USERNAME_@_FTP_.knmi.nl  -e "cd ./ehs/; get <file>; bye"

LFTP client: password in environment variable

Set your password in an environt variable (e.g., in your bash_profile):
```
LFTP_PASSWORD="--the_password_you_were_given--"
```

lftp sftp://_USERNAME_@_FTP_.knmi.nl  -e "pwd; cd ./ehs/; ls; bye"

Download a file using:

lftp sftp://_USERNAME_@_FTP_.knmi.nl  -e "cd ./ehs/; get <file> ; bye"

LFTP client: password in the command-line (not recommended)

lftp sftp://_USERNAME_:_password@_FTP_.knmi.nl  -e "pwd; cd ./ehs/; ls; bye"

Download a file using:

lftp sftp://_USERNAME:_password_@_FTP_.knmi.nl  -e "cd ./ehs/; get <file> ; bye"

SFTP client: password in the SSHPASS envinment variable (sshpass needed)

SFTP client does not use the .netrc file but the LFTP client is smart enough to use it. If you (for whatever reason) need to use the SFTP client consider the following:

export SSHPASS=..ftp-password..
sshpass -e sftp _USERNAME_@_FTP_.knmi.nl
- or -
export SSHPASS=..ftp-password..
sshpass -e sftp -oBatchMode=no -b - _USERNAME_@_FTP_.knmi.nl << EOFTP
cd 
ls 
get  <file>
bye
EOFTP

SFTP client: password entered manually, interactive session

sftp _USERNAME_@_FTP_.knmi.nl
pwd
cd ./ehs/;
ls
get  <file>
bye

SFTP client: UPLOAD files using temporary filename

# If needed: yum install sshpass 
export SSHPASS=..ftp-password..
export USER=ftp-user
export HOST=ftp-host
for f in $(find ./ -type f); do
    export TMPEXT=".tmp.$$"
    export DSTF=$RDIR/$(basename $f)
    echo "Uploading $f"
    sshpass -e sftp $USER@$HOST << EOFTP
put $f ${DSTF}${TMPEXT}
rename ${DSTF}${TMPEXT} ${DSTF}
EOFTP 
done

LFTP client: UPLOAD files using temporary filename

# ${HOME}/.netrc .. contains credentials; LFTP can use it

for f in $(find ./ -type f); do
    t="${f%.*}".tmp
    lftp  sftp://_USERNAME_@_FTP_.knmi.nl -c "put $f -o $t;mv $t $f;bye"
done 

# For LFTP versions > 4.6:

for f in $(find ./ -type f); do
    t="${f%.*}".tmp
    lftp  sftp://_USERNAME_@_FTP_.knmi.nl -c "set xfer:use-temp-file yes; set xfer:temp-file-name *.lftp;
    put $f; bye"
done

Important tips for SSH configuration

Since KNMI does NOT facilitate the use of ssh-(public)-keys during the sFTP server login authentification you should configure your ftp-client machine not to use the ssh-public-keys in combination with the 2 FTP servers:
FTPPRO.knmi.nl and FTPSERVICE.knmi.nl

The sftp authentification for the EMADDC FTP accounts is done using a password only.
For scripting/coding purposes the password is not entered manualy but provided via an environment variable or configuration file, depending on FTP client.

If you are using LFTP client the password can be stored into ~/.netrc file.
If you prefer to user SFTP client you have to install sshpass to provide the password to the sftp client.
We recommend LFTP client.

Now how you can tell your ssh-(sftp-) client NOT to use ssh-public-keys and to use ONLY password authentification.

One-time work-around looks like:
 
sftp -o PreferredAuthentications=password -o PubkeyAuthentication=no <_USER_NAME_>@ftppro.knmi.nl
sftp -o PreferredAuthentications=password -o PubkeyAuthentication=no <_USER_NAME_>@ftpservice.knmi.nl

As a permanent solution you can edit ~/.ssh/config:
 
Host ftppro.knmi.nl
  User <_USER_NAME_>
  PreferredAuthentications password
  PubkeyAuthentication no
 
Host ftpservice.knmi.nl
  User <_USER_NAME_>
  PreferredAuthentications password
  PubkeyAuthentication no
 
sftp -v <_USER_NAME_>@ftppro.knmi.nl
sftp -v <_USER_NAME_>@ftpservice.knmi.nl
 
(Test with options -v , -vv, -vvv for detailed verbosity.)

Substitue <_USER_NAME_> with your emaddc ftp-account.

SFTP log-in issue due to incorrectly used ssh-keys

If you need to have multiple ssh-keys in the ~/.ssh/ directory it is wise to instruct the ssh-(sftp-) client
WHICH ssh-keys should be used with WHICH ssh-(sftp-) server.
If you don't instruct SSH which ssh-keys should be used with which server then your ssh client will try every ssh-key in the directory.

The (s)FTP servers of KNMI have a limit login-attempts. That includes failed ssh-keys trials as well.

What can happen that even a correct password (manually) entered will not be accepted and the server would NOT allow to log in.

There are legitimate reasons to use multiple separate ssh-keys for different ssh-(sftp-) servers.
So, if you have multiple ssh-keys in ~/.ssh/ directory it makes no sense to let the SSH-client to try incorrect (unregistered on server) ssh-keys on wrong server.

Therefore you can edit explicitly ~/.ssh/config and be specific with respect to the ssh-keys.

Host <ftp-server-1>
  IdentityFile ~/.ssh/id_rsa_for_ftp-server-1
  User <ftp-user-name-1>
 
Host <ftp-server-2>
  IdentityFile ~/.ssh/id_rsa_for_ftp-server-2
  User <ftp-user-name-2>

Example usage of EMADDC SFTP servers

----------------------------------------
1) SFTP Access to EMADDC (DTA) products:
----------------------------------------
1a) "fast/5min" EMADDC (DTA) products:
--------------------------------------
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_20230419*; bye"      |tail -4
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4
------------------------------------------
1b) "regular/15min" EMADDC (DTA) products:
------------------------------------------
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt ehs/EMADDC_KNMI_20230419*; bye"       |tail -4
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt mrar/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4

Where:  
    SFTP_EMADDC_ACCOUNT=---emaddc-data-user-account-you-were-given---
    20230419 .. date

----------------------------------------
2) SFTP Access to EMADDC (PRD) products:
----------------------------------------
2a) "fast/5min" EMADDC (PRD) products:
--------------------------------------
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_20230419*; bye"      |tail -4
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4
------------------------------------------
2b) "regular/15min" EMADDC (PRD) products:
------------------------------------------
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt ehs/EMADDC_KNMI_20230419*; bye"       |tail -4
lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt mrar/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4

----------------------------------------
****     ....... EXAMPLES .....     ****
1) SFTP Access to EMADDC (DTA) products:
----------------------------------------
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_2*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***  2818320 apr 19 09:48 fast/EMADDC_KNMI_20230419_0935_20230419_0939.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  2407654 apr 19 09:48 fast/EMADDC_KNMI_20230419_0935_20230419_0939.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***  4305610 apr 19 09:49 fast/EMADDC_KNMI_20230419_0925_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  3630657 apr 19 09:49 fast/EMADDC_KNMI_20230419_0925_20230419_0929.bufr.gz
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***  4305610 apr 19 09:49 fast/EMADDC_KNMI_20230419_0925_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  3630657 apr 19 09:49 fast/EMADDC_KNMI_20230419_0925_20230419_0929.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***  4304499 apr 19 09:53 fast/EMADDC_KNMI_20230419_0930_20230419_0934.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  3632640 apr 19 09:53 fast/EMADDC_KNMI_20230419_0930_20230419_0934.bufr.gz
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt ehs/EMADDC_KNMI_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-*** 10912699 apr 19 09:32 ehs/EMADDC_KNMI_20230419_0900_20230419_0914.bufr.gz
-rw-r--r-- emaddc-*** emaddc-*** 11478700 apr 19 09:47 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.txt.gz
-rw-r--r-- emaddc-*** emaddc-*** 12425379 apr 19 09:47 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-*** 10501610 apr 19 09:47 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.bufr.gz
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftppro.knmi.nl -e "cls -alrt mrar/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***      683 apr 19 09:32 mrar/EMADDC_KNMI_MRAR_20230419_0900_20230419_0914.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***      444 apr 19 09:47 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-***      381 apr 19 09:47 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.txt.gz
-rw-r--r-- emaddc-*** emaddc-***      407 apr 19 09:47 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.bufr.gz

----------------------------------------
****     ....... EXAMPLES .....     ****
2) SFTP Access to EMADDC (PRD) products:
----------------------------------------
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***  4584209 apr 19 09:54 fast/EMADDC_KNMI_20230419_0935_20230419_0939.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  3893572 apr 19 09:54 fast/EMADDC_KNMI_20230419_0935_20230419_0939.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***  4278868 apr 19 09:54 fast/EMADDC_KNMI_20230419_0930_20230419_0934.csv.gz
-rw-r--r-- emaddc-*** emaddc-***  3611559 apr 19 09:54 fast/EMADDC_KNMI_20230419_0930_20230419_0934.bufr.gz
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt fast/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***     1790 apr 19 09:54 fast/EMADDC_KNMI_MRAR_20230419_0935_20230419_0939.csv.gz
-rw-r--r-- emaddc-*** emaddc-***     1223 apr 19 09:54 fast/EMADDC_KNMI_MRAR_20230419_0935_20230419_0939.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***      309 apr 19 09:54 fast/EMADDC_KNMI_MRAR_20230419_0930_20230419_0934.csv.gz
-rw-r--r-- emaddc-*** emaddc-***      276 apr 19 09:54 fast/EMADDC_KNMI_MRAR_20230419_0930_20230419_0934.bufr.gz

> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt ehs/EMADDC_KNMI_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-*** 11807943 apr 19 09:33 ehs/EMADDC_KNMI_20230419_0900_20230419_0914.txt.gz
-rw-r--r-- emaddc-*** emaddc-*** 11640617 apr 19 09:52 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.txt.gz
-rw-r--r-- emaddc-*** emaddc-*** 12604603 apr 19 09:52 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-*** 10638377 apr 19 09:52 ehs/EMADDC_KNMI_20230419_0915_20230419_0929.bufr.gz
> lftp sftp://${SFTP_EMADDC_ACCOUNT}@ftpservice.knmi.nl -e "cls -alrt mrar/EMADDC_KNMI_MRAR_20230419*; bye" |tail -4
-rw-r--r-- emaddc-*** emaddc-***      683 apr 19 09:33 mrar/EMADDC_KNMI_MRAR_20230419_0900_20230419_0914.bufr.gz
-rw-r--r-- emaddc-*** emaddc-***      381 apr 19 09:52 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.txt.gz
-rw-r--r-- emaddc-*** emaddc-***      444 apr 19 09:52 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.csv.gz
-rw-r--r-- emaddc-*** emaddc-***      407 apr 19 09:52 mrar/EMADDC_KNMI_MRAR_20230419_0915_20230419_0929.bufr.gz

Connecting with EMADDC through SFTP

FTP Servers

Folder Structure

Types of FTP Accounts

1) EMADDC Data Supplier + Data User

2) EMADDC Data User

3) EMADDC Data Supplier

Connecting with EMADDC through SFTP

LFTP client: account credentials stored in $HOME/.netrc file

LFTP client: password in environment variable

LFTP client: password in the command-line (not recommended)

SFTP client: password in the SSHPASS envinment variable (sshpass needed)

SFTP client: password entered manually, interactive session

SFTP client: UPLOAD files using temporary filename

LFTP client: UPLOAD files using temporary filename

Important tips for SSH configuration

SFTP log-in issue due to incorrectly used ssh-keys

Example usage of EMADDC SFTP servers

Share