transfer_baltrad

Sync files from Baltrad FTP server to the Aloft S3 bucket.

This function connects via SFTP to the BALTRAD server, downloads the available VP files (PVOL gets ignored), from the FTP server and upload the HDF5 file to the Aloft S3 bucket according to the defined folder path name convention. Existing files are ignored.

Designed to be executed via a simple scheduled job like cron or scheduled cloud function. Remark that files disappear after a few days on the BALTRAD server.

Configuration is loaded from the following environmental variables:

  • FTP_HOST: Baltrad FTP host ip address

  • FTP_PORT: Baltrad FTP host port

  • FTP_USERNAME: Baltrad FTP user name

  • FTP_PWD: Baltrad FTP password

  • FTP_DATADIR: Baltrad FTP directory to load data files from

  • DESTINATION_BUCKET: AWS S3 bucket to write data to

  • SNS_TOPIC: AWS SNS topic to report when routine fails

  • AWS_REGION: AWS region where the SNS alerting is defined

  • AWS_PROFILE: AWS profile (mainly useful for local development when working with multiple AWS profiles)

transfer_baltrad [OPTIONS]

vph5_to_vpts

Convert and aggregate HDF5 VP files to daily and monthly VPTS CSV files on S3 bucket

Check the latest modified ODIM HDF5 bird VP profile on the Aloft S3 bucket (as generated by vol2bird and transferred using the vpts.bin.transfer_baltrad CLI routine). Using an s3 inventory bucket, check which HDF5 files were recently added and convert those files from ODIM bird profile to the VPTS CSV format. Finally, upload the generated daily/monthly VPTS files to S3.

When using the path_s3_folder option, the modified date is not used, but a recursive search within the given s3 path is applied to define the daily/monthly files to recreate. E.g. vph5_to_vpts –path-s3-folder uva/hdf5/nldhl/2019 or vph5_to_vpts –path-s3-folder baltrad/hdf5/bejab/2022/10.

Besides, while scanning the S3 inventory to define the files to convert, the CLI routine creates the coverage.csv file and uploads it to the bucket.

Configuration is loaded from the following environmental variables:

- DESTINATION_BUCKET: AWS S3 bucket to read and write data to
- INVENTORY_BUCKET: AWS S3 bucket configured as `s3 inventory bucket
<https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html>`_
for the S3_BUCKET.
- SNS_TOPIC: AWS SNS topic to report when routine fails
- AWS_REGION: AWS region where the SNS alerting is defined
- AWS_PROFILE: AWS profile (mainly useful for local development when
working with multiple AWS profiles)
vph5_to_vpts [OPTIONS]

Options

--modified-days-ago <modified_days_ago>

Range of HDF5 VP files to include, i.e. files modified between now and Nmodified-days-ago. If 0, all HDF5 files in the bucket will be included.

--path-s3-folder <path_s3_folder>

Apply the conversion to VPTS to all files within a S3 sub-folders instead of using the modified date of the files. This option does not use the inventory files.