5 Processing pipeline
This chapter describes the processing pipeline from pvol over vp to vpts data.
5.1 European pvol
BALTRAD collects pvol data for Europe. For pvol data to be available in BALTRAD, several conditions need to be met:
- A country should send both reflectivity and radial velocity data to the OPERA data centre, which is called ODYSSEY. While many countries are sending reflectivity data, radial velocities are still unavailable for many countries.
- ODYSSEY should forward these data to the BALTRAD datahub. It is standard policy to do so, but in practice data is not yet forwarded correctly for some radars/countries.
- Currently BALTRAD and ODYSSEY store data at a 15 minute interval, higher resolution data is not yet available.
5.2 European pvol to vp
A server at BALTRAD processes any incoming pvol data with vol2bird. This pipeline is maintained by Günther Haase (SMHI).
The resulting vp files are stored for 2 days on a private FTP server to which we have access. The used vol2bird version is stored in the metadata of the vp files and can differ over time. The latest files are created with vol2bird 0.3.20 (last tested April 11, 2019).
5.3 vp archiving
European vp data are archived daily by a pipeline running on Amazon Web Services (AWS) which transfers vp files from the BALTRAD FTP server to a public S3 bucket. This pipeline is maintained by the Open science lab for biodiversity and its code and documentation are available in this repository. Note that any issues in the pipeline that are not resolved within 2 days (the time vp data are kept on the FTP server) can result in data loss.
The pipeline also updates a coverage file and bundles vp files in monthly zips per radar (e.g. bejab201904.zip). A public website allows easier file navigation of the S3 bucket.
United States vp data are currently not archived. See https://github.com/enram/globam-dmp/issues/19
5.4 vp to vpts
Not yet defined, but likely to be part of the ENRAM pipeline (see 5.3).