Which are theDHS Features?
The “Data Hub Software” (DHS) suite is a platform designed to support data discovery, distribution, processing and dataflow control for Earth Observation data.
DHS is highly reliable, scalable and extendible providing all functions for the set up and operations of an EO data retrieval and distribution centre as a Collaborative Ground Segment node. DHS includes frameworks dedicated to data ingestion, search and processing allowing multi-mission data management and wide system extendibility.
The DHS suite is composed by four software applications supported by a set of state of art open-source solutions. It can be automatically deployed entirely or partially to benefit from all or part of its functions.
Showcase
Dynamic dataflowvisualization andcontrolwithDAFNE
Interactivedata discovery anddownloadwithDHuS
Interactivedata gathering anddistributionwithGSS Suite
Modernweb-based productdiscoverywithCOPSI
Extendibledata processingworkflowswithTransformationFramework
Identify and associatereal word phenomena torelevant Sentinel datawithSemanticFramework
Harmonised installationmechanismwithDHS Suite Easy Deploy
Data Flow Network Environment
component (DAFNE)
The Dataflow Network Environment (DAFNE) component provides users with a strong dataflow visualization and control solution. It is designed as a powerful plug-and-play visualization tool exposing a standard HTTP interface for harvesting data to be visualized. The aim of DAFNE is to give an intuitive, modern interface for managing the dataflow within the DHR and Collaborative GS network, making easier routine operations.
Monitor and visualize on a 2D map retrieved and distributed data among data centres.
Monitor and visualize rolling policy configured on configured data centre.
Monitor and visualize archive status and entity in terms of published product. Possibility to compare archives.
Possibility to extract statistics, choosing between different chart types and exporting images and/or results in CSV format.
Measurement of their service performance, with regards to the service availability and publication timeliness.
User management based on an external Identity Access Management (IAM) system.
Data Hub Software (DHuS)
Data discovery and distribution via OData API, Opensearch API and GUI.
Extendible ingestion function managing EO data indexing and publication.
Intelligent synchronization function choosing the most efficient data source among many.
“One-click” embedded or scaled externalised deployment.
Collection and reporting of metrics about data exchange operations.
User Management embedded or delegated to an external Identity Access Management (IAM).
The Data Hub Software (DHuS) is open-source software developed to the purpose of supporting the ESA Copernicus data access.
The DHuS provides a simple web interface to allow interactive data discovery and download, and a powerful Application Programming Interface (API) that allows users to access the data via computer programs and scripts thereby automating/integrating the download within their workflow.
For the Collaborative and DHR community the software has been improved in order to increase robustness and efficiency in the data retrieving.
GSS Suite
The GSS Suite, or Gael Store Service, is a set of components aimed at retrieving and exposing products stored via the CSC OData API. With respect to the DHuS, the GSS is intended to offer an alternative way to retrieve and distribute data, able to interface with different kind of data sources, like DHuS archive but also CSC Data sources like the Copernicus Data Space Ecosystem or the other GSS softwares. It is a multi-modular solution that offers a wider scalability.
The GSS-Ingest component based on the producer-consumer paradigm.
Modularity, wider scalability, easier maintenance.
User management based on an external Identity Access Management (IAM) system.
New data gathering mechanism inheriting Datastore and metadata store technologies, based on producer and consumer paradigm.
Extendible ingestion function managing EO data indexing and publication.
New CSC OData API to expose products on catalogue, allowing also geographic queries and queries by attributes.
Different storage supported (Swift, HFS).
New eviction mechanism to set different rolling policies for different missions or product types.
The GSS Admin API component that is a REST API used to configure Ingester, Datastores, DatastoreGroups MetadataStores, User quotas and their properties.
New products Deletion feature with possibility to monitor the deletion process with a Notification endpoint.
Copernicus Space Interface (COPSI)
User management based on an external Identity Access Management (IAM) system.
Product visualization on 2D and 3D map.
Product list, with quicklook (when available).
Product search based on CSC OData filter syntax.
Advanced Search panel allowing the possibility to filter products on several attributes or dates.
Product details panel with property and attributes shown completely configurable.
Product download, metadata summary and products tags by mission.
Product footprints visualization on globe or plain map view and footprints style customization.
Geographic search with box and polygon AOI.
Transformation Framework
The Transformation Framework is intended to provide data transformation capabilities via the integration of processing elements applied on-demand to Copernicus Sentinel products, prior to delivery to the users.
The software includes a number of features realized thanks to an extendible and modular architecture allowing the definition and integration of custom workflows.
Processing of Sentinel-2 products from L1C to L2A via Sen2Cor v2.10.
Data processing interface based on OData REST API.
Ability to perform parallel processing.
Possibility of adding external plugins without need of changing the core elements of the framework.
User management based on an external Identity Access Management (IAM) system.
Integration with ESA Traceability Service Interface.
Implementation of quota management.
Implementation of a new plugin supporting the conversion of Sentinel products into new standard formats provided by ESA (NETCDF, ZARR and COG).
Semantic Framework
Ability to establish an open knowledge base associating real-world phenomena with relevant Sentinel data, accessible to DHS users via natural language search.
Possibility to use Cities/Countries information for Semantic queries.
Ability to retrieve Sentinel images before/after an earthquake event from defined Copernicus data sources.
Possibility to expand the supported use cases based on configurable Event Data Sources.
The Semantic Framework software introduces a new component in the Collaborative DHS Service, responsible for semantically representing metadata related with Earth-Observation and earthquakes, and intelligently exploring the relationship between them via a semantic searching mechanism; the obtained result is a combination of Sentinel-1 IW SLC products with the National Institute of Geophysics and Vulcanology (INGV) data.
DHS Suite Easy Deploy
The DHS Suite Easy Deploy is a tool that allows to install the DHS suite in a easy and efficient way, starting from a basic software configuration. In particular, this tool provides a harmonised installation mechanism, based on Docker Swarm, for all the elements, or docker images, that are part of the DHS Suite softwares, or a sub-set of them according to users needs.
Since it is used by all the software of the DHS suite, also Keycloak, in a customized version developed for the Colaboratove context, can be installed using the DHS suite Easy Deploy tool.
Installation based on Docker Swarm.
Possibility to install all the DHS components or a sub-set of them, according to user operations.
Installation on single VM or in a distributed environment of all the DHS suite or the selected components, including Keycloak.
Simply way to launch applications, restart or remove services (i.e. DHS components).
Possibility to assign specific DHS component software to individual nodes within the Docker Swarm cluster.
Supporting tools
