Open-Source Code Repository links #

The release package of Transformation Framework is available on a dedicated Open-Source GitHub
repository:

• https://github.com/DHS-TransformationFramework/esa_tf/releases

Public Docker Repository links #

The relevant Docker images are available on dedicated Public Docker Hub repositories:

• https://hub.docker.com/r/collaborativedhs/esa_tf_restapi/tags
• https://hub.docker.com/r/collaborativedhs/esa_tf_worker/tags

Installation and Configuration Guidelines #

Installation of the Transformation Framework takes place via docker compose.

Software pre-requirements #

The following software must be pre-installed on the Virtual Machine:

• Docker engine (>=20.10)
• Docker compose (>=2.0)
• make (>=3.81)
• curl (>=7.75)
• unzip (>=6.0)
• tar (>=2.8)

Installation #

The following commands will install a version of Transformation Framework software:

curl -LJO https://github.com/DHS-
TransformationFramework/esa_tf/archive/refs/tags/v<TF-VERSION>.zip

unzip esa_tf-<TF-VERSION>.zip

cd esa_tf-<TF-VERSION>/esa_tf

make setup

In order to start the docker compose, it is then necessary to configure properly the following files:

• .env
• config/hubs_credentials.yaml
• config/esa_tf.config
• config/traceability_config.yaml

Finally, the following command pulls the docker images from the Public Docker Repository and starts the docker compose:

nohup make up &

Configuration #

Please refer to the Installation and Configuration Manual for details about the configuration.

Please find it here attached.

API User Guide #

The Transformation Framework exposes a REST API, which stores and catalogues data.
Through this interface, the Framework allows:

• the retrieval of an input product to be transformed.
• the submission of request for data transformation, including user defined parameters.
• monitoring the progress of transformation processes.

Explore available Workflows #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/Workflows

Monitor submitted Transformation Orders #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/TransformationOrders

Monitor Transformation Orders with Manager user #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/admin/TransformationOrders

Submit a Transformation Order #

“sen2cor” Plugin #

Method: HTTP POST

{{SERVICE-ROOT-URI}}/TransformationOrders


Body:

Content Type =application/json

{

    "WorkflowId": "sen2cor_l1c_l2a", 

    "InputProductReference": {

        "Reference": 
"S2A_MSIL1C_20240415T045911_N0510_R133_T43HDT_20240415T081534.zip"  

    },

    "WorkflowOptions":{

        "Aerosol_Type": "RURAL",

        "Mid_Latitude": "AUTO",

        "Ozone_Content": 250,

        "Cirrus_Correction": false,

        "DEM_Terrain_Correction": false,

        "Resolution": 10

    }

}

“eopf” Plugin #

Method: HTTP POST

{{SERVICE-ROOT-URI}}/TransformationOrders


Body:

Content Type =application/json

{

    "WorkflowId": "eopf_convert_to_zarr", 

    "InputProductReference": {

        "Reference": 
"S1A_IW_GRDH_1SSH_20240415T074035_20240415T074100_053441_067BB8_CE1E
.zip"  

    },

    "WorkflowOptions":{

        "dask_compression": "zstd",

        "dask_comp_level": 1,

        "dask_shuffle": 2

    }

}

Request specific Transformation Order (by ID) #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/TransformationOrders('<UUID>')

Filter accessible Orders (by Status) #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/TransformationOrders?$filter=Status eq 'completed'

{{SERVICE-ROOT-URI}}/TransformationOrders?$filter=Status eq 'in_progress'

{{SERVICE-ROOT-URI}}/TransformationOrders?$filter=Status eq 'failed'

Filter accessible Orders (by SubmissionDate) #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/TransformationOrders?$filter=SubmissionDate ge '2022-
06-07T00:00:00'

Filter accessible Orders (by CompletedDate) #

Method: HTTP GET

{{SERVICE-ROOT-URI}}/TransformationOrders?$filter=CompletedDate  le  '2022-
06-07T00:00:00'

Keycloak Interface #

In case the Transformation Framework is configured to support Keycloak Interface for Authentication
and Authorization, an Access Token from the Public Keycloak client is needed to be retrieved and used.

Please refer to Installation and Configuration Manual [COPE-SERCO-TN-21-1218] and Keycloak
Installation and Configuration Manual [COPE-SERCO-TN-21-1229] for details about the configuration
involving Keycloak.

Please find it here attached.

Access Token request #

Method: HTTP POST

{{KEYCLOAK-SERVICE-URI}}/auth/realms/{{KEYCLOAK-REALM}}/protocol/openid-
connect/token



Body:

Content Type = application/x-www-form-urlencoded

--data-urlencode "grant_type=password" 

--data-urlencode "client_id=<PUBLIC_CLIENT_ID>" 

--data-urlencode "username=<USERNAME>" 

--data-urlencode "password=<PASSWORD>"

Extract the “access_token” variable from obtained response.

Queries Execution #

Queries towards the Transformation Framework instance should always provide an Authorization
Header.

Method: HTTP GET

Authorization: Bearer <ACCESS-TOKEN>

{{SERVICE-ROOT-URI}}/Workflows

Method: HTTP GET

Authorization: Bearer <ACCESS-TOKEN>

{{SERVICE-ROOT-URI}}/TransformationOrders

Tutorials #

Please find available Tutorials at the following link: