Latest CREODIAS Statistics

  TB of Locally Available Data

Repository	Total	Processed	In progress	Not mirrored	Queued new	Queued old / reprocessed	Last 1h	Last 24h	Last 1h (GB)	Last 24h (GB)	Avg. per product (GB)	Total (TB)

Data Offer

• Online, locally, immediately access to Earth observation data, growing by around 20 TB daily
• EODATA, available online - full (time and space coverage) Sentinel-1 GRD, Sentinel-2, Sentinel-3, new Sentinels (like 5P), ESA/Landsat, Envisat/Meris, full Sentinel-1 SLC for Europe and 6 months rolling archive for Sentinel-1 SLC outside Europe, elements of Copernicus Services (check available datasets in the Data Catalogue)
• Immense cache (starting at 1 PB) cache for external data; API call to order any data available in the Finder;
• Wide range of data access interfaces: S3-like object interface, WMS/WMTS access, file system (best effort, not recommended) interface for legacy software (read about Data Access Interfaces)
• No need to download data – it can be processed directly from the cloud; for sample processing app check S2Scenes
• Easy tools to discover and browse data
• EODATA+, commercial VHR (Very High Resolution - submetered) Satellite images - Archive imagery, New tasking, Video and Luminos imagery.

Please read the License for the use of Copernicus DEM datasets, which are available in the EODATA repository throuh the path /eodata/auxdata/CopDEM and Finder UI/API.

* Can be ordered from ESA via CREODIAS Finder
** If L2A does not exist in ESA archive, it can be processed by CloudFerro with Sen2Cor
*** Kept in rolling cache of 1 PB (mostly European coverage). Oldest data will be removed when cache is full
**** Sentinel-5P L2 products currently stored: Aerosol, Cloud, CO, HCHO, NO2, NP, O3, SO2

Mandatory, Complementary and Additional Data Collections

CREODIAS provides several main data categories. There are three main data categories available in the CREODIAS – these are:

• So called Mandatory CREODIAS Data collections comprising all Copernicus satellites data – namely Sentinel-1, Sentinel-2, Sentinel-3 – present time missions together with future missions like Sentinel-4, Sentinel-5 and Sentinel-6; and all Copernicus Services information. These satellite data and Services data shall be available free of charge (with or without a license) and in their entirety, in compliance with the EU regulations concerning the use of Copernicus Sentinel Data and Service Information.
• So called Complementary CREODIAS Data Collections – all data outside the above category but also based on free licenses and provided in the standard CREODIAS offer. In case of this offer this is so called ESA/Landsat satellite data – legacy (starting with Landsat-5) till present Landsat-8 data and legacy Envisat/Meris data. Legacy data is especially useful for obtaining long-term comparative data series. This category also includes Copernicus Core datasets and data from Copernicus Contributing Missions.
• Additional CREODIAS Data Collections – all data sets that can be supplied either by other satellite providers (like e.g.  Planetek, AD&S  with high resolution sat images) or produced and delivered by the Third Parties operating on the CREODIAS platform. While the two first data categories (Mandatory and Complementary data collections) are based on free licenses and are free and openly available to everybody (usually subject to accepting a license agreement) – this Additional CREODIAS Data can be accessible according to varying license rights. In particular the use of this data can be based on commercial purchases.

The attractiveness of the CREODIAS platform for Third Party Users is, among others, measured by the breadth of its data offer and the ease of searching, accessing and using this data. The whole system is designed and optimized to:

• deliver all existing Mandatory and Complementary Data to the CREODIAS users without any additional charge
• maximize the utility of the data available locally online on the CREODIAS Platform
• avoid wasting storage space on rarely used voluminous data
• provide easy and transparent methods for accessing any data available on the CREODIAS Platform (also from remote repositories where applicable)
• provide homogenous metadata, catalogue and access API-s and functionalities across all the data collections
• provide Users with clear, machine-consumable information on the data sets availability status

Availability of Sentinel core products and Copernicus Services information for Third Parties

For the CREODIAS Data Offer we assumed that:

• Any data (EO satellite data) or information (Copernicus Services) available on the CREODIAS Platform is indexed and can be searched via its Search Tools. These Tools can be either used interactively or programmatically via an API. Search can be performed at the level of data collections and at the level of data or information products. At the level of collections a User can get information about collections compatible with his search criteria. At the level of products as a result of the search Users will get a list of data or information products compatible with the search criteria together with related metadata. Each product is going to be attributed information about its CREODIAS Data Location. A product’s CREODIAS Data Location informs the User on general availability of the product: the actions that can be performed on this product and Data Access Interfaces that can be applied to this product.
• There are four Data Locations that are provided by the CREODIAS Platform:
  • CREODIAS Data Location: Local – data and information are mirrored online on the Platform.
  • CREODIAS Data Location: Cached – data and information has been downloaded or generated  per User’s Order and placed in the CREODIAS Shared Cache which is a part of the local storage.
  • CREODIAS Data Location: Orderable – Data and information are indexed in the CREODIAS Platform Catalogue but the bulk data is physically stored in other repositories (like Scihub, Collaborative Ground Segments, Copernicus Services); or the data is indexed but is not actually existent and has to be generated on-the-fly.
  • CREODIAS Data Location: External – External data or information are indexed and can be accessed as links in their original web-based sources (services) with the use of a Proxy interface described below.
• Local data is most convenient for the Users. In this Location the data and/or information was automatically downloaded from the source archives and its full copy is kept in the CREODIAS Data Repository. It can be reliably accessed, processed and/or copied by Users online within miliseconds. This data is kept in the CREODIAS Data Repository indefinitely – is not going to be removed from the archive as long as the CREODIAS Platform is going to be operational. Most of the Mandatory and Complementary Data is stored locally – namely: all Sentinel-2  L-1 data, all Sentinel-3, Sentinel-4, Sentinel-5 and Sentinel-6 data (when relevant satellites will be launched), a lot of current and new Copernicus Services information, all ESA/Landsat data, legacy ESA/Landsat data and Envisat/Meris data, all Sentinel-1 GRD data worldwide and Sentinel-1 SLC data for the area of Europe and Africa (see plot below). Data available in the Location Local can be accessed via all the access mechanisms available on the platform –  Object Data Interface (SWIFT/S3), via Filesystem Interface (NFS), via the EO Data Access Hub (OGC WMS) and HTTP Download. Information available in the Location Local can be served via a subset of the above interfaces depending on this information format and nature. It is worth to notice that also data and information in the Data Location Cached is available via all the interfaces available for Local Location.

Figure 1 – Coverage of Sentinel-1 SLC data kept in Data Location Local

• Another main Data Location is Orderable. Orderable data may not be  immediately present in the CREODIAS Data Repository and it cannot then be accessed immediately. Orderable products can be ordered by the User and then automatically downloaded or generated by the CREODIAS Platform in the background and then placed in the CREODIAS Shared Cache. The Order can be placed by a registered User according to  a rich set of data parameters covering collection, instruments, area and time range. Users are informed when the download process is completed. As a result of the Order – data is automatically downloaded (or generated). After the Order is fulfilled, the data is placed in the CREODIAS Cache, its Data Location becomes Cached and it can be accessed with the same Data Access mechanisms as Local data: Object Data Access, Filesystem Interface, EO Data Access Hub and HTTP Download. When it is not cached, data in Location Orderable can be downloaded via HTTP Download.

Below we summarize the information about the CREODIAS Shared Cache:

• Orderable data can be downloaded to the Shared Cache.

• It can be downloaded per User’s Order.

• The Shared Cache can be used by multiple Users.

• The non-discriminative character of the use of the Shared Cache is governed be a special fair use policy.

• Within this policy each User obtains his Quota – if his data Orders are within his Quota – his Cached data will not be removed from the Cache.

• The User’s Quota depends on the total number of Users using the Cache and can change with time. Quota has to be monitored by the User to prevent his Cached data from being removed from the Cache.

• The Shared Cache is very big (1 PB at the start of the Platform with yearly increments of 0,5 PB) so most Users can effectively use and keep their Cached data.

• However any User can increase his Quota by a purchase of additional quota which is billed in the same manner as other storage. This is a way to permanently keep even big amounts of Cached data.

There are two main data sets from the Mandatory data that are not in Data Location Local but they are in Orderable and Cached Location. These are: Sentinel-1 data that is not already present in the Location Local (SLC data from outside the area presented in the plot above)and Sentinel-2 Level 2 data. Some Additional data can be also in the Locations Orderable and Cached and can be ordered.

Some of the CREODIAS Data collections, in particular Sentinel-2 Level 2 will not be downloaded but generated dynamically on demand from data already available on the CREODIAS platform in the CREODIAS Location Local. A special cluster of machines will be assigned to that task. The CREODIAS platform will be able to generate 40 Sentinel-2 products per hour (which is half of the full Sentinel-2 products’ data stream). In case of Users’ interest we will increase the capacity of the cluster to accommodate the needs. The platform will ensure fair sharing of this bandwidth when several users access the Data Generation Mechanism at the same time. From the User’s perspective, the Data Generation Mechanism will be accessible through the same Order functionality as described above. When a User orders a data set from a dynamically generated data collection, data products are dynamically generated by the CREODIAS Platform and placed in the CREODIAS Cache of  CREODIAS Data Repository instead of being downloaded from their source. The collections available via dynamic generation are described in detail in the “CREODIAS data and information integration plan”.

The use of the Cache is free of charge for the Users – but there will be a way to permanently increase User Quota in the CREODIAS Cache . A User will be able to reserve/buy additional CREODIAS Cache quota in the same way he buys other storage services.

The data from the Data Locations: Local, Orderable and Cached can be downloaded directly to Users storage – it can be a storage in the User’s cloud resources on the CREODIAS Platform or in his local remote location. Some data or information cannot be however downloaded but is also indexed in the CREODIAS Catalogue – it can be then accessed only as http links in their original web services. Such information or data has Data Location External and can be accessed only via a special Proxy interface.

A more detailed information about the standard CREODIAS Data Locations for Mandatory and Complementary data sets is presented in tables below and in the “Copernicus CREODIAS data and information integration plan”. All the changes in the datasets available via the CREODIAS Platform, in particular inclusion of new datasets, are going to be documented in the “Copernicus CREODIAS data and information integration plan” as well.

Conditions for integrating Additional Data

It should be noted that data produced by the Third Parties can be placed in the CREODIAS Data Repository. In this case however it can be necessary for Third Parties to pay for the relevant storage space and they may choose to limit access to the data to selected or paying customers. This data has also to be equipped with metadata according to the rules for Data Market Service.