Skip to main content
U.S. flag

An official website of the United States government

Deep Storage Data Release

By ScienceBase Instructions and Documentation

Deep storage data releases in ScienceBase enable USGS researchers to release large volumes of data that may not be feasible or cost-effective to release through traditional publishing paths.

As the size of datasets increase, it may not always be feasible or cost-effective to host the data on standard ScienceBase on-premises or cloud storage. In these cases, the ScienceBase Data Release (SBDR) Team can help authors leverage BlackPearl storage that is made accessible through Globus. The SBDR Team is calling data released through BlackPearl/Globus “Deep Storage” data releases. 

 

What is BlackPearl? 

BlackPearl is an object-based storage platform with an S3-like interface. It can simplify workflows for managing large volumes of data on various storage targets. USGS provides disk-based and tape-based storage targets behind the BlackPearl interface. 

 

What is Globus? 

Globus is a service that allows users to efficiently, reliably, and securely move data between systems through a single web interface. Globus enables data sharing through “Guest Collections” on top of USGS storage systems, such as BlackPearl and S3 buckets. 

 

Under what circumstances are Deep Storage data releases appropriate?  

Data releases with large volumes of data files that, when compiled, total multiple TBs of data may be appropriate for Deep Storage data releases.   

 

Under what circumstances are Deep Storage data releases NOT appropriate?  

Deep Storage data releases are NOT appropriate when the data authors need fast, programmatic access to data file content via web services (e.g., Cloud Optimized GeoTIFFs). For example, web applications or workflows requiring streaming data cannot be built on top of the data in this deep storage. 

 

What do I need to do to create a Deep Storage data release? 

  1. Request a ScienceBase data release landing page and digital object identifier (DOI) through the ScienceBase Data Release Tool, as you would for a standard data release. 

  1. First-time Globus users will need to log into Globus for the first time to instantiate their account. Select U.S. Geological Survey and you will be routed through the USGS multifactor authentication flow. 
     

    Screenshot of the Globus login screen displaying "U.S. Geological Survey" in the organizational login dropdown menu
    Sources/Usage: Public Domain. View Media Details
  1. Contact the SBDR Team at sciencebase_datarelease@usgs.gov with information about your data, including the total size, number of individual files, and anticipated downstream use of the data. If the SBDR Team agrees that the data warrants a Deep Storage data release, they will create a Globus guest collection to host the data and will provide you with the URL. 

  1. Transfer your data to the Globus guest collection using Globus File Manager. If the files are located on a personal computer or server, you may need to install Globus Connect Personal on that machine to enable the transfer. Globus Connect Personal is available for Windows, Mac, and Linux. Please read the installation instructions thoroughly before installing. You should NOT need USGS service desk assistance to install the software. 
     

    Screenshot of the Globus File Manager interface with the Panels option circled in red
    Sources/Usage: Public Domain. View Media Details
  1. Upload a collection metadata record to the ScienceBase landing page 
  1. Add one of the following statements to the end of the summary on the ScienceBase landing page: 
    1. Data files for this data release are large and cannot be easily downloaded through standard protocols. Therefore, the data files are provided via a Globus Access Portal (linked under Related External Resources). Globus* is a fast, secure, and reliable way to move large data files and will automatically resume transfers when there are network disruptions. Learn more at https://www.globus.org/data-transfer.   
       
      * Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.  
       
    2. Data files for this data release are numerous and cannot be easily displayed for browsing in the ScienceBase interface. Therefore, the data files are provided via a Globus Access Portal (linked under Related External Resources). Globus* is a fast, secure, and reliable way to move large data files and will automatically resume transfers when there are network disruptions. Learn more at https://www.globus.org/data-transfer   
       
      * Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. 
  1. Contact the SBDR Team at sciencebase_datarelease@usgs.gov to make the data release public. 

 

What steps will the SBDR Team take to finalize a Deep Storage data release? 

  1. Perform all the same checks that are performed for a standard data release (see ScienceBase Data Release Checklist) 
  1. The SBDR Team will add a link from the ScienceBase data release landing page’s Related External Resources section to the Globus guest collection 
  1. A copy of the final XML metadata and the ScienceBase landing page JSON will be uploaded to the Globus guest collection 
  1. Update permissions to the Globus guest collection (authors’ write permissions will be removed and read permissions will be granted to all Globus users). To access the data, users will need a free Globus account. They can use their institutional, GitHub, Google, or ORCID identity to create an account. 
  1. Make the ScienceBase landing page and DOI public. 

How can end users access data from the Deep Storage data release?

End users will need a free Globus account and will need to log into the Globus system to access data in a Deep Storage data release. They can log in with a number of identity providers including Google, GitHub, or ORCID. Additionally, many universities and government institutions are also available as identity providers for Globus. Once they have logged into Globus and have access to the Collection, they can execute a Globus transfer to their local machine using Globus Connect Personal or any other Globus Collection to which they have write access. Learn more about logging into Globus and transferring files here: https://docs.globus.org/guides/tutorials/manage-files/transfer-files/. 

Related

The following data releases are examples of deep storage data releases in ScienceBase.

CONUS404 climate forcing variable subset for hydrologic models, 1979-2022: downscaled to 1 km and bias-adjusted for precipitation and temperature

This metadata record serves as documentation for the bias adjusted version of the CONUS404 (CONtiguous United States for 40-years at 4-kilometer grid spacing) atmospheric forcing dataset; however this dataset has been revised to include 43 years of data at 1-kilometer grid spacing. This is a dataset of historical conditions (water years 1980-2022, October 1, 1979-September 30, 2022) and has suffic
By
Water Resources Mission Area

CONUS404: Four-kilometer long-term regional hydroclimate reanalysis over the conterminous United States (ver. 2.0, December 2023)

CONUS404 is a unique, high-resolution hydro-climate dataset appropriate for forcing hydrological models and conducting meteorological analysis over the contiguous United States. CONUS404, so named because it covers the CONtiguous United States for 40 years at 4-km resolution, was produced by the Weather Research and Forecasting (WRF) Model simulations run by National Center for Atmospheric Resear
By
Water Resources Mission Area

Collection of COAWST model forecast for the US East Coast and Gulf of Mexico

The COAWST modeling system has been used to simulate ocean and wave processes along the of US East Coast and Gulf of Mexico. The grid has a horizontal resolution of approximately 5km and is resolved with 16 vertical terrain following levels. The model has been executed on a daily basis since 2010 with outputs written every hour. Data access is available through a Globus access portal here: https:
By
Coastal and Marine Hazards and Resources Program, Woods Hole Coastal and Marine Science Center

Related

The following data releases are examples of deep storage data releases in ScienceBase.

CONUS404 climate forcing variable subset for hydrologic models, 1979-2022: downscaled to 1 km and bias-adjusted for precipitation and temperature

This metadata record serves as documentation for the bias adjusted version of the CONUS404 (CONtiguous United States for 40-years at 4-kilometer grid spacing) atmospheric forcing dataset; however this dataset has been revised to include 43 years of data at 1-kilometer grid spacing. This is a dataset of historical conditions (water years 1980-2022, October 1, 1979-September 30, 2022) and has suffic
By
Water Resources Mission Area

CONUS404: Four-kilometer long-term regional hydroclimate reanalysis over the conterminous United States (ver. 2.0, December 2023)

CONUS404 is a unique, high-resolution hydro-climate dataset appropriate for forcing hydrological models and conducting meteorological analysis over the contiguous United States. CONUS404, so named because it covers the CONtiguous United States for 40 years at 4-km resolution, was produced by the Weather Research and Forecasting (WRF) Model simulations run by National Center for Atmospheric Resear
By
Water Resources Mission Area

Collection of COAWST model forecast for the US East Coast and Gulf of Mexico

The COAWST modeling system has been used to simulate ocean and wave processes along the of US East Coast and Gulf of Mexico. The grid has a horizontal resolution of approximately 5km and is resolved with 16 vertical terrain following levels. The model has been executed on a daily basis since 2010 with outputs written every hour. Data access is available through a Globus access portal here: https:
By
Coastal and Marine Hazards and Resources Program, Woods Hole Coastal and Marine Science Center
Was this page helpful?