ScienceBase Updates - Fall 2021
Fall 2021 topics include information on ScienceBase Cloud Upload, SBDR Tool retry service and dashboard, a tip on rearranging your contacts, and a featured data release on rainfall totals resulting from Hurricane Harvey.
Update on Latency IssueScienceBase latency issues affecting the system in July and August have been resolved. The problem, which resulted in slow or non-loading page views or searches, stemmed from the failure of hardware component and the configuration of the ScienceBase ElasticSearch index. Both of these issues have been addressed and performance has been back to normal, with uptime at 99.8% for the last month. Thank you for your patience as the team addressed these issues. ScienceBase Cloud File UploadSince coming online, ScienceBase has implemented several different protocols for file upload, all focused on increasing the size of the files the system can support and the type of functionality offered for those files once they’ve been uploaded. Most users are familiar with the traditional mechanism for attaching files in the ‘Manage Item’>’Edit Item’>’Files’ interface and the 'Large File Uploader' to handle larger files ranging from 1-10 GB in size. As part of a continued effort to support larger files more easily for users, and offer expanded cloud-based file features, ScienceBase now has a new approach for file upload and management that works directly against the ScienceBase cloud storage in the USGS Cloud Hosting Solutions (CHS) environment. This interface will replace the functionality of the Large File Uploader, and will eventually become the single interface for file management in ScienceBase. Logged in users will see the option to ‘Access File Manager’ on the bottom left of the item page. Users will be prompted through another login step to use the file manager from an item. Accessing the ScienceBase cloud file manager is possible on or off the USGS network; in testing most users have reported higher speeds of upload while on the USGS network or VPN. While performance may still be dependent on users’ local internet connections, files up to ~30 GB in size were successfully transferred during development and testing. 
Sources/Usage: Public Domain. View Media Details
The cloud uploader should now give users the best performance for any larger files being uploaded to the system and should replace the Large File Uploader in their workflows with this feature. The Large File Uploader interface is slated to be removed from the item page shortly. Note that, similar to the behavior of the Large File Uploader currently, no automated scraping or processing will occur for files that go through the cloud file manager (e.g., no automatic metadata ingest into item, no map service generation, etc.). Users needing those features for given files should continue to use the traditional file upload in the ‘Manage Item’>’Edit Item’>’Files’ interface. 
Sources/Usage: Public Domain. View Media Details
An example of a dataset published in ScienceBase. The author provided the file in a Cloud Optimized GeoTIFF (COG) file format to support direct access from storage. The item is available here: https://www.sciencebase.gov/catalog/item/5e6be1bfe4b01d5092638a75.
|
Featured Data Release
Sources/Usage: Public Domain. View Media Details
Rainfall totals in southeastern Texas and southwestern Louisiana from August 25 through September 1, 2017, resulting from Hurricane Harvey and locations of physiographic sections and climate divisions. (Watson and others, 2018). Watson, K.M., Welborn, T.L., Stengel, V.G., Wallace, D.S., and McDowell, J.S., 2018, Data used to characterize peak streamflows and flood inundation resulting from Hurricane Harvey of selected areas in southeastern Texas and southwestern Louisiana, August–September 2017: U.S. Geological Survey data release, https://doi.org/10.5066/F7VH5N3N. USGS Data Owner: Oklahoma-Texas Water Science Center Hurricane Harvey devastated areas of Texas and Louisiana with record amounts of rainfall causing catastrophic flooding in August and September of 2017. The USGS and FEMA (Federal Emergency Management Agency) worked in cooperation to evaluate and map the extent of the flood in Texas. One result was this dataset containing flood inundation polygons, flood-depth rasters, mapped boundaries, and high-water mark (HWM) locations for the selected river basins, coastal basins, and coastal areas in Texas and Louisiana that flooded Hurricane Harvey and its aftermath. The data release landing page in ScienceBase has been heavily accessed, with over 20,000 downloads of the data, and over 130,000 visits to the main landing page, not including APIs (data obtained from the ScienceBase Data Release Summary Dashboard). Further, the data released on ScienceBase has been reused to identify zip code areas affected by flooding in Texas for a publication on the use of Twitter in disaster rescue (Mihunov and others, 2020). The related publication has also received attention, having been mentioned in articles from three news outlets, tweeted about by 62 users on Twitter, and mentioned as an example of the effective use of streamgages in the policy document “U.S. Geological Survey (USGS) Streamgaging Network: Overview and Issues for Congress”. If you know of a data product available in ScienceBase that has gone on to be reused in other projects, inform policy decisions, garner attention in major media outlets, or any other interesting use, we'd love to hear about it. Please complete this form to contribute your data story. |
ScienceBase Data Release Tool Retry Service and My Dashboard
The ScienceBase Data Release (SBDR) Tool relies on connections to ScienceBase and the Digital Object Identifier (DOI) Tool to help researchers and data managers initiate a landing page and DOI for their data releases. Sometimes, network outages can pose a challenge in connecting to both tools. We’ve updated the SBDR Tool to be more resilient against these types of short outages and downtime. Now, when someone submits a landing page request form through the tool, if ScienceBase or the DOI Tool cannot be reached immediately, the request will enter into a retry service. The service will continue trying to reach ScienceBase or the DOI Tool every 15 minutes for up to 36 hours.
When a network outage like this occurs, users may see a Bad Gateway error upon submission. In these cases, the submission usually has gone through. To verify, users can check their new My Dashboard to see if their submission has shown up as a new record. The My Dashboard page is the first thing someone will see upon logging into the SBDR Tool (https://sciencebase.gov/datarelease/home). If the submission has gone through, the data release title, date submitted, and Science Center will appear in the table. If the SBDR Tool is still waiting to receive a response back from the DOI Tool or ScienceBase, the word “Pending” will appear in those fields. Once a connection occurs, the fields will be populated with the DOI and/or the ScienceBase URL, and an email will be sent to the submitter and their data manager, if applicable. If an email has not come through after 36 hours, the submitter should contact the ScienceBase data release team at sciencebase_datarelease@usgs.gov.
In addition to understanding if a submission went through, the My Dashboard page is also a great resource for remembering if a landing page and DOI was already requested for a particular data release and finding the link to the landing page. The My Dashboard table can be sorted by any of the columns and a search box is also available. Please note that the My Dashboard feature is intended to provide a history of submissions through the SBDR Tool, but the information is not intended to be a current view and status of data release submissions. For example, if the title of a data release is changed on the landing page, that update will not be reflected in the SBDR Tool - My Dashboard page.
Please direct questions about these new features to sciencebase_datarelease@usgs.gov.
Subscribe to the ScienceBase Mailing List for Quarterly Updates.