USGS Water Data API Modernization

This blog post provides a summary of updates to the USGS Water Data APIs (https://api.waterdata.usgs.gov/), providing information about the transition from the old API to the modernized API. For the complete USGS presentation, check out the YouTube video below.

The U.S. Geological Survey (USGS) is undertaking a long-term mission to modernize its systems for sharing water data, aiming to expand accessibility and usability. This includes changes to the USGS water data APIs, which have been sharing data for decades.

The new modernized APIs, which began rolling out in 2024, offer several key changes and features:

• Standard Adoption: The new APIs implement the OGC API Features standard, a widely adopted standard that provides significant interoperability benefits and should make integration into existing workflows easier.

• New Base URL: All new endpoints are now located at api.data.usgs.gov, with specific paths like OGCAPI/Vzero/collections.

• Query Changes:

  • Queries are designed to be more explicit and readable, moving towards plain language and fewer abbreviations (e.g., "daily" instead of "DV", "monitoringLocationIds" instead of "sites").

  • Monitoring location IDs now include an agency code to create unique identifiers, addressing potential duplicates across different data-furnishing agencies.

  • The format argument has changed from format=JSON to F=JSON due to the new standard.

• Simplified Response Objects:

  • The new APIs return much simpler, slimmer GeoJSON objects. This means faster downloads.

  • Less metadata is included directly with observational data (e.g., daily values, continuous values). Fields like server name, detailed monitoring location, and parameter code metadata are largely removed to improve performance and reduce wasted data transfer for common high-volume requests.

  • Metadata can be looked up separately using keys (e.g., time series ID, monitoring location ID, parameter code) in dedicated metadata endpoints, allowing users to build their desired datasets more efficiently.

  • Missing values are now represented as nulls instead of stand-in numerical values (e.g., -999).

  • Qualifiers (e.g., "ice-impacted," "estimated") are separated from approval status. Qualifiers are returned as an array, using full words where possible, while approval status (provisional or approved) is a distinct field.

• Pagination: Responses that exceed a certain number of rows (default 10, up to 10,000 configurable) are split into multiple pages. Users must follow "next" links to download the full dataset. CSV downloads currently do not have a good way to include these "next" links.

• Versioning: The current version is Vzero, indicating that features are still being added. A V1 will be released later, running in parallel with Vzero for at least six months before Vzero is decommissioned, providing ample time for migration and preventing sudden breaks in service. Changes between V0 and V1 are expected to be much smaller.

• Flexible Queries for Fields: Users can request specific fields (e.g., just value and time) to further reduce response size and improve performance for applications like web apps or live maps.

• Direct GIS Access: A highly requested feature, the new APIs allow direct access to data from Geographic Information Systems (GIS) like QGIS. A tutorial for QGIS is available, and an ESRI ArcGIS tutorial is in development.

• API Keys: Signing up for an API key at api.waterdata.usgs.gov/signup increases the rate limit from 100 requests per hour per IP address to 1,000 requests per hour for that specific key, which is highly recommended for efficient use.

Currently Available Endpoints include daily values, time series metadata, monitoring locations, latest continuous measurements, parameter codes, and a new field measurement service (combining surface and groundwater measurements). The statistics service and samples data service were modernized separately in 2024. 

Upcoming Endpoints include a full replacement for the instantaneous values (IV) service (called "continuous"), and proper APIs for peaks and ratings pages, expected in winter 2024 or early spring 2025. 

Computational Tools (R and Python packages like dataRetrieval) are being updated to work with the new APIs. New functions (e.g., read_waterdata_daily in R) are being developed and will run alongside legacy functions during an overlap period. A notable change in these tools is that new functions will return data in a "long format" (each value for each parameter code in its own row), unlike the "wide format" returned by legacy functions. Python functions are expected to be ready "sometime this fall".

The USGS recommends migrating to the modernized services as soon as practical, as there may be performance degradation to the legacy water services due to hosting environment changes. 

Resources for migration, including documentation, tutorials, and a migration guide, are available at api.waterdata.usgs.gov.