Will end users demand to see inside climate services ‘black boxes’?

Will end users demand to see inside climate services ‘black boxes’?

As the demand for climate services grows, questions are beginning to be asked about the assumptions behind some of the tools and methods used to translate climate science into information that can be used by decision-makers. Many of these assumptions are considered to be proprietary, and therefore exist within a ‘black box’, with end users unable to scrutinise the methodologies. This means that users are not able to fully understand the assumptions that underpin the findings of the climate risk analysis, and the recommended course of action. Are we making resilience decisions in the dark?

There are an estimated 350 commercial climate and weather services providers in the U.S., a rapidly growing figure as the market continues to mature, and the impacts of climate change to commerce become more frequent and intense. In the U.S., commercial service providers rely on publicly available data, from agencies like NOAA and NASA, to develop value-added products and services for decision-support. This may be targeted at a particular sector, or in relation to a particular climate or weather hazard. An example of a value-added service could include a flood risk assessment tool that alerts property developers to flood risk potential. The property developer leverages this information to avoid investment in a particular area or build to withstand future flood risk potential.

The U.S. model of free and open access to data has created the foundation for a commercial value-added weather service industry that has enabled a sub-sector worth around $7 billion. For meteorologists or technological savvy entrepreneurs, this presents a large market opportunity to develop value-added tools based on information that is freely available. While anybody could theoretically access NOAA or NASA data to inform climate and weather-related decision-making the data is usually not available in formats that are accessible to a non-expert user. Therefore, the value-added process is an important step to make the information useable, and inform decision-making.

The users of the products likely do not understand the climate and weather data behind the applications, nor the value-added processes that render this information usable. The end results – a level of risk, or a dollar sign – may be the only information they are interested in. However, for those users that want to dig a bit deeper into the data and processes that inform the service, they may run into a black box. In crude terms, the general approach to developing a client-facing service appears to be: meteorological data (i.e. NOAA / NASA) + other data + client specific information + proprietary algorithm = value-added service.

While service providers may disclose the meteorological information that they are accessing (such as station data from NOAA) the process of adding value to this information is often considered proprietary. Commercial service providers generally do not disclose their methodology in fear of compromising their cutting edge against a competitor. They provide demos of the service but do not open up the contents. This raises the question of how can the products be open to scrutiny and comparison if they are proprietary? How can a user make an informed decision between one service and the next, if they don’t have substantial information about either?

Importantly, there is no consensus on governance standards for developing, or applying climate services (Adams et al, 2015). The WMO’s Global Framework on Climate Services is guiding the development of climate services for decision-support in climate sensitive sectors, particularly in developing countries, however there is no entity – to our knowledge – vetting commercial applications.

Climate services are a big and growing industry, with new firms continuing to enter the market. As individuals and businesses are increasingly making risk-management decisions based on the output of these services, worth large sums of money, what mechanisms are in place to ensure the integrity of these services? Will the black boxes become more transparent? Will a third-party be allowed to peer inside the black boxes? Or will the market grow in a similar to that of the catastrophe modeling industry where black boxes are the norm

Cover photo by Kelvin Yan on Unsplash.

About the Author