America is erasing the data the world needs

In recent months, thousands of web pages and datasets have been removed from United States government websites. An informal army of "data rescuers" has emerged to download, save, and republish vital information, including some 300,000 datasets on data.gov, before it is lost.

But preserving existing data is only a temporary measure. The bigger question is how future data – particularly the health and climate data that are essential to guide policy – will be produced and published.

This is not just an American problem. The US government has long supported the production of official statistics in low- and middle-income countries. For example, the Demographic and Health Surveys (DHS) program has been helping to produce survey data on key indicators such as child health and nutrition in 90+ countries for more than 40 years. 

These data have guided the development of countless valuable initiatives, from a support program for pregnant women in Pakistan to an app that broadens access to support for victims of domestic violence in Uganda. Ten of the 17 United Nations Sustainable Development Goals rely on DHS data to track progress.

But US President Donald Trump's administration has now suspended the DHS program indefinitely and is dissolving the US Agency for International Development (USAID), which runs it. Alternative means of producing important health, demographic, and other social data are urgently needed.

One useful change would be to make better use of administrative data – the data that flow through government systems every time a new patient enters a hospital, a classroom register is taken in a school, or a new baby is born – rather than relying primarily on surveys. With efforts to consolidate and streamline major household-survey programs already underway, greater reliance on administrative data would simply require a shift in focus – and investment – by governments and development partners.

Climate data are also under growing pressure. The US administration has removed or made less accessible datasets the world relies on to track and predict changes to the climate. Meanwhile, proposed funding cuts are casting doubt on the future collection and analysis of such data. Even the data that are collected might not be shared internationally, given America's withdrawal from multilateral initiatives like the Intergovernmental Panel on Climate Change.

While the European Union, Japan, and others also have huge climate-data programs, filling the gap left by the US will take considerable time and money, both of which are in short supply. Given this, innovators are needed to devise new tools and models, and to realize the full potential of new technologies, such as AI.

The US government is not just erasing or obscuring data; it is also floating proposals to alter how data are reported. For example, Secretary of Commerce Howard Lutnick has suggested dropping government-spending measures from GDP calculations. This change would make it more difficult to assess the impact of the US administration's massive federal spending cuts on the US economy.

But established norms exist for a reason: they ensure the reliability and comparability of data over time and across countries. Lutnick's proposed change would undermine the reliability and usefulness of cross-country GDP comparisons. The international statistical community must therefore hold firm in defending long-established methods and principles, with the support of partners who recognize the critical importance of this often-overlooked but essential government function.

Statisticians, data scientists, and open-data advocates around the world are developing strategies and taking action to recover, protect, and future-proof data. Data producers and users, in the US and elsewhere, can support these efforts – which will be neither easy nor straightforward – in five ways.

First, they should monitor developments carefully. What changes in data policy are governments announcing, and on what time scale? When datasets are removed and then reposted, have they been altered? Tracking such changes is essential to support advocacy to restore essential data.

Second, they should develop compelling use cases and advocacy alliances. Government data not only support social progress; businesses and AI systems also depend on comprehensive, high-quality data for innovation and decision-making. Major corporations, including tech companies at the forefront of AI development, are potential allies in advocating for the continued production and sharing of robust data in every country.

Third, new data-collection strategies must be implemented. Faced with reduced funding, countries should streamline household surveys, expand the use of sources like administrative and citizen data, share satellite data, and contribute to a robust global data ecosystem.

Fourth, to discourage arbitrary changes that undermine analysis and complicate comparisons, standards for collecting and publishing statistical and other data, such as GDP, should be strengthened and publicized.

Finally, given that prevailing methods for collecting and publishing national data were never perfect, the current upheaval can be viewed as an opportunity to collaborate to improve methodology and data governance. Any such effort to rethink how data is collected and used should emphasize global and regional collaboration, information sharing, and alignment of methods and standards.

We cannot yet predict how extensive and lasting the impact of changes to the data landscape will be, and how different organizations and institutions will respond. But if we develop comprehensive counter-strategies now, the global data community can ensure that systems essential to sound policymaking not only survive whatever comes next, but emerge stronger.

Shaida Badiee, a former director of the World Bank's Development Data Group, is a co-founder and Managing Director of Open Data Watch. Joel Gurin, President of the Center for Open Data Enterprise, is the author of Open Data Now: The Secret to Hot Startups, Smart Investing, Savvy Marketing, and Fast Innovation (McGraw-Hill Education, 2014). Claire Melamed, a former managing director of the Overseas Development Institute, is CEO of the Global Partnership for Sustainable Development Data.

Disclaimer: This article first appeared on Project Syndicate, and is published by special syndication arrangement.