The recent Tech@State conference featured an address from Robert Kirkpatrick of United Nations (UN) Global Pulse. There have been many applications of social media to the relatively short-term process of crisis management, but Global Pulse instead focuses on the utility of real-time data for longer-term development. The initiative has focused on analyzing new, digital data sources — social media, online news, online prices, etc. — to demonstrate how this can enhance traditional statistics. For example, in one of Global Pulse’s initial projects, the number of Indonesian tweets including the words “rice” and “price” was found to correlate closely with official Indonesian government food price inflation rates. The work has intriguing potential, and Pulse Labs will soon be operational in Indonesia and Uganda. The Wilson Center’s Zachary Bastian spoke with Robert about the project.
Where did Global Pulse begin?
Global Pulse began with recognition, at the height of global economic crisis in 2009, that we were seeing something new. In today’s volatile and hyper-connected world, socioeconomic shocks emerge and reverberate around the globe almost with the speed of natural disasters. World leaders became very concerned about the impact of the crisis on vulnerable populations — they knew millions would be at risk. Jobs would be lost, and families would end up selling assets, making difficult trade-offs, and falling back into the grip of poverty and hunger. Yet as they sought to prioritize responses with dwindling resources, they found that up-to-date, household-level information on who was being impacted, and how, simply wasn’t available. National census data and official statistics about how populations are faring lag by months — and often by years.
As the global economy becomes more volatile and the pace of change accelerates, we have found that traditional 20th century tools for tracking development cannot keep up. A great deal of money is spent on aid, but these investment decisions are often made without real time awareness of needs or real-time feedback on the efficacy of our policies and programs. Recognizing both the need for more timely and accurate information, as well as the opportunity presented by the emergence of “Big Data,” the UN Secretary-General established the Global Pulse initiative to help the global community gain the situational awareness required for protecting fragile development gains in the 21st century.
What is your background on these issues?
My background is in software development and organizational change. Since 2002, I have focused on technology for disaster relief and global development, with a particular focus on information sharing and evidence-based decision making. I co-founded the Humanitarian Systems Team at Microsoft, and from 2007-2009 I was with the nonprofit INSTEDD, developing technology tools for disease outbreak surveillance and crisis response. This work ended up being relevant to Global Pulse, because here, too, we are focused on mining for patterns in new data streams.
There is an important lesson that global development can learn from software development. In the old days, software was built using a process called the “waterfall method.” You would spend a year understanding your customer’s needs, a year designing a solution, and another year building it. All too often, you would discover at the end of the process that your customer’s needs had changed as the years passed, and that what you had built was no longer the right tool for the job. So about 10 years ago, software developers starting adopting a new approach called Agile Software Development, where you rapidly prototype a few software features at a time, and use constant, rapid customer feedback to extend and refine the product every few weeks. Even if the process still takes three years, you can be sure that you have built the right product, because your customer was right there with you the whole time.
Our challenge today is that global development still uses the waterfall method. We make five-year implementation plans using three-year-old statistics. Based on the research we’ve been doing at Global Pulse, we see an opportunity for the global community to move toward a more iterative and adaptive approach, powered by real-time digital feedback from beneficiaries that will yield results sooner.
What is the next step towards putting these tools to concrete use?
Global Pulse is setting up a network of innovation labs around the world to learn how to harness big data for development and support institutional adoption of real-time data into policy-making. We see a lot of innovation happening with the rise of data analytics in the private sector, driven largely by the advertising industry. If the private sector can use this data to understand its customers in real time, we are looking to put the UN in the lead in adapting these new tools and approaches to strengthen the public sector. We need to understand in real time when people are losing their jobs, migrating for work, or struggling to make ends meet. We can bring the diverse expertise from the UN, government, academia, and private sector to learn together which data sets to analyze, what patterns to look for, what technology tools are needed, and how to ensure that privacy is protected in the process.
Digital data will not replace traditional statistics, but it has the potential to more quickly reveal anomalies. For example, in the public health sector, data mining technologies are used to monitor for anomalies in collective behavior that could indicate outbreaks of diseases like the flu: Analysts pay attention to trends like school attendance, purchases of certain types of medicines, etc. And when those patterns vary significantly from the norm, it triggers an investigation and verification process that allows for earlier detection and earlier response. And this isn’t just about health. Changing patterns of mobile airtime purchases have been shown to correlate with job loss. Analysis of Twitter has been use for rapid and accurate detection of earthquakes.
People openly share stories on social media about how they are coping with rising prices of food, healthcare, and education. In some cases, big data may give us the same information we already use to shape policy decisions, but faster and at lower cost; in other cases, it will give us entirely new insights. With the ocean of digital data out there that people are now generating just by going about their daily lives, we have the opportunity to apply this type of approach both to improve how we monitor progress toward the Millennium Development Goals, and to enable early detection and response to the types of socio-economic shocks that threaten to reverse those gains.
People grab onto an issue when it’s a hot topic, but interest can wane and support can fade. How do you perceive the skepticism about the value of this data?
That gets at how quickly this all came about. Ninety percent of all the data created by humanity was generated in the past two years. The public sector is only now beginning to realize the potential here. Tapping into this new data that is all around us is like gaining new senses, like learning to see and hear for the first time. Based both on our own research and on the constant stream of news stories about Big Data, the UN is getting excited about the potential here for improving how they design, implement, monitor, and evaluate global development programs. A few years ago, for example, they saw social media as tools for advocacy; now they see new ways to listen directly to the voices of beneficiaries and to enable grassroots community action in response. Similarly, many companies worldwide are willing to share tools and data, universities want to work with us on research, and the open data and open source software communities are fully on board.
Of course we do recognize that Big Data, like all new phenomena, follows a hype curve. There is huge excitement out there right now, and many in the statistics community are rightly pointing out that Big Data is still data, with all the usual problems related to sample bias. The field of statistics itself has benefited from decades of improvement and is still far from perfect, and Big Data has years of refinement ahead of it. More data is not always better data, but as anyone in emergency medicine or disaster management knows, faster data — if viewed with a critical eye and applied in appropriate ways — can change outcomes.
If anything, our challenge has been fundraising, and I’m not talking just about the general difficulty of raising money when there is less of it to go around. The Global Pulse project is not funded out of the UN’s annual budget. We rely on voluntary donations from governments and private foundations for support, and here we often encounter conceptual, generational gaps in understanding. It can be challenging to engage with traditional donors. For example, Global Pulse points to the ocean of data, but not everyone is on the same page; many in development are still not aware of the types of digital data streams being generated by populations living below the poverty line in developing countries, nor are they aware that there are already tools and technologies already available to analyze and visualize that information.
Many may not yet understand that real-time data isn’t just faster — it’s fundamentally different — and that it’s not a replacement for the statistics used to make policy decisions, but a way to learn sooner where you need to go collect actionable evidence. So, I don’t think we are as concerned with being a hot-button topic that fizzles away. There’s a paradigm shift underway here analogous — one analogous to the industrial revolution — and we’re not even halfway through it. Our concern is finding more early adopter partners in the donor community who are willing to support the research that will cut through the hype around Big Data, rigorously test out new approaches on the ground, and discover the genuine opportunities to change outcomes for vulnerable populations.
How far along is the setup of the Global Pulse Labs in Jakarta, Indonesia and Kampala, Uganda?
The initial steps for establishing a country-level Pulse Lab involves finding the right institutional partners and getting appropriate agreements in place between the UN and government stakeholders. With initial seed funding from Sweden and the UK, Global Pulse has now completed scoping and planning for Pulse Lab Kampala, and we currently planning for Pulse Lab Jakarta. We are still working to raise funds to launch both labs. At this stage, we have received funding from the Australian government to undertake two initial projects in advance of Pulse Lab Jakarta’s launch, and for these we’re partnering with some world class research teams and private sector leaders in data analytics. These projects will include analyzing archives of both Twitter data, of which there is a lot in Indonesia, and anonymized mobile phone calling records from the past few years.
We will look at the priority issues of employment, food prices, and fuel prices for specific populations and time periods, then compare these new data sources with official statistics. Our goal is to understand whether, and what changes show up in these real-time data streams when crises began impacting different populations, and when policy responses where initiated. We’re looking to discover real-time proxy indicators of changes in population well-being. We hope to have the Lab fully launched before the end of summer 2012. The Indonesian and Ugandan governments are strongly supportive and excited to partner with us on R&D. We hope these are the first of many collaborative efforts at the country level.
About the Author
Zachary Bastian is a third-year student at George Washington University Law School. Before joining the Woodrow Wilson International Center for Scholars, he worked in the United States Senate studying policy and supporting the work of personal and subcommittee offices. His interests include intellectual property, disaster mitigation and recovery, and leveraging technology to support better government. He can be reached at zack[dot]bastian[at]gmail[dot]com.