Big data, big insight

For me, the lesson of Wednesday’s Data Deluge session was that “big data” is neither a problem or solution on its own, but a means to an end – “big insight”, if you will. As my fellow panellists demonstrated, it’s already happening.

True insight is incredibly difficult, because it lies buried in the quantification of everyday things. Let’s say you had records of everyone in the US with heart disease, or everyone with bad credit. Both datasets would be “big”, but without deeply nuanced understanding, we can only describe a problem. Refining data into meaningful experiments is a highly adaptive process, including the balance of human and computational insight required.

Microsoft’s Craig Mundie described how his team used machine learning to examine re-admittance, a top contributor to healthcare costs. Examining 10 years of congestive heart failure cases, their tool confirmed correlations doctors reported anecdotally, but also identified several overlooked scenarios. By deriving additional insight from suspected cofactors, doctors can decide when early intervention makes sense.

Jose Ferreira’s company, Knewton, helps teachers to personalize content and approaches based on what is most effective in a given context. Knewton products include a recommendation engine that enhances other teaching tools, and a complete math course. This approach combines two massive levers – technology and education – to derive insight on the nature of insight, suggesting profound implications for a wide spectrum of endeavours.

We also learned this surprising figure: 1.5 million people move into cities every week. The downsides are obvious: overcrowding, crime, strained infrastructure and disease vectors all come to mind. Yet, this presents untold opportunities to optimize public decision-making. New York City’s government is using existing data to improve smart resource allocation, public health and safety – benefits that will no doubt multiply as these concepts spread and evolve.

Natural disasters are another big data frontier, as I saw when Palantir partnered with the veterans’ organization Team Rubicon to assist in Hurricane Sandy relief efforts. It began with identifying the hardest-hit areas and greatest needs. As help poured in, the data landscape shifted to tracking projects, allocating manpower and coordinating 10,000 volunteers in real time.

No discussion of big data would be complete without addressing governance. Alan Murray, President of the Pew Research Center, noted that most people wouldn’t want their medical records to be generally accessible, but if they were hit by a bus, they would absolutely want the data made available to emergency personnel. Mirroring the challenges described above, governance requires both force and nuance, adaptations and fail-safes.

If any of this sounds too Utopian, we should acknowledge that getting from big data to big insight will be inherently chaotic, as evolutionary processes always are. Generating order to match the chaos will require us to reach the limits of human and technical ingenuity – yet, we may also discover that those limits are greater than we ever imagined.

Author: Shyam Sankar is the President of Palantir Technologies, a company working to radically change how groups analyse information. Founded in 2004 and headquartered in Palo Alto, California, the company currently offers Palantir Government and Palantir Finance, platforms for integrating, visualizing and analysing the world’s information. Palantir Government is broadly deployed in the intelligence, defence and law enforcement communities. Palantir Finance is in use at some of the world’s leading hedge funds and financial institutions. 

Image: A man looks at computer monitors in Tokyo REUTERS/Nicky Loh