Big data holds critical importance in the current generation of information technology, with applications ranging from financial, industrial, academic to defense sectors. With the exponential rise of open source data from social media and increasing government monitoring, big data is now also linked with national security, and subsequently to the intelligence community. In this study I review the scope of big data sciences in the functioning of intelligence community. The major part of my study focuses on the inherent limitations of big data, which affects the intelligence agencies from gathering of information to anticipating surprises. The limiting factors range from technical to ethical issues connected with big data. My study concludes the need of experts with domain knowledge from intelligence community to efficiently guide big data analysis for timely filling the knowledge gaps. As a case study on limitations of using big data, I narrate some of the ongoing work in nuclear intelligence using simple analytics and argue on why big data analysis in that case would lead to unnecessary complications. For further investigation, I highlight cases of crowdsource forecasting tournaments and predicting unrest from social media.
By Karan Jani
Sam Nunn Security Program
Sam Nunn School of International Affairs Georgia Institute of Technology
October 26, 2016
Link to the publication: https://arxiv.org/pdf/1610.08629.pdf
About the author
Karan Jani is a doctoral candidate in the Center for Relativistic Astrophysics at Georgia Tech. He works on the international relations and security concernes with developing nation's space-science program and the role of big data sciences for intelligence community.