Falling behind: Big data and the analysis race

'Big data' is not a new concept. In fact, it is the processing of big data that determines your suggested books on Amazon, or the advertisement that plays before your YouTube video. Digital corporations such as Google and IBM have been harnessing big data for much longer than mainstream academia has been discussing it, the latter for at least two decades. Big data is, in its simplest form, the analysis of extremely large data sets to reveal patterns, behaviour and trends, especially in a human behavioural sense. So why has it only been noticed by government institutions and think-tanks over the last two years?

In 2013, DARPA Director Regina Dugan told US Congress that 'detecting dismounted fighters in Afghanistan required collecting and processing about 100,000 times more data than it takes to detect a strategic bomber'. By international standards, the population of Afghanistan has an internet penetration rate of 5.5 per cent, placing it in the bottom 10 per cent of connected countries in the world. This figure reflects urbanised areas of the country, not that part where the majority of fighting occurred in the ten-year conflict. There is little disagreement that conflict in Afghanistan was largely rural and digitally disconnected, but is unlikely, given current connectivity trends, that Australia will fight in such a disconnected space again.

What about the future that is described in the Future Land Warfare Report – the urban littoral operating environment? Does Army have the processing power required to store and, perhaps more importantly, analyse digital information in the most densely connected part of the globe?

David Kilcullen recently noted that connectivity has grown so quickly over the last decade, that the Asia–Pacific region now accounts for more than half of the mobile-cellular subscriptions across the globe (3.5 billion out of 6.8 billion). The worldwide connectivity boom has brought with it a number of significant challenges for both government and law-enforcement agencies across the globe, particularly in the tracking, collecting and storing of data. Realisation of the potential of big data analytics is only beginning, particularly as more interest and subsequent investment is stored in practical systems like IBM's Watson or i2. For Army, this means that the effective analysis of open-source information may identify patterns of behaviour and therefore support ground troops, enhance planning and facilitate the propagation of lessons in real-time.

The presence of digital mechanisms within the battlespace for both combatants and civilians will increase in time, but the intelligence edge on the battlefield will only be gained with dynamic storage and effective analysis of information. If the enemy can deliver orders and expedite movement over mobile phones faster than you can analyse and interpret that information, what then is the point of analysing that information in the first place? And that was only one phone call. Development and integration of automated data analysis systems needs to occur before it's too late, particularly as the proliferation and subsequent use of commercial technology outpaces that of government agencies.

Lieutenant Hayden Lammiman is currently posted as a Platoon Commander in Bravo Company, 2nd Battalion, Royal Australian Regiment.

The views expressed in this article and subsequent comments are those of the author(s) and do not necessarily reflect the official policy or position of the Australian Army, the Department of Defence or the Australian Government. Further information.


  • response from Major Mark Mankowski, 13 March 2015

    I really enjoyed Lieutenant Hayden Lammiman’s article 'Falling behind:Big data and the analysis race'. It did an excellent job of introducing a technology or system that Army needs to grapple with in the future.

    Hayden argues that 'Development and integration of automated data analysis systems needs to occur before it's too late'. A solution is in development for Australia, but we probably do not even know it. As a former scientist I am familiar with the recent announcement that Australia will be collaborating in the Square Kilometre Array(SKA), which is next-generation radio telescope project. However, when I heard about the project, its relevance to the Army was not clear. It was only reading Hayden's article that the application for this science in the military became clear.

    SKA's website highlights it is a big data instrument. The antenna is not a square kilometre, but composed on many small antenna’s that combined make up the SKA. Multiple antennas means that the SKA will gather a lot of data. The quantity of data coming in will be staggering and there is only a limited amount of space available. As a consequence, it will be necessary to process and store the useful data and then delete the rest quickly so that more data can be collected. The article further adds that the scientists will have to write very sophisticated computer programs to be able to search for what they want in the stored data, like a needle in a very big haystack.

    This sounds like the solution to intelligence’s big data issues. Already the computer techniques have been demonstrated to have relevance to scientific fields outside of astronomy. Scientists at Cambridge University have used successfully astronomers' techniques for sifting through data in cancer research.

    What Hayden’s article has shown is that the Army requires both a forum for debate and a workforce that is scientifically literate. The forum is useful for highlighting future challenges and scientific soldiers can contribute technological breakthroughs and their potential application in the military.