Skip to main content

Big Data - Big Limits

It must be more than twenty years since I predicted that one day, we would be able to cram all the world's data into a single optical disk. I recall, that I was writing an article about Apple Computer, perhaps it was for Computer Weekly, with Apple very much a niche player at the time and Steve Jobs, as ever, well ahead of the curve, had expressed real excitement about the technology.

Of course, I was wrong in one very important respect. All the world's data or the sum of human knowledge, twenty years ago, is only a fraction of what it is today, at a time when IDC reports that three exabytes of new data are created each twenty-four hours. That's two to the sixtieth bytes or simply a billion gigabytes; I'm told about 50,000 years worth of DVD quality video. As volumes of data continue to grow at near exponential rates, the prevailing 'Big Data' debate surrounds the very real challenge of storing and making sense of it all, let alone the the privacy implications of capturing every keystroke and query for pattern and behaviour analysis.

In October's ARC Magazine, Samuel Arbesman worries that soon, we will no longer be able to understand a large fraction of the knowledge we have created and in 'Scientific American 2010' Danny Hillis makes a similar point. Hillis argued that that we have moved from the Enlightenment, a period where logic and reason could bring understanding, to the Entanglement, where everything is so unbelievably interconnected, that we can no longer understand systems of our own making.

Arbesman speculates that for a greater part of human history, the vast majority of humanity has understood its surroundings according to the knowledge of the day: "From the four elements to the workings of the screw and the pulley, a significant fraction of the world's knowledge was within the grasp of most individuals." He adds: "As our world as become more complex and knowledge has increased rapidly, a smaller and smaller fraction of society has felt it has a true-enough understanding of everything."

This is a theme I would like to explore a little further in the weeks and months ahead, as I think further on the nature of physical limits and our ability to draw meaningful and useful conclusions from seemingly infinite volumes of human behaviour, expressed and delivered in neat digital packets.

Popular posts from this blog

Civilisational Data Mining

It’s a new expression I haven’t heard before. ‘Civilisational data mining.’

Let me start by putting it in some context. Every character, you or I have typed into the Google search engine or Facebook over the last decade, means something, to someone or perhaps ‘something,’ if it’s an algorithm.


In May 2014, journalists revealed that the United States National Security Agency, the NSA, was recording and archiving every single cell-phone conversation that took place in the Bahamas. In the process they managed to transform a significant proportion of a society’s day to day interactions into unstructured data; valuable information which can of course be analysed, correlated and transformed for whatever purpose the intelligence agency deems fit.

And today, I read that a GOP-hired data company in the United States has ‘leaked’ personal information, preferences and voting intentions on… wait for it… 198 million US citizens.

Within another decade or so, the cost of sequencing the human genome …

The Nature of Nurture?

Recently, I found myself in a fascinating four-way Twitter exchange, with Professor Adam Rutherford and two other science-minded friends The subject, frequently regarded as a delicate one, genetics and whether there could exist an unknown but contributory genetic factor(s) or influences in determining what we broadly understand or misunderstand as human intelligence.

I won’t discuss this subject in any great detail here, being completely unqualified to do so, but I’ll point you at the document we were discussing, and Rutherford’s excellent new book, ‘A Brief History of Everyone.”

What had sparked my own interest was the story of my own grandfather, Edmond Greville; unless you are an expert on the history of French cinema, you are unlikely to have ever hear of him but he still enjoys an almost cult-like following for his work, half a century after his death.

I've been enjoying the series "Genius" on National Geographic about the life of Albert Einstein. The four of us ha…
The Mandate of Heaven

eGov Monitor Version

“Parliament”, said my distinguished friend “has always leaked like a sieve”.

I’m researching the thorny issue of ‘Confidence in Public Sector Computing’ and we were discussing the dangers presented by the Internet. In his opinion, information security is an oxymoron, which has no place being discussed in a Parliament built upon the uninterrupted flow of information of every kind, from the politically sensitive to the most salacious and mundane.

With the threat of war hanging over us, I asked if MPs should be more aware of the risks that surround this new communications medium? More importantly, shouldn’t the same policies and precautions that any business might use to protect itself and its staff, be available to MPs?

What concerns me is that my well-respected friend mostly considers security in terms of guns, gates and guards. He now uses the Internet almost as much as he uses the telephone and the Fax machine and yet the growing collective t…