Skip to main content

Big Data - Big Limits

It must be more than twenty years since I predicted that one day, we would be able to cram all the world's data into a single optical disk. I recall, that I was writing an article about Apple Computer, perhaps it was for Computer Weekly, with Apple very much a niche player at the time and Steve Jobs, as ever, well ahead of the curve, had expressed real excitement about the technology.

Of course, I was wrong in one very important respect. All the world's data or the sum of human knowledge, twenty years ago, is only a fraction of what it is today, at a time when IDC reports that three exabytes of new data are created each twenty-four hours. That's two to the sixtieth bytes or simply a billion gigabytes; I'm told about 50,000 years worth of DVD quality video. As volumes of data continue to grow at near exponential rates, the prevailing 'Big Data' debate surrounds the very real challenge of storing and making sense of it all, let alone the the privacy implications of capturing every keystroke and query for pattern and behaviour analysis.

In October's ARC Magazine, Samuel Arbesman worries that soon, we will no longer be able to understand a large fraction of the knowledge we have created and in 'Scientific American 2010' Danny Hillis makes a similar point. Hillis argued that that we have moved from the Enlightenment, a period where logic and reason could bring understanding, to the Entanglement, where everything is so unbelievably interconnected, that we can no longer understand systems of our own making.

Arbesman speculates that for a greater part of human history, the vast majority of humanity has understood its surroundings according to the knowledge of the day: "From the four elements to the workings of the screw and the pulley, a significant fraction of the world's knowledge was within the grasp of most individuals." He adds: "As our world as become more complex and knowledge has increased rapidly, a smaller and smaller fraction of society has felt it has a true-enough understanding of everything."

This is a theme I would like to explore a little further in the weeks and months ahead, as I think further on the nature of physical limits and our ability to draw meaningful and useful conclusions from seemingly infinite volumes of human behaviour, expressed and delivered in neat digital packets.

Popular posts from this blog

Mainframe to Mobile

Not one of us has a clue what the world will look like in five years’ time, yet we are all preparing for that future – As  computing power has become embedded in everything from our cars and our telephones to our financial markets, technological complexity has eclipsed our ability to comprehend it’s bigger picture impact on the shape of tomorrow.

Our intuition has been formed by a set of experiences and ideas about how things worked during a time when changes were incremental and somewhat predictable. In March 1953. there were only 53 kilobytes of high-speed RAM on the entire planet.

Today, more than 80 per cent of the value of FTSE 500* firms is ‘now dark matter’: the intangible secret recipe of success; the physical stuff companies own and their wages bill accounts for less than 20 per cent: a reversal of the pattern that once prevailed in the 1970s. Very soon, Everything at scale in this world will be managed by algorithms and data and there’s a need for effective platforms for ma…

The Big Steal

I’m not here to predict the future;” quipped the novelist, Ray Bradbury. “I’m here to prevent it.” And the future looks much like one where giant corporations who hold the most data, the fastest servers, and the greatest processing power will drive all economic growth into the second half of the century.

We live in an unprecedented time. This in the sense that nobody knows what the world will look like in twenty years; one where making confident forecasts in the face of new technologies becomes a real challenge. Before this decade is over, business leaders will face regular and complex decisions about protecting their critical information and systems as more of the existing solutions they have relied upon are exposed as inadequate.

The few real certainties we have available surround the uninterrupted march of Moore’s Law - the notion that the number of transistors in the top-of-the-line processors doubles approximately every two years - and the unpredictability of human nature. Exper…

An Ockham of Gatwick

The 13th century theologian and philosopher, William of Ockham, who once lived in his small Surrey village, not so very far from what is today, the wide concrete expanse of Gatwick airport is a frequently referenced source of intellectual reason. His contribution to modern culture was Ockham’s Razor, which cautions us when problem solving, that “The explanation requiring the fewest assumptions is most likely to be correct;” sound advice which constantly proves to be true.

A week further-on since Britain’s second busiest airport was bought to a complete standstill by two or perhaps two hundred different drone sightings, it is perhaps time to revisit William of Ockham’s maxim, rather than be led astray by an increasingly bizarre narrative, one which has led Surrey police up several blind alleys with little or nothing in the way of measurable results.

 Exploring the possibilities with a little help in reasoning from our medieval friar, we appear to have a choice of two different account…