Skip to main content

Big Data - Big Limits

It must be more than twenty years since I predicted that one day, we would be able to cram all the world's data into a single optical disk. I recall, that I was writing an article about Apple Computer, perhaps it was for Computer Weekly, with Apple very much a niche player at the time and Steve Jobs, as ever, well ahead of the curve, had expressed real excitement about the technology.

Of course, I was wrong in one very important respect. All the world's data or the sum of human knowledge, twenty years ago, is only a fraction of what it is today, at a time when IDC reports that three exabytes of new data are created each twenty-four hours. That's two to the sixtieth bytes or simply a billion gigabytes; I'm told about 50,000 years worth of DVD quality video. As volumes of data continue to grow at near exponential rates, the prevailing 'Big Data' debate surrounds the very real challenge of storing and making sense of it all, let alone the the privacy implications of capturing every keystroke and query for pattern and behaviour analysis.

In October's ARC Magazine, Samuel Arbesman worries that soon, we will no longer be able to understand a large fraction of the knowledge we have created and in 'Scientific American 2010' Danny Hillis makes a similar point. Hillis argued that that we have moved from the Enlightenment, a period where logic and reason could bring understanding, to the Entanglement, where everything is so unbelievably interconnected, that we can no longer understand systems of our own making.

Arbesman speculates that for a greater part of human history, the vast majority of humanity has understood its surroundings according to the knowledge of the day: "From the four elements to the workings of the screw and the pulley, a significant fraction of the world's knowledge was within the grasp of most individuals." He adds: "As our world as become more complex and knowledge has increased rapidly, a smaller and smaller fraction of society has felt it has a true-enough understanding of everything."

This is a theme I would like to explore a little further in the weeks and months ahead, as I think further on the nature of physical limits and our ability to draw meaningful and useful conclusions from seemingly infinite volumes of human behaviour, expressed and delivered in neat digital packets.

Popular posts from this blog

A Christmas Tale

It’s pitch blackness in places along the sea wall this evening and I'm momentarily startled by a small dog with orange flashing yuletide antlers along the way. I’m the only person crazy enough to be running and I know the route well enough to negotiate it in the dark, part of my Christmas exercise regime and a good way of relieving stress.

Why stress you might ask. After all, it is Christmas Day.

True but I’ve just spent over two hours assembling the giant Playmobil ‘Pony Farm’ set when most other fathers should be asleep in front of the television.



I was warned that the Playmobil ‘Pirate Ship’ had driven some fathers to drink or suicide and now I understand why. If your eyesight isn’t perfect or if you’ve had a few drinks with your Christmas lunch then it’s a challenge best left until Boxing day but not an option if you happen to have a nine year old daughter who wants it ready to take horses by tea time.

Perhaps I should stick to technology but then, the instruc…

Merlins over Thanet

Marooned, temporarily at Manston this afternoon are the Merlins over Malta team on the way to the Mediterranean for a display to mark the historic Second World War defence of the island.


Charlie Brown

Unfortunately, the weather over Thanet is appalling this afternoon and the Spitfire and Hurricane can’t get airborne again until it clears, so the celebrity Battle of Britain aircraft pilots, Charlie Brown, Clive Denny and their team-mates are contemplating an evening among the fleshpots of Margate.


Clive Denny (Hurricane) & Charlie Brown (Spitfire) Pilots

I’m rather hoping the weather it will clear through though as they have to get to Jersey before dusk if possible and I have to take some photos of the Spitfire and Hurricane for Pilot Magazine and I’ve always wanted a chance to get in either aircraft!

An Interview with Charlie Brown

They just got off, squadron scramble or what? They were ready and gone in ten minutes towards the nearest patch of blue sky!

An interview with the legendary S…
Median Saleh

I mentioned in the last post, the 1981 expedition that took in Median Saleh, the ruined Nabatean city in Saudi Arabia


A temple carved from the rock from Petra's sister city.

By coincidence, one of the most important train stations on the Hejaz railway sat next to the ruins and when Lawrence of Arabia blew the line in 1917, the trains were trapped there and are still there today, gathering dust and with "Krupp" on the engine casings.


One of the trains, sitting where T.E. Lawrence left themwith Dr Paul Garnett as the passenger

Below, you can see one of the fortified train stations that Lawrence attacked along the Hejaz railway between Damascus and Medina.



More photos Medain Saleh can be found on THIS Site - Apparently you can catch a tourist bus these days, rather different from risking life and limb to cross an unfriendly Saudi Arabia twenty years ago!