Skip to content

Reading the conversation cloud

For those of you who know Hari Seldon, he needs no introduction. But for those who don’t, let me briefly introduce him. Hari Seldon was the star of Isaac Asimov’s Foundation Trilogy, a science-fiction masterwork which asked the question: What if we could predict the future the way we can predict the weather? Seldon, a fictional mathematics professor, developed a science called psychohistory, which allowed him to predict the future in probabilistic terms. That is, not exactly, but based on statistics and probability, he could predict the most probable futures, the way we predict the weather today.

Map of the Foundation galaxy
Map of the foundation galaxy by Cygnus.

It’s not hard to imagine that such a thing might be possible, if only we had the data.

Predicting the weather

Predictions and data go hand in hand. Consider the history of weather forecasting. For millennia, we tried to predict the weather in informal ways, by reading the clouds, throwing sticks, stargazing, and observing seasonal cycles and patterns.

But in the 1800s the electric telegraph was invented, and suddenly it became possible to share temperature and weather data across distances. Once we had the data things started to change.

Weather map from 1906
Weather map from 1906 by Justhus Perthes.

We began to apply a scientific approach to understanding and predicting the weather, by collecting quantitative measurements and using those measurements to develop models of atmospheric processes. Perhaps it is not a coincidence that Asimov’s Foundation Trilogy was published in the 1950s, just as weather forecasting was coming into its own as a reliable predictor of atmospheric activity.

In the 1950s, the development of computers meant that we could make more accurate, quantitative predictions on a regular basis. Today, these algorithms have become so sophisticated that many predictions are based on computer models that crunch huge volumes of data, and human forecasters make their predictions by choosing between multiple computer-generated options. That data and those models are what we now use to make ever-more-precise predictions of weather around the globe.

Predicting the social weather

Today, the people of the world are generating social data at a pace never before imagined, except perhaos by Isaac Asimov and his invented protégé, Hari Seldon.

Facebook’s 750 million users create and share a billion pieces of content every day. Twitter users tap in 350 billion tweets per day. Every two days, more information is created than between the dawn of civilization and 2003. One out of six minutes people spend online is spent on social media.

All this social activity generates a massive cloud of data, representing the social activity of the planet: Status messages, tweets and retweets, likes, plus-ones, comments, pokes, photo uploads, check-ins and so on. Like a global weather pattern, the conversation cloud is readable if we only had the tools to do so.

If only we had the data.

When it comes to the conversation cloud, we are using methods more appropriate to ancient times. We are throwing sticks and trying to read the stars. We try to read this whirl of information and look for patterns, but so far there has not been any systematic, quantitative approach to collecting and interpreting social data. The cloud is there. But every day a treasure trove of data is generated, only to slowly dissipate, like smoke on a windy day.

We can’t look back at historical data. We can’t search for patterns in the data. We can’t build predictive models. All because we just don’t have the data. And until we have consistent, reliable data, these things just aren’t going to happen.

If only we had the data.

Well now we do.

At the Dachis Group, we have been developing just such a system, a massive data repository we call Social Business as a Service (SBIaaS), collected from hundreds of millions of sources all over the web. SBIaaS is a platform that provides real-time social business data on demand.

Today we launch our first application that taps into this data service, the Social Business Index (SBI). The Social Business Index tracks more than 20,000 companies, providing real-time ranking, analysis and benchmarking of their social business adoption and performance. I hope you will pardon the expression, but it’s pretty effing cool.

The Social Business Index. Check it out. It’s free :)

Predicting the future

Social business is a young field. We don’t yet have well-formed metrics and performance indicators than easily be compared across industries. We’re still a long way from being able to make accurate predictions and social forecasts, but the SBI is a good start. It monitors conversations initiated by companies in real time. It measures engagement levels, response times, conversation volume and velocity. It tracks social business initiatives on Facebook, Twitter, YouTube, blogs, forums and other social platforms. It ranks companies by social engagement and execution and updates its rankings dynamically, based on real-time information.

SBIaaS is a platform that provides Data as a Service (DaaS), and the Social Business Index is just the first of many applications. There will be future developments so stay tuned.

I think Hari Seldon would be proud.

As always, your comments, thoughts and feedback are much appreciated.


Keep in touch! Sign up to get updates and occasional emails from Dave.

One Comment

  1. These are exciting times and the idea of generating meaning and predictive insight from the masses of social data generated by our networked lives is intriguing.

    After the development of methods for converting the measurement of meteorological data into predictive tools, there have been two big issues for accuracy in predicting weather: comprehensiveness of data measurement and computing power.

    The “butterfly effect” posited by Edward Lorenz was later held to create intractable problems for computation. This means, so the hypothesis goes, that due to the complexity of the interdependencies in the physical systems that make weather, we will only ever be able to accurately predict weather within a three day window.

    Of course, that degree of accuracy turns out to be plenty helpful for all sorts of things. So the fact that we cannot have perfect knowledge of the future of weather is, to put it scientifically, no biggie.

    The tempting world of Hari Seldon and the poetic phrase “social weather”, notwithstanding, I wonder what it is we mean by “social” these days. Surely, what we mean is not exactly what we used to mean, when some of our daily activities left no trace.

    As our lives steadily throw off more and more data, there will be many new questions to ask ourselves. Does that which can be measured or tracked matter more than what can’t or isn’t. What do we want to know about each other and ourselves and why?

    I think that emerging “windows” like the Social Business Index are fascinating. I also think that we must be careful and thoughtful in considering what it is that they allow us to see.

    The world of data and its measurement and meaning is exploding. Experiments like those at IBM’s Many Eyes lab, the Sunlight Foundation and those conducted by design firms like Stamen, not to mention the department at the New York Times, along with the SBI from Dachis are harbingers, to be sure. But of what?

    Wednesday, September 28, 2011 at 5:54 am | Permalink