Big Data Hasn’t Changed Everything

  • Big Data Hasn’t Changed Everything

    Technology has a long way to go in mapping the variables of humanlife.



    The more I hear the term “big data,” the more suspicious I become. Not in an Edward Snowden, the evil government’s spying on us sort of way. If the curious of Fort Meade, Md., the National Security Agency’s home, wish to poke through my electronic sock drawers for signs of terror, they are more than welcome. Happy to do my bit for national security.

    No, the problem comes when the term becomes ubiquitous. It’s one thing for the NSA’s quants or scientists at the Large Hadron Collider or genome sequencers to talk about big data. Big nails need big hammers, and the phenomenon of big data is certainly real. The three Vs of volume, velocity and variety, coined by the techies at the Gartner GroupIT -0.47% have created a gusher of data that clever minds can use to great effect.

    But it’s quite another thing when you start to hear how big data is going to upend everything. In their essential new guide to the subject, “Big Data: A Revolution That Will Transform How We Live, Work and Think,” Viktor Mayer-Schönberger and Kenneth Neil Cukier write: “Society will need to shed some of its obsession for causality in exchange for simple correlations: not knowing why but only what. This overturns centuries of established practices and challenges our most basic understanding of how to make decisions and comprehend reality.” We need to learn to trust what the data is telling us before we fully understand why.

    They add that such a drastic change to how we weigh evidence and decide will take a lot of fine tuning. Ethics, morality, civil liberties, everything risks being thrown under the big-data bus, unless we are exceedingly careful.

    Getty Images

    Fortunately we have one giant data set around which to pivot this discussion. In the 1980s, the financial industry was transformed by the arrival of what at the time seemed like very big data indeed. Brokers were removed from trading floors and replaced by digital exchanges. Money which once bumped into borders now started to flow unimpeded. Myriad frictions were removed and the industry boomed. Computer scientists started popping up inside banks and hedge funds to identify trading opportunities within the great torrent of data. Many were and are truly brilliant. In a war, you would want them on your side breaking enemy codes.

    Things went awry when everyone started to think they could do it. When the decidedly adult pursuits facilitated by the new torrent of data fell into the hands of intellectual children, who had no real idea of what they were doing. When the likes of Citibank and Societe Generale GLE.FR -1.18% started to think they could play with the best of the hedge funds was the time to start stuffing your cash in a mattress. When all you do is watch data models for instructions, for whats, with no idea of the whys, you become easy prey for the monsters of risk.

    What seems blindingly obvious in retrospect, that there was no alchemy capable of turning subprime loans into sure-thing derivatives, was either missed or intentionally glossed over by Wall Street’s big-data thinking.

    Managers are constantly being told that they are one hardware or software installation away from business nirvana. But they should bear in mind the lessons of the financial crisis the next time a consultant waltzes into their office declaring big data the next big thing, or even worse, a “paradigm shift.”

    Because big data as a technological opportunity and big data as a management theory are two separate things. However much big data can yield, information will never be perfect. As efficient as these data models become, managers will still have to make decisions with limited certainty about the outcomes. Data helps and has since the scouts of ancient armies returned with reliable numbers. Eisenhower at D-Day had more data than Hannibal at Cannae, but waging war remained a beast of a task. The challenge for managers has always been the human mind and heart, which seems punier than ever in the shadow of the terabyte.

    Consultants are already whipping up a flurry of winking big-data dashboards for managers, with every organizational activity reduced to a few key numbers. But still there will be rogue traders, rats in restaurant kitchens, and drunk machine operators. If big data is to escape the graveyard of managerial fads—knowledge management anyone?—it has a lot to prove.

    Data has been big for a while now and is getting exponentially bigger. But we shouldn’t feel inadequate because we rely on our animal traits like gut, intuition and bias. Technology has a long way to go in mapping the variables of human life. And the moment it starts to feel like tyranny, we have one lethal weapon in our arsenal. It is called the off switch.

    Mr. Broughton is the author of “The Art of the Sale: Learning from the Masters about the Business of Life” (Penguin Press, 2012).

    • 20 hours ago

    If you torch the inputs by bribing the rating agencies, Big Data will give you garbage.

    1 Recommendation

    • 19 hours ago

    Big data creates the need for more powerful analytic methods able to crack increasing complexity. It’s a never ending chase where mere humans lag way behind the very few schooled in the new tools. The end result is collapse.


    • 19 hours ago

    <<Managers are constantly being told that they are one hardware or software installation away from business nirvana. >>

    Yep. I’ve been hearing that for decades. It works wonders for computer vendors’ hardware and software sales, though.

    For the record, “big data” is the industry’s latest buzzward to identify data analysis techniques that have been used for decades. What’s happened recently is that computer hardware has become fast enough to crunch millions and billions of data records in a reasonable amount of time, so the illusion has been spawned by IT sales reps that “crunching big data” is the latest magic bullet that will fix whatever ails dysfunctional businesses without their managements having to do any real work.

    The problem, of course, is there is a limit to the “resolution” of the data. Just crunching numbers faster does not always, or even usually, yield meangful results. An analog is that the usefulness of a telescope is limited by the size of its mirror. You can only add powers of magnification up to the limit of the mirror’s ability to resolve distant objects. It’s useless to add a magnification power of 1000x to a 3 inch mirror because all’s you’ll see is a big meaningless blur.

    “Big data” is like that too. You can “drill down” into the data all you want, but if the data isn’t useful enough to “resolve” anything, you’re not going to get meaningful results. You can ask Mitt Romney all about that one. He hired the best wonder boys money could buy to program his computers to “crunch big data” in order to turn out the vote, and, if anything, they caused him to lose by a larger margin than he would have lost without them.

    Not picking on Mitt Romney, but I am sure that his campaign people placed a false sense of security in relying on computers to do what their candidate COULDN’T do, which was to turn out the voters. They should have spent zero dollars on computer systems and a couple thou on a P.R. firm to hone the image.

    Business does that all the time too. They expect that a computer system is going to sort out their bogus procedures that are due to sloppy management. No, it doesn’t work that way. You fix your procedures FIRST, then invest in computer systems. Even “big data” won’t bail out a dysfunctional management.

    6 Recommendations

      • 17 hours ago

      Actually, there are a lot of new techniques out there, too. This company has something different, and I’ve been using their tools

      But in general I agree. You have to actually have data containing the signal you want to find in order to find anything. And looking for signals in the same old ways isn’t likely to help – if the signal wasn’t there when you had 1 million samples, why will it be there when you have a quadrillion?

      1 Recommendation

      • 2 hours ago

      I’m not sure your example of Mitt Romney in the most recent presidential election fits or enhances your argument. I recently heard a discussion involving a strategist for the Obama campaign, describing how their use of sophisticated data analysis techniques allowed them to identify the “persuadable” voters in the districts that could sway the election and concentrate their efforts where they would get the biggest payoff. This was of course using computing power to sift through the huge data bases to isolate specific instances, not general trends. But it was “Big Data” and modern information technology that made it possible.


    • 18 hours ago

    I’ve worked with small, medium, large, and big data my whole career. There are many problems with it, the biggest being correlation doesn’t mean causation, not to mention non-normally distributed samples and universes. I did a very simple lecture at a So Cal university to explain this using cars in one example. Take a 500 person lecture hall and ask all the people who drive a Honda to stand up. Then ask each their demographic and why they chose a Honda. You had the few that match Honda’s positioning (value, reliability, mileage), but the vast majority of those standing had no correlation to anyone else nor did they correlate to Honda’s positioning. While you can detect interesting things in the data, and there actually have been successes in NSA identification of terrorists looking at “outlier” data, Big Data has a long way to go, and understanding what the data means once you see the results is an art as much as a science. And I haven’t even talked about the meaning of missing or no data and its effect. Then, of course, there’s the issue of getting your organization to act or change based on it. If you don’t get people trained in critical thinking to manage the “art”, the science/Math behind it isn’t of much value.

    2 Recommendations

      • 18 hours ago

      The classic example of this for those of us who grew up in the ’60s is the “Input / Output Analysis” that LBJ’s “Whizzkid” Secretary of War Robert McNamera tried to apply to his computerized models of the Vietnam War. McNamara was confident, based on his computer analysis, that the U.S. would prevail in Vietnam if we killed off “x” number of NVA and Viet Cong.

      That theory didn’t survive contact with reality. McNamara was discredited after he and his boss LBJ got a lot of our best young men killed in nondecisve combat operations that killed all the requisite NVA/VC that McNamara’s model said needed to be killed, but without causing them to call off the war.

      McNamara learned too late that if you’re going to rely on data analysis to make your decisions, you’d better make sure that you have ALL the inputs and outputs to model. Human behavior is notoriously fickle to model in a computer as McNamara and many who followed him have learned to their chagrin.

      2 Recommendations

        • 7 hours ago

        And yet even today McNamara is viewed as a visionary… the left, at least.


        • 3 hours ago

        Having grown up in the Vietnam era of body counts on the news every night, If found Sorley’s 2011 “Westmoreland: The General Who Lost Vietnam” very interesting. Westmorland was the perfect enabler of LBJ’s and McNamara’s schemes and biases.


    • 14 hours ago

    Read Nate Silver’s “The Signal and the Noise”. Big data means more signals and more noise.

    1 Recommendation

    • 5 hours ago

    The problem isn’t big data. The problem is management fads. Big data, of course, retains its significant (though limited) usefulness. And if your management loves fads, that’s the real problem…


    • 5 hours ago

    The article seemed interesting until he gets to the part about banks trying to use big data to make sub-prime mortgages work by creating CMO’s. He implies that they were driven to this by something they saw in “Big Data”. The truth is sub-prime mortgages were jammed down their throats by Congress as a result of changes made to the Community Reinvestment Act when President Clinton signed the Graham, Leach, Bliley bill in 1998. Andrew Cuomo was Secretary of HUD at the time and he put the sub-prime lending program into warp drive once that bill was passed. All that the banks tried to do was find ways to layoff their risk using CMO’s and Credit Default Swaps. Guess what? That didn’t work!


    • 5 hours ago

    Can you trust the data when the desired meta-analysis is all skewed up:

    Tarnished Gold: The Sickness of Evidence-Based Medicine

    When the data generates statistics which become the average patient, then one can treat the average not the individual. This is anathema to P4 medicine which is Personalized, Predictive , Preventive and Participatory.

    See also, 50 studies every doctor should know:

    Has anyone met the average human. That is someone with one breast and one testicle?


    • 2 hours ago

    Great piece. All of business wants to embrace big data while forgetting that data does not equal insight. Data can point you to correlations that may or may not be worth looking into, but data will not yielded any insight into why those things are correlated which is what you really need to make it actionable.

    I’ve also seen teams and companies try to be “data driven” only to run smack into a wall when they realize that the data will not literally tell them what to do. Data can inform strategy, illuminate possible paths and outcomes, but it cannot tell you definitively which path to follow.

    One could make an argument that American business executives have unprecedented amounts of data at their fingertips, yet are making worse decisions than ever before because any notion of judgment, management decisions based on core principles, or old fashioned leadership are rapidly being lost.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s