Big Data Quotes

More data—such as paying attention to the eye colors of the people around when crossing the street—can make you miss the big truck.

Used in combination with genomics, AI could help pharma companies to develop new drugs for rare diseases. The rarer a disease is, the smaller the market is and so the less likely it is to have been addressed. Big pharma is hesitant to take on the high development costs for new drugs if there’s no sign of a return on investment. Biological processes are complex, and that means that they lead to multidimensional data that human beings struggle to wrap their heads around. The good news is that AI is the perfect tool to spot patterns in this kind of data.

Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.

No matter how powerful a computer you have, if you put lousy data in you will get lousy predictions out.

The next Freud will be a data scientist. The next Marx will be a data scientist. The next Salk might very well be a data scientist.

More than a building that houses books and data, the library has always been a window to a larger world--a place where we've always come to discover big ideas and profound concepts that help move the American story forward. . . . . Libraries remind us that truth isn't about who yells the loudest, but who has the right information. Because even as we're the most religious of people, America's innovative genius has always been preserved because we also have a deep faith in facts. And so the moment we persuade a child, any child, to cross that threshold into a library, we've changed their lives forever, and for the better. This is an enormous force for good.

Big Data has announced the end of the person who possesses free will.

I sometimes suspect that inside every data scientist is a kid trying to figure out why his childhood dreams didn't come true.

Big Data processes codify the past. They do not invent the future.

Netflix learned a similar lesson early on in its life cycle: don’t trust what people tell you; trust what they do.

In this era of fake news and paid news artificial intelligence is more and more used as a political tool to manipulate and dictate common people, through big data, biometric data, and AI analysis of online profiles and behaviors in social media and smart phones. But the days are not far when AI will also control the politicians and the media too.

In distributed systems, suspicion, pessimism, and paranoia pay off.

A change in Quantity also entails a change in Quality

Big data does not predict anything beyond the assumption of an idealized situation in a stable system.

The moral of the story is that a NoSQL system may find itself accidentally reinventing SQL, albeit in disguise.

Google and Facebook don’t have “users” or “customers”. Instead, they have participants under machine surveillance, whose activities are algorithmically combined within Big Data silos.

We are now creating tame humans that produce enormous amounts of data and function as very efficient chips in a huge data-processing mechanism, but these data-cows hardly maximize the human potential. Indeed, we have no idea what our full human potential is, because we know so little about the human mind. And yet we don’t invest much in exploring the human mind, instead focusing on increasing the speed of our internet connections and the efficiency of our Big Data algorithms. If we are not careful, we will end up with downgraded humans misusing upgraded computers to wreak havoc on themselves and on the world.

data outlives code.

Serving humanity intelligently is held up as the “gold standard” of AI based systems. But, with the emergence of new technologies and AI systems with bio-metric data storage, surveillance, tracking and big data analysis, humanity and the society is facing a threat today from evilly designed AI systems in the hands of monster governments and irresponsible people. Humanity is on the verge of digital slavery.

Big Data algorithms might create digital dictatorships in which all power is concentrated in the hands of a tiny elite while most people suffer not from exploitation, but from something far worse – irrelevance.

It’s important to remember that big data all comes from the same place – the past. A new campaigning style, a single rogue variable or a ‘black swan’ event can throw the most perfectly calibrated model into chaos.

I am now convinced that Google searches are the most important dataset ever collected on the human psyche. This

In a system in which cheating is the norm, following the rules amounts to a handicap.

The greatest danger of Big data and Artificial Intelligence is robots and bots will track you and manipulate you in every step.

A reminder that Goodreads is owned by Amazon, and everything you do here supports Big Data and corporate surveillance. You should be concerned, especially if you read books about liberation. A friend recommended StoryGraph, a Black-owned independent alternative. Download your data and GTFO.

Code is not like other how-computers-work books. It doesn't have big color illustrations of disk drives with arrows showing how the data sweeps into the computer. Code has no drawings of trains carrying a cargo of zeros and ones. Metaphors and similes are wonderful literary devices but they do nothing but obscure the beauty of technology.

the next darwin is more likely to be a data wonk than a naturalist wandering through an exotic landscape

To clarify, *add* data.

So the problem is not the algorithms, or the big datasets. The problem is a lack of scrutiny, transparency, and debate.

As with Google, so with everyone else trying to use data to understand the world. The Big Data revolution is less about collecting more and more data. It is about collecting the right data. But

The merger of infotech and biotech might soon push billions of humans out of the job market and undermine both liberty and equality. Big Data algorithms might create digital dictatorships in which all power is concentrated in the hands of a tiny elite while most people suffer not from exploitation but from something far worse—irrelevance.

Here we see that models, despite their reputation for impartiality, reflect goals and ideology. When I removed the possibility of eating Pop-Tarts at every meal, I was imposing my ideology on the meals model. It’s something we do without a second thought. Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

Justice cannot just be something that one part of society inflicts on the other.

My advice was to start a policy of making reversible decisions before anyone left the meeting or the office. In a startup, it doesn’t matter if you’re 100 percent right 100 percent of the time. What matters is having forward momentum and a tight fact-based data/metrics feedback loop to help you quickly recognize and reverse any incorrect decisions. That’s why startups are agile. By the time a big company gets the committee to organize the subcommittee to pick a meeting date, your startup could have made 20 decisions, reversed five of them and implemented the fifteen that worked.

Tech’s love affair with the myth of meritocracy is ironic for an industry so in thrall to the potential of Big Data, because this is a rare case where the data actually exists. But if in Silicon Valley meritocracy is a religion, its God is a white male Harvard dropout.

Paradoxically, the sources available today (in the era of big data) are less precise than those that were available a century ago due to the internationalization of wealth, the proliferation of tax havens, and above all, lack of political will to enforce financial transparency, so it is quite possible that we are underestimating the level of wealth inequality in recent decades.

Sometimes the constraints that we live with, and presume are the same for everything, are really only functions of the scale in which we operate.

If you can't understand a study, the problem is with the study, not with you.

If you are going to use the results of market research to make a big business decision, then it’s a good idea to do quantitative research rather than qualitative.

Rightly or wrongly, people might reach the same conclusions about Big Data algorithms: they have lots of hitches, but we have no better alternative.

One big bonus: e-mail! Just like the days back on Hermes, I get data dumps. Of course, they relay e-mail from friends and family, but NASA also sends along choice messages from the public. I’ve gotten e-mail from rock stars, athletes, actors and actresses, and even the President. One of them was from my alma mater, the University of Chicago. They say once you grow crops somewhere, you have officially “colonized” it. So technically, I colonized Mars. In your face, Neil Armstrong!

The value we provide at Mayflower-Plymouth exists at the convergence of various technologies and studies including Blockchain, cryptography, quantum computing, permaculture design principles, artificial intelligence, stigmergy, forestry, economics, additive manufacturing, big data, advanced logistics and more.

What else can it predict?” Now the other jocks encircled her like a bullseye. “Any event with data,” Holly said and really felt the need to leave. This was a set-up. Big Bob grinned. “Like when I’ll get a date?” Holly’s smile slid across her face. “Low probability events are hard to forecast.” “Huh?” Josh punched his shoulder. “She means, you are not likely to get a date.

the hand has programmable fingerprints, a vibration motor, data interface capabilities-” “Wait. A vibration motor in your hand? Why?” “I’m a man, and I’m alone on the planet. Figure it out.

Big data is transitioning from a tool primarily for targeted advertising to an instrument with profound applications for diverse corporate sectors and for addressing chronic social problems.

We essentially had to build a docking mechanism between the two capsules. We didn't have to share a lot of data, and we did that at the height of the Cold War, which was pretty symbolic." –Bill Gerstenmaier

we dont pay for most digital services in dollars - but we do pay dearly, with our data and our attention. people are the resource that's being monetized. we think we are the consumers. in fact, we are the product.

Aim for simplicity in Data Science. Real creativity won’t make things more complex. Instead, it will simplify them.

Money is not a magic wand but a measuring stick, not wealth but a gauge of it.

whenever people meet you, they take an instant mental snapshot. That image of you becomes the data they deal with for a very long time.

The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.

However, when you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.

Big Data: by itself doesn't yield insights...information doesn't serve us...without knowing why....unless something transformative is brought to these data sets (to create) understanding Gathering data is easy, understanding why is hard.

The value we provide at Mayflower exists at the convergence of various new technologies and studies including Blockchain, cryptography, quantum computing, artificial intelligence, stigmergy, additive manufacturing, big data, advanced logistics and more.

In the Google era, Newton’s system of the world—one universe, one money, one God—is now in eclipse. His unitary foundation of irreversible physics and his irrefragable golden money have given way to infinite parallel universes and multiple paper moneys manipulated by fiat. Money, like the cosmos, has become relativistic and reversible at will.

The other buzzword that epitomizes a bias toward substitution is “big data.” Today’s companies have an insatiable appetite for data, mistakenly believing that more data always creates more value. But big data is usually dumb data. Computers can find patterns that elude humans, but they don’t know how to compare patterns from different sources or how to interpret complex behaviors. Actionable insights can only come from a human analyst (or the kind of generalized artificial intelligence that exists only in science fiction).

Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.

Never compare your insides to everyone else’s outsides.

The IT revolution is evident all around us, but the emphasis has mostly been on the T, the technology. It is time to recast our gaze to focus on the I, the information.

I was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify.

Midas’s error was to mistake gold, wealth’s monetary measure, for wealth itself. But wealth is not a thing or a random sequence. It is inextricably rooted in hard won knowledge over extended time.

At the federal level, this problem could be greatly alleviated by abolishing the Electoral College system. It's the winner-take-all mathematics from state to state that delivers so much power to a relative handful of voters. It's as if in politics, as in economics, we have a privileged 1 percent. And the money from the financial 1 percent underwrites the microtargeting to secure the votes of the political 1 percent. Without the Electoral College, by contrast, every vote would be worth exactly the same. That would be a step toward democracy.

Silver noticed that the areas where Trump performed best made for an odd map. Trump performed well in parts of the Northeast and industrial Midwest, as well as the South. He performed notably worse out West. Silver looked for variables to try to explain this map. Was it unemployment? Was it religion? Was it gun ownership? Was it rates of immigration? Was it opposition to Obama? Silver found that the single factor that best correlated with Donald Trump’s support in the Republican primaries was that measure I had discovered four years earlier. Areas that supported Trump in the largest numbers were those that made the most Google searches for “nigger.

Arno Penzias, the Nobel Prize–winning scientist who codiscovered the cosmic microwave background radiation that provided strong support for the Big Bang in the first place, states, “The best data we have are exactly what I would have predicted, had I nothing to go on but the five Books of Moses, the Psalms, the Bible as a whole.

Facebook is digital brag-to-my-friends-about-how-good-my-life-is serum. In Facebook world, the average adult seems to be happily married, vacationing in the Caribbean, and perusing the Atlantic. In the real world, a lot of people are angry, on supermarket checkout lines, peeking at the National Enquirer, ignoring the phone calls from their spouse, whom they haven’t slept with in years.

Data is the new fuel for growth in multiple industries, from manufacturing to retail to financial services. But unlike other assets, it doesn’t necessarily fuel job growth, but rather, profit growth. And those profits tend to be diverted directly into executives’ and shareholders’ wallets.

They all looked at Holly. She turned to face the cheerleader and said, “You need to learn that some things are more valuable than good looks. Data manipulation is more important than big boobs. Analytics is more useful than lip gloss.” Wow, she said that? Everyone laughed a bit, surprised, shocked. Holly turned and headed toward the concert hall. Grinning.

It is really hurting; how big media plagiarize everyday and no one judges them; The real heroes are those tiny and small self-funded websites and blogs that provide all primary data for them to survive and it will continue as far they exist

Many people underreport embarrassing behaviors and thoughts on surveys. They want to look good, even though most surveys are anonymous. This is called social desirability bias.

If it moves tax it. If it keeps moving regulate it. If it stops moving subsidize it.

Although the method is simple, it shows how, mathematically, random brute force can overcome precise logic. It's a numerical approach that uses quantity to derive quality.

Torture the data, and it will confess to anything,” as

One major irony here is that law, which always lags behind technological innovation by at least a generation, gives substantially more protections to a communication’s content than to its metadata—and yet intelligence agencies are far more interested in the metadata—the activity records that allow them both the “big picture” ability to analyze data at scale, and the “little picture” ability to make perfect maps, chronologies, and associative synopses of an individual person’s life, from which they presume to extrapolate predictions of behavior.

I've got data incoming. Do you want me to transfer it to my portable unit?" "No, you stay here, finish the runs. I shouldn't be more than a couple of hours. When you're done with this, I want you to go find a hammer." Peabody had taken out her memo book, nearly plugged in the order, when she stopped, frowned up at Eve. "Sir? A hammer?" "That's right. A really big, heavy hammer. Then you take it into my office and beat that fucking useless excuse for a data spitter on my desk to dust." "Ah." Because she was a wise woman, Peabody cleared her throat rather than loosen the chuckle. "As an alternate to that action, Lieutenant, I could call maintenance.

a social epidemic, Mavens are data banks. They provide the message. Connectors are social glue: they spread it. But there is also a select group of people—Salesmen—with the skills to persuade us when we are unconvinced of what we are hearing, and they are as critical to the tipping of word-of-mouth epidemics as the other two groups. Who

What we—both as individuals and as a society—should learn from Mom and Locke is that we must be extremely careful about allowing online information acquisition—Google-knowing—to swamp other ways of knowing.

Designing passive tracking apps as if women have pockets big enough to hold their phones is a perennial problem with an easy solution: include proper pockets in women’s clothing (she types, furiously, having just had her phone fall out of her pocket and smash on the floor for the hundredth time).

A lot can be learned from the big companies of today. Companies like Amazon have revolutionized logistics, companies like Tesla have revolutionized sustainable systems, companies like Microsoft and Google have revolutionized data mining and data distribution, companies like Maersk have revolutionized Supply Chains, companies like Gardein and Beyond Meat have revolutionized food. Every company can serve as a case study of some kind with various lessons that can be learned.

I’ve laid down ten statistical commandments in this book. First, we should learn to stop and notice our emotional reaction to a claim, rather than accepting or rejecting it because of how it makes us feel. Second, we should look for ways to combine the “bird’s eye” statistical perspective with the “worm’s eye” view from personal experience. Third, we should look at the labels on the data we’re being given, and ask if we understand what’s really being described. Fourth, we should look for comparisons and context, putting any claim into perspective. Fifth, we should look behind the statistics at where they came from—and what other data might have vanished into obscurity. Sixth, we should ask who is missing from the data we’re being shown, and whether our conclusions might differ if they were included. Seventh, we should ask tough questions about algorithms and the big datasets that drive them, recognizing that without intelligent openness they cannot be trusted. Eighth, we should pay more attention to the bedrock of official statistics—and the sometimes heroic statisticians who protect it. Ninth, we should look under the surface of any beautiful graph or chart. And tenth, we should keep an open mind, asking how we might be mistaken, and whether the facts have changed.

In this, humans are similar to other domesticated animals. We have bred docile cows that produce enormous amounts of milk but are otherwise far inferior to their wild ancestors. They are less agile, less curious, and less resourceful.34 We are now creating tame humans that produce enormous amounts of data and function as very efficient chips in a huge data-processing mechanism, but these data-cows hardly maximize the human potential. Indeed, we have no idea what our full human potential is, because we know so little about the human mind. And yet we don’t invest much in exploring the human mind, instead focusing on increasing the speed of our internet connections and the efficiency of our Big Data algorithms. If we are not careful, we will end up with downgraded humans misusing upgraded computers to wreak havoc on themselves and on the world.

To create a model, then, we make choices about what’s important enough to include, simplifying the world into a toy version that can be easily understood and from which we can infer important facts and actions. We expect it to handle only one job and accept that it will occasionally act like a clueless machine, one with enormous blind spots.

The very idea of penalizing based on propensities is nauseating. To accuse a person of some possible future behavior is to negate the very foundation of justice: that one must have done something before we can hold him accountable for it. After all, thinking bad things is not illegal, doing them is. It is a fundamental tenet of our society that individual responsibility is tied to individual choice of action. [...] Were perfect predictions possible, they would deny human volition, our ability to live our lives freely. Also, ironically, by depriving us of choice they would exculpate us from any responsibility.

Just imagine if police enforced their zero-tolerance strategy in finance. They would arrest people for even the slightest infraction, whether it was chiseling investors on 401ks, providing misleading guidance, or committing petty frauds. Perhaps SWAT teams would descend on Greenwich, Connecticut. They’d go undercover in the taverns around Chicago’s Mercantile Exchange.

If you don't want a man unhappy politically, don't give him two sides to a question to worry him; give him one. Better yet, give him none. Let him forget there is such a thing as war. If the government is inefficient, top-heavy, and tax-mad, better it be all those than that people worry over it. Peace, Montag. Give the people contests they win by remembering the words to more popular songs or the names of state capitals or how much corn Iowa grew last year. Cram them full of noncombustible data, chock them so damned full of 'facts' they feel stuffed, but absolutely 'brilliant' with information. Then they'll feel they're thinking, they'll get a sense of motion without moving. And they'll be happy, because facts of that sort don't change. Don't give them any slippery stuff like philosophy or sociology to tie things up with. That way lies melancholy. Any man who can take a TV wall apart and put it back together again, and most men can nowadays, is happier than any man who tries to slide-rule, measure and equate the universe, which just wont be measured or equated without making man feel bestial and lonely.

Racism, at the individual level, can be seen as a predictive model whirring away in billions of human minds around the world. It is built from faulty, incomplete, or generalized data. Whether it comes from experience or hearsay, the data indicates that certain types of people have behaved badly. That generates a binary prediction that all people of that race will behave that same way. Needless to say, racists don’t spend a lot of time hunting down reliable data to train their twisted models. And once their model morphs into a belief, it becomes hardwired. It generates poisonous assumptions, yet rarely tests them, settling instead for data that seems to confirm and fortify them. Consequently, racism is the most slovenly of predictive models. It is powered by haphazard data gathering and spurious correlations, reinforced by institutional inequities, and polluted by confirmation bias.

Presidential campaigns are on the verge of turning into media contests between master operators of the Internet. What once had been substantive debates about the content of governance will reduce candidates to being spokesmen for a marketing effort pursued by methods whose intrusiveness would have been considered only a generation ago the stuff of science fiction. The candidates’ main role may become fund-raising rather than the elaboration of issues. Is the marketing effort designed to convey the candidate’s convictions, or are the convictions expressed by the candidate the reflections of a “big data” research effort into individuals’ likely preferences and prejudices? Can democracy avoid an evolution toward a demagogic outcome based on emotional mass appeal rather than the reasoned process the Founding Fathers imagined?

People frequently lie—to themselves and to others. In 2008, Americans told surveys that they no longer cared about race. Eight years later, they elected as president Donald J. Trump, a man who retweeted a false claim that black people are responsible for the majority of murders of white Americans, defended his supporters for roughing up a Black Lives Matters protester at one of his rallies, and hesitated in repudiating support from a former leader of the Ku Klux Klan. The same hidden racism that hurt Barack Obama helped Donald Trump.

One time I listened to Farmer give a talk on HIV to a class at the Harvard School of Public Health, and in the midst of reciting data, he mentioned the Haitian phrase “looking for life, destroying life,” Then he explained, “It’s an expression Haitians use if a poor woman selling mangoes falls off a truck and dies.” I felt as if for that moment I could see a little way into his mind, It seemed like a place of hyperconnectivity, At moments like that, I thought that what he wanted was to erase both time and geography, connecting all parts of his life and tying them instrumentally to a world in which he saw intimate, inescapable connections between the gleaming corporate offices of Paris and New York and a legless man lying on the mud floor of a hut in the remotest part of remote Haiti. Of all the world’s errors, he seemed to feel, the most fundamental was the “erasing” of people, the “hiding away” of suffering. “My big struggle is how people can not care, erase, not remember.

[I]t seems to me that a lot of the stranger ideas people have about medicine derive from an emotional struggle with the very notion of a pharmaceutical industry. Whatever our political leanings, we all feel nervous about profit taking any role in the caring professions, but that feeling has nowhere to go. Big pharma is evil; I would agree with that premise. But because people don’t understand exactly how big pharma is evil, their anger gets diverted away from valid criticisms—its role in distorting data, for example, or withholding lifesaving AIDS drugs from the developing world—and channeled into infantile fantasies. “Big pharma is evil,” goes the line of reasoning; “therefore homeopathy works and the MMR vaccine causes autism.” This is probably not helpful.

The math-powered applications powering the data economy were based on choices made by fallible human beings. Some of these choices were no doubt made with the best intentions. Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong or harmful, were beyond dispute or appeal. And they tended to punish the poor and the oppressed in our society, while making the rich richer.

Sex differences in the brain are irresistible to those looking to explain stereotypic differences between men and women,” she told reporters when her paper came out. “They often make a big splash, in spite of being based on small samples. But as we explore multiple data sets and are able to coalesce very large samples of males and females, we find these differences often disappear or are trivial.

The Internet’s abundant capacity has removed the old artificial constraints on publishing—including getting our content checked and verified. The new strategy of publishing everything we find out thus results in an immense cloud of data, free of theory, published before verified, and available to anyone with an Internet connection. And this is changing the role that facts have played as the foundation of knowledge.

Sometimes new data reveals cultural differences I had never even contemplated. One example: the very different ways that men around the world respond to their wives being pregnant. In Mexico, the top searches about “my pregnant wife” include “frases de amor para mi esposa embarazada” (words of love to my pregnant wife) and “poemas para mi esposa embarazada” (poems for my pregnant wife). In the United States, the top searches include “my wife is pregnant now what” and “my wife is pregnant what do I do.

Frankly, the overwhelming majority of academics have ignored the data explosion caused by the digital age. The world’s most famous sex researchers stick with the tried and true. They ask a few hundred subjects about their desires; they don’t ask sites like PornHub for their data. The world’s most famous linguists analyze individual texts; they largely ignore the patterns revealed in billions of books. The methodologies taught to graduate students in psychology, political science, and sociology have been, for the most part, untouched by the digital revolution. The broad, mostly unexplored terrain opened by the data explosion has been left to a small number of forward-thinking professors, rebellious grad students, and hobbyists. That will change.

The success of college towns and big cities is striking when you just look at the data. But I also delved more deeply to undertake a more sophisticated empirical analysis. Doing so showed that there was another variable that was a strong predictor of a person’s securing an entry in Wikipedia: the proportion of immigrants in your county of birth. The greater the percentage of foreign-born residents in an area, the higher the proportion of children born there who go on to notable success. (Take that, Donald Trump!) If two places have similar urban and college populations, the one with more immigrants will produce more prominent Americans. What

So, while we cannot trust the stories we are told, tradition, faith, convenient or reassuring narratives, charismatic figures, or even our own memories, we can slowly and carefully build a process by which to evaluate all claims to truth and knowledge. A big part of that process is science, which systematically tests our ideas against reality, using the most objective data possible. Science is still a messy and flawed process, but it is a process. It has, at least, the capacity for self-correction, to move our beliefs incrementally in the direction of reality. In essence, science is the process of making our best effort to know what’s really real.

For example, I do a little bit of data analysis, like I was saying. For tech companies mostly. They’ll give me a lot of data – say user experience data, like how long users spend on each section of a website – and I’ll spend a few hours making graphs and whatnot. Say it takes me – I don’t know, four hours to make these graphs, and I’ll pretend it took me ten hours, to get extra money. He glances over at her again, and adds: You might think that’s immoral, I don’t know. But anyway, never mind that for a second. The four hours that I actually spend making the graphs, and the ten hours that I get paid for: what is that? Like, any of that: what is it? At least when I worked as a delivery driver, I knew what I was doing. Someone wanted a Big Mac, and I brought it to them, and the amount I got paid was like, what it was worth to that person not to have to collect their own burger. The amount they will pay, not to leave the house, is the amount I will accept, yes to leave the house. Minus whatever the app is taking. If you get me. I get you. You’re making perfect sense. Oh good, he says. Because in the data analysis example, my question is, what is the money that’s being paid to me? It’s the money that the company will pay, to have their own information explained back to them in a graph. And how much money should that be? Clearly no one knows, because at the end I’ll make up a number of hours and they’ll just pay me for that number. I guess the graph is supposed to make the company more profitable, in theory, but no one knows by how much, it’s all made up.

According to the prevailing notion, to be free means to be free to satisfy one’s preferences. Preferences themselves are beyond rational scrutiny; they express the authentic core of a self whose freedom is realized when there are no encumbrances to its preference-satisfying behavior. Reason is in the service of this freedom, in a purely instrumental way; it is a person’s capacity to calculate the best means to satisfy his ends. About the ends themselves we are to maintain a principled silence, out of respect for the autonomy of the individual. To do otherwise would be to risk lapsing into paternalism. Thus does liberal agnosticism about the human good line up with the market ideal of “choice.” We invoke the latter as a content-free meta-good that bathes every actual choice made in the softly egalitarian, flattering light of autonomy. This mutually reinforcing set of posits about freedom and rationality provides the basic framework for the discipline of economics, and for “liberal theory” in departments of political science. It is all wonderfully consistent, even beautiful. But in surveying contemporary life, it is hard not to notice that this catechism doesn’t describe our situation very well. Especially the bit about our preferences expressing a welling-up of the authentic self. Those preferences have become the object of social engineering, conducted not by government bureaucrats but by mind-bogglingly wealthy corporations armed with big data. To continue to insist that preferences express the sovereign self and are for that reason sacred—unavailable for rational scrutiny—is to put one’s head in the sand. The resolutely individualistic understanding of freedom and rationality we have inherited from the liberal tradition disarms the critical faculties we need most in order to grapple with the large-scale societal pressures we now face.

It is well-known that a big percentage of all marriages in the United States end in divorce or separation (about 39 percent, according to the latest data).[30] But staying together is not what really counts. Analysis of the Harvard Study data shows that marriage per se accounts for only 2 percent of subjective well-being later in life.[31] The important thing for health and well-being is relationship satisfaction. Popular culture would have you believe the secret to this satisfaction is romantic passion, but that is wrong. On the contrary, a lot of unhappiness can attend the early stages of romance. For example, researchers find that it is often accompanied by rumination, jealousy, and “surveillance behaviors”—not what we typically associate with happiness. Furthermore, “destiny beliefs” about soul mates or love being meant to be can predict low forgiveness when paired with attachment anxiety.[32] Romance often hijacks our brains in a way that can cause the highs of elation or the depths of despair.[33] You might accurately say that falling in love is the start-up cost for happiness—an exhilarating but stressful stage we have to endure to get to the relationships that actually fulfill us. The secret to happiness isn’t falling in love; it’s staying in love, which depends on what psychologists call “companionate love”—love based less on passionate highs and lows and more on stable affection, mutual understanding, and commitment.[34] You might think “companionate love” sounds a little, well, disappointing. I certainly did the first time I heard it, on the heels of great efforts to win my future wife’s love. But over the past thirty years, it turns out that we don’t just love each other; we like each other, too. Once and always my romantic love, she is also my best friend.

In fact, as these companies offered more and more (simply because they could), they found that demand actually followed supply. The act of vastly increasing choice seemed to unlock demand for that choice. Whether it was latent demand for niche goods that was already there or a creation of new demand, we don't yet know. But what we do know is that the companies for which we have the most complete data - netflix, Amazon, Rhapsody - sales of products not offered by their bricks-and-mortar competitors amounted to between a quarter and nearly half of total revenues - and that percentage is rising each year. in other words, the fastest-growing part of their businesses is sales of products that aren't available in traditional, physical retail stores at all. These infinite-shelf-space businesses have effectively learned a lesson in new math: A very, very big number (the products in the Tail) multiplied by a relatives small number (the sales of each) is still equal to a very, very big number. And, again, that very, very big number is only getting bigger. What's more, these millions of fringe sales are an efficient, cost-effective business. With no shelf space to pay for - and in the case of purely digital services like iTunes, no manufacturing costs and hardly any distribution fees - a niche product sold is just another sale, with the same (or better) margins as a hit. For the first time in history, hits and niches are on equal economic footing, both just entries in a database called up on demand, both equally worthy of being carried. Suddenly, popularity no longer has a monopoly on profitability.