Is Wolfram Quotes

What will limit us is not the possible evolution of technology, but the evolution of human purposes.

We never let our people just go. (Joe) What are you? Wolfram and Hart? (Steele) Oh, no, sweetie, they just take your soul for service. We intend to take even more than that. (Tee)

It's always seemed like a big mystery how nature, seemingly so effortlessly, manages to produce so much that seems to us so complex. Well, I think we found its secret. It's just sampling what's out there in the computational universe.

While not all elements in the Periodic Table are represented by letters of the alphabet, some in this book (Magical Elements of the Periodic Table Presented Alphabetically by the Metal Horn Unicorns), are introduced by alternate designations. For instance, Tungsten is also known as Wolfram so “W” is used as the entry for that alphabetical letter in this book. The letter “W” is also used as the atomic symbol for Tungsten in all periodic tables.

While researching this answer, I managed to lock up my copy of Mathematica several times on balloon-related differential equations, and subsequently got my IP address banned from Wolfram|Alpha for making too many requests. The ban-appeal form asked me to explain what task I was performing that necessitated so many queries. I wrote, “Calculating how many rental helium tanks you’d have to carry with you in order to inflate a balloon large enough to act as a parachute and slow your fall from a jet aircraft.” Sorry, Wolfram.

…you know, sometimes an electric lightbulb goes out all of a sudden. Fizzles, you say. And this burned-out bulb, if you shake it, it flashes again and it’ll burn a little longer. Inside the bulb it’s a disaster. The wolfram filaments are breaking up, and when the fragments touch, life returns to the bulb. A brief, unnatural, undeniably doomed life—a fever, a too-bright incandescence, a flash. The comes the darkness, life never returns, and in the darkness the dead, incinerated filaments are just going to rattle around. Are you following me? But the brief flash is magnificent! “I want to shake… “I want to shake the heart of a fizzled era. The lightbulb of the heart, so that the broken pieces touch… “…and produce a beautiful, momentary flash…

And Wolfram knows about cellular automata?” “Oh, my goodness, yes,” said Anna. “He wrote a book you could kill a man with—twelve hundred pages—called A New Kind of Science. It’s all about them.” “We should totally ask him what he thinks!” Caitlin said.

Funny thing about straddling fences, though: eventually you end up with a pain in the butt and not much ground covered in any direction.

Then the King of Arragon pushed old Utepandragun over his horse’s tail down on to the meadow – the King of Britain! – where he lay in a bed of flowers!

Von Neumann was in many ways a traditional mathematician, who (like Turing) believed he needed to turn to partial differential equations in describing natural systems.

Let mercy keep company with courage. Follow my advice in this: if in battle you win a man’s surrender, then unless he has done you such grievance as amounts to heart’s sorrow, accept his oath, and let him live.

It is perhaps a little humbling to discover that we as humans are in effect computationally no more capable than cellular automata with very simple rules. But the Principle of Computational Equivalence also implies that the same is ultimately true of our whole universe. So while science has often made it seem that we as humans are somehow insignificant compared to the universe, the Principle of Computational Equivalence now shows that in a certain sense we are at the same level as it is. For the principle implies that what goes on inside us can ultimately achieve just the same level of computational sophistication as our whole universe.

If doubt is near neighbor to the heart, that may turn sour on the soul. There is both scorning and adorning when a man’s undaunted mind turns pied like the magpie’s hue. Yet he may still enjoy bliss, for both have a share in him, Heaven and Hell. Inconstancy’s companion holds entirely to the black colour and will, indeed, take on darkness’s hue, while he who is constant in his thoughts will hold to the white.

Like large areas of analytic philosophy today, scholasticism, too, preferred to busy itself with the fetishization of fine distinctions on an apparently secure investigative foundation, rather than engaging in the adventure of providing a relevant contribution to the understanding of its own age, with its shifting foundational structures.

If you’re having trouble with a math problem, plug the equation into WolframAlpha.com and it will solve it for you.

His ways were a refuge from falsity.

I'm committed to seeing this project done. To see if within this decade we can finally hold in our hands the rule for our universe, and know where our universe lies in the space of all possible universes.

There are whole villages in Extremadura in Spain that are built of rock that has very high grade wolfram ore and the stone fences of the peasant’s field are all made of this ore. Yet the peasants are very poor. At this time it was so valuable that we were using DC-2’s, transport planes such as fly from here to Miami, to fly it over from a field at Nam Yung in Free China to Kai Tak airport at Kowloon. From there it was shipped to the States. It was considered very scarce and of vital importance in our preparations for war

So when I hear this snarky question (and I hear it everywhere): Are librarians obsolete in the Age of Google? all I can say is, are you kidding? Librarians are more important than ever. Google and Yahoo! and Bing and WolframAlpha can help you find answers to your questions, sometimes brilliantly; but if you don't know how to phrase those questions, no search engine can help provide the answers.

Sir, if you are otherwise discreet, you will consider that you have gone far enough. At my brother's request I am treating you no less kindly than Ampflise treated my uncle Gahmuret, without going to bed together. My kindness would in the long run outweigh hers, if anyone were to weigh us properly. And besides, Sir, I don't know who you are, and yet in such a short space of time you want to have my love.

Mainittakoon lyhyesti, että kiroilussa suomalainen on tasapainon vaalija, siis keskitien kulkija. Sillä siinä missä romaanisten kielten puhuja keskittyy toisia tai itseään häväistääkseen genitaalialueeseen ja germaani pysyttelee tiukasti anaalilinjalla, suomlainen hallitsee sekä uloste- että genitaalirekisterin ja rikastaa sadatteluaan vielä perkeleen tai itsensä saatanan kaltaisella demonikuvastolla. Tämän herkän kielen rumin ilmaus on naisen sukuelintä tarkoittava diabolinen manaus.

There was much more she would have liked to tell her brother. But within a few months, she would be able to tell him in person. When he learned of the attack on the airship, nothing would stop Archimedes and his wife from coming. But at least they would fly to the Red City instead of Krakentown, where he might be recognized as the smuggler Wolfram Gunther-Baptiste. One day, she might write a story inspired by that part of his career. She would call it The Idiot Smuggler Who Destroyed the Horde Rebellion’s War Machines and Changed His Name to Avoid the Rebel Assassins. Zenobia would take pity on the idiot’s sister and leave her out of the tale. She

Wolfram, one of the most innovative thinkers in scientific computing and in the theory of complex systems, has been best known for the development of Mathematica, a computer program/system that allows a range of calculations not accessible before. After ten years of virtual silence, Wolfram is about to emerge with a provocative book that makes the bold claim that he can replace the basic infrastructure of science. In a world used to more than three hundred years of science being dominated by mathematical equations as the basic building blocks of models for nature, Wolfram proposes simple computer programs instead. He suggests that nature's main secret is the use of simple programs to generate complexity.

Die Frage, wie ein Wesen, das nur ein paar Weltzeitsekunden existiert, dazu bewegt werden soll, sein Verhalten nach Zeiträumen auszurichten, die es sich gar nicht vorstellen kann, bleibt ohne Antwort. Was könnte dieses Wesen dazu bringen, weniger rücksichtslos und stattdessen achtsamer zu sein, weniger zu konsumieren und wegzuschmeißen, weniger Nachkommen zu zeugen und die Erde nicht als Wegwerfartikel, sondern als Leihgabe, als Teil des eigenen Körpers zu begreifen? So wie es steht, wird dieses Wesen in seiner weltzeitlichen Sekundenexistenz aus eigenem Antrieb oder eigener Einsicht wohl niemals aufhören, die wundersame Kugel, auf der und von der er lebt, ungebremst und erbarmungslos auszuweiden und zuzumüllen.

hardening steel, yet anyone could go out and dig up as much of it in the hills of the New Territories as he or she could carry on a flat basket balanced on the head to the big shed where it was bought clandestinely. I found this out when I was hunting wood pigeons and I brought it to the attention of people purchasing wolfram in the interior. No one was very interested and I kept bringing it to the attention of people of higher rank until one day a very high officer who was not at all interested that wolfram was there free to be dug up in the New Territories said to me, ‘But after all, old boy, the Nam Yung set-up is functioning you know.’ But when we shot in the evenings outside the women’s prison and would see an old Douglas twin-motor plane come in over the hills and slide down toward the airfield, and you knew it was loaded with sacked wolfram and had just flown over the Jap lines, it was strange to know that many of the women in the women’s prison were there for having been caught digging wolfram illicitly.

Ulkomaalaispöydässä istunut Mike oli juhlien kestäessä väittänyt taas kerran vakavissaan, että suomalaiset ovat tosiasiassa ulkoavaruudesta kotoisin tai että se ainakin on uskottavin selitys tämän kansan olemassaololle. Ja maassa joka puolella näkyvien vesitornien hän selitti olevan lähtöramppeja, joista suomalaiset jonakin päivänä starttaavat palatessaan taas kaukaiselle kotiplaneetalleen.

Someone pumps sentences into my brain, long-forgotten images from childhood; meaningless objects and conversations peel layers from my heart. I am again a river faun, paralyzed by longing for a river nymph. I walk through wolframic space, my mouth and nose threaded with wire, and whenever I deviate from my course, I feel a sharp pain in my jaws.

Neural nets—perhaps a bit like brains—are set up to have an essentially fixed network of neurons, with what’s modified being the strength (“weight”) of connections between them.

How about something like ChatGPT? Well, it has the nice feature that it can do “unsupervised learning”, making it much easier to get it examples to train from. Recall that the basic task for ChatGPT is to figure out how to continue a piece of text that it’s been given. So to get it “training examples” all one has to do is get a piece of text, and mask out the end of it, and then use this as the “input to train from”—with the “output” being the complete, unmasked piece of text.

But there’s something potentially confusing about all of this. In the past there were plenty of tasks—including writing essays—that we’ve assumed were somehow “fundamentally too hard” for computers. And now that we see them done by the likes of ChatGPT we tend to suddenly think that computers must have become vastly more powerful—in particular surpassing things they were already basically able to do (like progressively computing the behavior of computational systems like cellular automata). But this isn’t the right conclusion to draw. Computationally irreducible processes are still computationally irreducible, and are still fundamentally hard for computers—even if computers can readily compute their individual steps. And instead what we should conclude is that tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought. In other words, the reason a neural net can be successful in writing an essay is because writing an essay turns out to be a “computationally shallower” problem than we thought. And in a sense this takes us closer to “having a theory” of how we humans manage to do things like writing essays, or in general deal with language.

much like for humans, if you tell it something bizarre and unexpected that completely doesn’t fit into the framework it knows, it doesn’t seem like it’ll successfully be able to “integrate” this. It can “integrate” it only if it’s basically riding in a fairly simple way on top of the framework it already has.

Wolfram,

In a TED talk watched by over a million people, Wolfram (2010) proposes that working on mathematics has four stages: Posing a question Going from the real world to a mathematical model Performing a calculation Going from the model back to the real world, to see if the original question was answered The first stage involves asking a good question of some data or a situation—the first mathematical act that is needed in the workplace.

Wolfram,” Dan mused, a far-off look in his eye. “I’ve heard of that from somewhere.” Amy was skeptical. “Are you sure you’re not thinking of Wolfgang?

For example, a telegram is a "lightning-letter"; a wireless telegram is a "not-have-wire-lightning-communication"; a fountain-pen is a "self-flow-ink-water-brush"; a typewriter is a "strike-letter-machine". Most of these neologisms are similar in the modern languages of China and Japan.

between 1195 and 1220, Wolfram composed his epic romance Parzival, he conferred on the Templars a most exalted status.

He broke fresh ground—because, and only because, he had the courage to go ahead without asking whether others were following or even understood. He had no need for the divided responsibility in which others seek to be safe from ridicule, because he had been granted a faith which required no confirmation—a contact with reality, light and intense like the touch of a loved hand: a union in self-surrender without self-destruction, where his heart was lucid and his mind was loving.16 The crux of the story of Parzival and his quest for the Grail is suggested in his first encounter in the Grail castle. After various adventures, Parzival has sort of stumbled into the Grail castle. This is the wisdom of innocence. The purity of the simple fellow gets him into the Grail castle. In the castle lives a king who is sorely wounded. The king’s illness has brought devastation to the kingdom—it has become the Wasteland. The theme of the Grail is the bringing of life into what is known as ‘the wasteland.’ The wasteland is the preliminary theme to which the Grail is the answer…It’s the world of people living inauthentic lives—doing what they are supposed to do. Joseph Campbell Parzival can redeem the king and kingdom by asking a simple question. The wounded king is brought before him, and Parzival wants to ask, “What ails thee, brother?” But he has been told good knights don’t ask a lot of questions. The decisive moment for him is the choice between acting spontaneously from his heart or conventionally from his role as a knight. He fails; innocence is not enough, for he has already been socially indoctrinated. It has caused him to doubt the promptings of his heart, and as Wolfram says in the very first line of his Parzival, “If vacillation dwell with the heart the soul will rue it.”17 Life’s most urgent question is, what are you doing for others? Martin Luther King, Jr.

In the context of physics, 137 is equal to the integer part of the inverse of the fine structure constant ... The fine structure constant α is the key to the physicist’s quest for a Grand Unified Theory ... The number 137 has intrigued numerous prominent theoretical physicists ... All told, we believe that it is much easier, and more motivating, to remember a number that has deep significance in numerous disciplines, ... with the following terse ode to 137: Bethe was mischievous with 137 Bohr was intrigued by 137 Born was mystified by 137 Fermi was frisky with 137 Feynman was mesmerized by 137 Heisenberg was fascinated by 137 Lederman was enchanted by 137 Pauli was consumed by 137 Turing was matched by 137

rural India live like refugees is not that they don’t work as hard as we do, or are not as smart as we are, but that they live in an economic system that doesn’t allow them to be productive. The basis of our economic prosperity is market capitalism, individual liberty and responsibility, and limited government.

is rather a system of voluntary exchange based on private property rights, limited government, and individual freedom.

Voluntary exchange ensures that only businesses that provide what consumers want at the right price will survive, fostering continuous innovation.

We do not take the time to consider that millions of people will awake in our largest cities tomorrow and there will be the right amount of coffee, dental floss, toilet paper, and an astonishing array of other goods and services sold during the day. Yet if we do stop to think about it, it is a miracle.

Imagine the dawn of man, clans of hunter-gatherers following the

fall of 2011 with the Occupy Wall Street protest. Interviews with these anticapitalism protesters reminded me of a scene in the 1979 Monty Python film Life of Brian

One becomes wealthy in a market system by pleasing others, and the more individuals you please the wealthier you become.

Societies that do not have market economies have been forced to concede that only free markets are capable of producing on a scale that affords even the poorest person a standard of living well above what would have been unthinkable just a few hundred years ago.

Frédéric Bastiat noted, it is not possible to develop a science of politics without understanding how the economic system works. Nobel laureate Friedrich Hayek refined this idea by offering that if people do not understand and believe in market capitalism, they will ask their government to undertake actions that in the end will make us less wealthy and free. This

In other words, the good economist can imagine the unintended consequences of a policy action. The goal of this book is to make you a good economist.

Yet Wyatt Earp, who is an adult when he participates in the gunfight at the OK Corral, sees the movement from four-wheeled carts to the Model T.

Each day we go about our business in complete confidence that the rest of society will provide for our basic needs. Typically, we do not stop to wonder how food gets to our table, clothes into our closet, or how our

But it may be that some students would be willing to loan their money only if compensated by receiving interest,

Nonetheless, some may be bothered by the fact that the wealthier students have a better chance of remaining on the bus.

In a market economy people act to improve their well-being, not necessarily their wealth or number of possessions.

Even at this primitive stage, other members of the clan would have specialized in the making of tools or the tanning of hides, activities

characteristics distinguishing human behavior from that of all other animals; we alone have developed the ability to satisfy our needs and wants through the peaceful exchange of value for

Capitalism is not a collusion between big business and big government to advance the interests of stockholders and management at the expense of workers

One solution is an impromptu capital market where people can borrow from each other. Some students will give up some of their current purchasing power in order to receive the money at a later date. Being friends, and supposing that they will all be back at their dorms later to settle their accounts, the students might simply loan the money to one another at no cost.

This illustrates that in the market process, exchange will occur whenever two people value a good differently and when both will benefit from the exchange.

Those earning a greater share of the bounty would exchange some of their perishable surplus for a straighter spear or warmer clothing

West gradually developed the economic system of market capitalism and a compatible political system.

A solution that came immediately to my colleague and me was to have everyone exit the bus and then buy their way back on.

They appear to think that the cell phones they use, food they eat, hotels and tents they stay in, their sleeping bags and clothes, the cars they drive and the fuel that powers them and all the goods and services they consume every day would exist under a different system, perhaps in more abundance.

and students can express their intensity of preference by the amount they are willing to pay to get back on the bus.

Let’s suppose for now that the money is divided among the ten students who did not ride the bus. This would compensate them for their misfortune and might seem like the right thing to do. Another obvious question: What does one do about those students who forgot to bring their money?

This is because we will always pay less for a mere chance than for the object itself.3 In either case, free market exchange allows everyone the opportunity to improve his or her position.

But if you inflated the balloons quickly, possibly by connecting many canisters to it at once, you’d be able to slow your fall. Just don’t use too much helium, or you’ll end up floating at 16,000 feet like Larry Walters. While researching this answer, I managed to lock up my copy of Mathematica several times on balloon-related differential equations, and subsequently got my IP address banned from Wolfram|Alpha for making too many requests. The ban-appeal form asked me to explain what task I was performing that necessitated so many queries. I wrote, “Calculating how many rental helium tanks you’d have to carry with you in order to inflate a balloon large enough to act as a parachute and slow your fall from a jet aircraft.” Sorry, Wolfram. 1 While researching impact speeds for this answer, I came across a discussion on the Straight Dope Message Board about survivable fall heights. One poster

Now a member of the National Socialist Party, he addressed the German student body in a newspaper article accompanying his appointment: “Let not theoretical principles and ‘ideas’ be the rules of your Being. The Führer himself and he alone is the German reality and its law today and in the future.”1

In January 1930 he was to begin teaching a course at Cambridge. Shortly before he set off for the holidays he was asked by one of his colleagues there what title his course should be given on the lecture list.Wittgenstein thought for a while. Finally he replied: “‘Philosophy.’ What else?”2

exploitation. Benjamin was a few years older than Scholem, which gave him a certain advantage in terms of maturity and knowledge from the outset. This, too, is typical, for Benjamin preferred to maintain in his friendships a mutually acknowledged hierarchy of knowledge.

Being a philosopher is a way of leading one’s own life consciously, giving it pull, form, and direction through constant, probing questioning.

When indecision’s in the heart The soul is bound to grieve and smart.

The power held by corporate giants was terrifying even before the CEO decided to leverage that power for their own murderous ends. A supply shortage. A profit-driven business decision. Cost cuts or poorly thought-out policies that reduced safety margins, forced people into unemployment, or added more pressure to frontline workers already stretched thin. A price hike of an essential medicine. (Wolfram hadn’t forged new ground there.) These things, especially in the health and medical industry, routinely killed far more people than the average serial killer could ever aspire to. And yet so few of them resulted in criminal charges. Indirect manslaughter for profit was far more societally acceptable than one person purposefully ending lives on a smaller scale.

This short book is an attempt to explain from first principles how and why ChatGPT works. In some ways it’s a story about technology. But it’s also a story about science.

The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.” So

So what can we do? The big idea is to make a model that lets us estimate the probabilities with which sequences should occur—even though we’ve never explicitly seen those sequences in the corpus of text we’ve looked at. And at the core of ChatGPT is precisely a so-called “large language model” (LLM) that’s been built to do a good job of estimating those probabilities.

also do this for sequences of words, or indeed whole blocks of text.

Not surprisingly, this is nonsense.

(As a personal comparison, my total lifetime output of published material has been a bit under 3 million words, and over the past 30 years I’ve written about 15 million words of email, and altogether typed perhaps 50 million words—and in just the past couple of years I’ve spoken more than 10 million words on livestreams. And, yes, I’ll train a bot from all of that.)

Whatever input it’s given, the neural net is generating an answer. And, it turns out, to do it in a way that’s reasonably consistent with what humans might do. As I’ve said above, that’s not a fact we can “derive from first principles”. It’s just something that’s empirically been found to be true,

Here’s what happens if one repeatedly “applies the model”—at each step adding the word that has the top probability (specified in this code as the “decision” from the model): What happens if one goes on longer? In this (“zero temperature”) case what comes out soon gets rather confused and repetitive:

In a crawl of the web there might be a few hundred billion words; in books that have been digitized there might be another hundred billion words. But with 40,000 common words, even the number of possible 2-grams is already 1.6 billion—and the number of possible 3-grams is 60 trillion. So there’s no way we can estimate the probabilities even for all of these from text that’s out there. And by the time we get to “essay fragments” of 20 words, the number of possibilities is larger than the number of particles in the universe, so in a sense they could never all be written down. So what can we do? The big idea is to make a model that lets us estimate the probabilities with which sequences should occur—even though we’ve never explicitly seen those sequences in the corpus of text we’ve looked at. And at the core of ChatGPT is precisely a so-called “large language model” (LLM) that’s been built to do a good job of estimating those probabilities.

Or you could do what is the essence of theoretical science: make a model that gives some kind of procedure for computing the answer rather than just measuring and remembering each case.

It is worth understanding that there’s never a “model-less model”. Any model you use has some particular underlying structure—then a certain set of “knobs you can turn” (i.e. parameters you can set) to fit your data. And in the case of ChatGPT, lots of such “knobs” are used—actually, 175 billion of them. But the remarkable thing is that the underlying structure of ChatGPT—with “just” that many parameters—is sufficient to make a model that computes next-word probabilities “well enough” to give us reasonable essay-length pieces of text.

ChatGPT is based on the concept of neural nets—originally invented in the 1940s as an idealization of the operation of brains. I myself first programmed a neural net in 1983—and it didn’t do anything interesting. But 40 years later, with computers that are effectively a million times faster, with billions of pages of text on the web, and after a whole series of engineering innovations, the situation is quite different. And—to everyone’s surprise—a neural net that is a billion times larger than the one I had in 1983 is capable of doing what was thought to be that uniquely human thing of generating meaningful human language.

ChatGPT effectively does something like this, except that (as I’ll explain) it doesn’t look at literal text; it looks for things that in a certain sense “match in meaning”. But the end result is that it produces a ranked list of words that might follow, together with “probabilities”: And the remarkable thing is that when ChatGPT does something like write an essay what it’s essentially doing is just asking over and over again “given the text so far, what should the next word be?”—and each time adding a word. (More precisely, as I’ll explain, it’s adding a “token”, which could be just a part of a word, which is why it can sometimes “make up new words”.)

In human brains there are about 100 billion neurons (nerve cells), each capable of producing an electrical pulse up to perhaps a thousand times a second. The neurons are connected in a complicated net, with each neuron having tree-like branches allowing it to pass electrical signals to perhaps thousands of other neurons. And in a rough approximation, whether any given neuron produces an electrical pulse at a given moment depends on what pulses it’s received from other neurons—with different connections contributing with different “weights”. When we “see an image” what’s happening is that when photons of light from the image fall on (“photoreceptor”) cells at the back of our eyes they produce electrical signals in nerve cells. These nerve cells are connected to other nerve cells, and eventually the signals go through a whole sequence of layers of neurons. And it’s in this process that we “recognize” the image, eventually “forming the thought” that we’re “seeing a 2” (and maybe in the end doing something like saying the word “two” out loud).

There’s nothing particularly “theoretically derived” about this neural net; it’s just something that—back in 1998—was constructed as a piece of engineering, and found to work. (Of course, that’s not much different from how we might describe our brains as having been produced through the process of biological evolution.)

We somehow want all the 1’s to “be attracted to one place”, and all the 2’s to “be attracted to another place”. Or, put a different way, if an image is somehow “closer to being a 1” than to being a 2, we want it to end up in the “1 place” and vice versa.

In the traditional (biologically inspired) setup each neuron effectively has a certain set of “incoming connections” from the neurons on the previous layer, with each connection being assigned a certain “weight” (which can be a positive or negative number). The value of a given neuron is determined by multiplying the values of “previous neurons” by their corresponding weights, then adding these up and adding a constant—and finally applying a “thresholding” (or “activation”) function. In mathematical terms, if a neuron has inputs x = {x1, x2 ...} then we compute f[w . x + b], where the weights w and constant b are generally chosen differently for each neuron in the network; the function f is usually the same. Computing w . x + b is just a matter of matrix multiplication and addition. The “activation function” f introduces nonlinearity (and ultimately is what leads to nontrivial behavior). Various activation functions commonly get used; here we’ll just use Ramp (or ReLU):

(And—as we’ll discuss later—these weights are normally determined by “training” the neural net using machine learning from examples of the outputs we want.)

Ultimately, every neural net just corresponds to some overall mathematical function—though it may be messy to write out. For the example above, it would be: The neural net of ChatGPT also just corresponds to a mathematical function like this—but effectively with billions of terms.

The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.

So what happens if one goes on longer? Here’s a random example. It’s better than the top-word (zero temperature) case, but still at best a bit weird: This was done with the simplest GPT-2 model (from 2019). With the newer and bigger GPT-3 models the results are better. Here’s the top-word (zero temperature) text produced with the same “prompt”, but with the biggest GPT-3 model: And here’s a random example at “temperature 0.8”: Where Do the Probabilities Come From?

And we have a “good model” if the results we get from our function typically agree with what a human would say. And the nontrivial scientific fact is that for an image-recognition task like this we now basically know how to construct functions that do this. Can we “mathematically prove” that they work? Well, no. Because to do that we’d have to have a mathematical theory of what we humans are doing. Take the “2” image and change a few pixels. We might imagine that with only a few pixels “out of place” we should still consider the image a “2”. But how far should that go? It’s a question of human visual perception. And, yes, the answer would no doubt be different for bees or octopuses—and potentially utterly different for putative aliens.

Because for some reason—that maybe one day we’ll have a scientific-style understanding of—if we always pick the highest-ranked word, we’ll typically get a very “flat” essay, that never seems to “show any creativity” (and even sometimes repeats word for word). But if sometimes (at random) we pick lower-ranked words, we get a “more interesting” essay. The fact that there’s randomness here means that if we use the same prompt multiple times, we’re likely to get different essays each time. And, in keeping with the idea of voodoo, there’s a particular so-called “temperature” parameter that determines how often lower-ranked words will be used, and for essay generation, it turns out that a “temperature” of 0.8 seems best. (It’s worth emphasizing that there’s no “theory” being used here; it’s just a matter of what’s been found to work in practice.

With sufficiently much English text we can get pretty good estimates not just for probabilities of single letters or pairs of letters (2-grams), but also for longer runs of letters. And if we generate “random words” with progressively longer n-gram probabilities, we see that they get progressively “more realistic”: But let’s now assume—more or less as ChatGPT does—that we’re dealing with whole words, not letters. There are about 40,000 reasonably commonly used words in English. And by looking at a large corpus of English text (say a few million books, with altogether a few hundred billion words), we can get an estimate of how common each word is. And using this we can start generating “sentences”, in which each word is independently picked at random, with the same probability that it appears in the corpus. Here’s a sample of what we get: Not surprisingly, this is nonsense. So how can we do better? Just like with letters, we can start taking into account not just probabilities for single words but probabilities for pairs or longer n-grams of words.

When we make a neural net to distinguish cats from dogs we don’t effectively have to write a program that (say) explicitly finds whiskers; instead we just show lots of examples of what’s a cat and what’s a dog, and then have the network “machine learn” from these how to distinguish them. And the point is that the trained network “generalizes” from the particular examples it’s shown.

But it’s notable that the first few layers of a neural net like the one we’re showing here seem to pick out aspects of images (like edges of objects) that seem to be similar to ones we know are picked out by the first level of visual processing in brains.

So how does neural net training actually work? Essentially what we’re always trying to do is to find weights that make the neural net successfully reproduce the examples we’ve given. And then we’re relying on the neural net to “interpolate” (or “generalize”) “between” these examples in a “reasonable” way.

Particularly over the past decade, there’ve been many advances in the art of training neural nets. And, yes, it is basically an art.