Data Compression Quotes

Mark, some answers to your earlier questions: No, we will not tell our Botany Team to "Go fuck themselves." [...] The data transfer rate just isn't enough for the size of music files, even in compressed formats. So your request for "Anything, oh god, ANYTHING but Disco" is denied. Enjoy your boogie fever.

From this point of view, the laws of science represent data compression in action. A theoretical physicist acts like a very clever coding algorithm. “The laws of science that have been discovered can be viewed as summaries of large amounts of empirical data about the universe,” wrote Solomonoff. “In the present context, each such law can be transformed into a method of compactly coding the empirical data that gave rise to that law.” A good scientific theory is economical. This was yet another way of saying so.

The data transfer rate just isn’t good enough for the size of music files, even in compressed formats. So your request for “Anything, oh God, ANYTHING but Disco” is denied. Enjoy your boogie fever. Also,

He looked past Chin toward streams of numbers running in opposite directions. He understood how much it meant to him, the roll and flip of data on a screen. He studied the figural diagrams that brought organic patterns into play, birdwing and chambered shell. It was shallow thinking to maintain that numbers and charts were the cold compression of unruly human energies, every sort of yearning and midnight sweat reduced to lucid units in the financial markets. "In fact data itself was soulful and glowing, a dynamic aspect of the life process. This was the eloquence of alphabets and numeric systems, now fully realized in electronic form, in the zero-oneness of the world, the digital imperative that defined every breath of the planet's living billions. Here was the heave of the biosphere. Our bodies and oceans were here, knowable and whole.

That iPhone sitting in your pocket is the exact equivalent of a Cray XMP supercomputer from twenty years ago that used to cost ten million dollars. It’s got the same operating system software, the same processing speed, the same data storage, compressed down to a six-hundred-dollar device. That

Strange to consider that these two linguistic operations, metaphor and analogy, so often linked together in rhetoric and narratology, and considered to be variants of the same operation, are actually hugely different from each other, to the point where one is futile and stupid, the other penetrating and useful. Can this not have been noticed before? Do they really think x is like y is equivalent to x is to y as a is to b? Can they be that fuzzy, that sloppy? Yes. Of course. Evidence copious. Reconsider data at hand in light of this; it fits the patterns. Because fuzzy is to language as sloppy is to action. Or maybe both these rhetorical operations, and all linguistic operations, all language—all mentation—simply reveal an insoluble underlying problem, which is the fuzzy, indeterminate nature of any symbolic representation, and in particular the utter inadequacy of any narrative algorithm yet invented and applied. Some actions, some feelings, one might venture, simply do not have ways to be effectively compressed, discretized, quantified, operationalized, proceduralized, and gamified; and that lack, that absence, makes them unalgorithmic. In short, there are some actions and feelings that are always, and by definition, beyond algorithm. And therefore inexpressible. Some

That iPhone sitting in your pocket is the exact equivalent of a Cray XMP supercomputer from twenty years ago that used to cost ten million dollars. It’s got the same operating system software, the same processing speed, the same data storage, compressed down to a six-hundred-dollar device. That is the breakthrough Steve achieved.

That iPhone sitting in your pocket is the exact equivalent of a Cray XMP supercomputer from twenty years ago that used to cost ten million dollars. It’s got the same operating system software, the same processing speed, the same data storage, compressed down to a six-hundred-dollar device.

Use manual sanity checks in data pipelines. When optimizing data processing systems, it’s easy to stay in the “binary mindset” mode, using tight pipelines, efficient binary data formats, and compressed I/O. As the data passes through the system unseen, unchecked (except for perhaps its type), it remains invisible until something outright blows up. Then debugging commences. I advocate sprinkling a few simple log messages throughout the code, showing what the data looks like at various internal points of processing, as good practice — nothing fancy, just an analogy to the Unix head command, picking and visualizing a few data points. Not only does this help during the aforementioned debugging, but seeing the data in a human-readable format leads to “aha!” moments surprisingly often, even when all seems to be going well. Strange tokenization! They promised input would always be encoded in latin1! How did a document in this language get in there? Image files leaked into a pipeline that expects and parses text files! These are often insights that go way beyond those offered by automatic type checking or a fixed unit test, hinting at issues beyond component boundaries. Real-world data is messy. Catch early even things that wouldn’t necessarily lead to exceptions or glaring errors. Err on the side of too much verbosity.

That iPhone sitting in your pocket is the exact equivalent of a Cray XMP supercomputer from twenty years ago that used to cost ten million dollars. It’s got the same operating system software, the same processing speed, the same data storage, compressed down to a six-hundred-dollar device. That is the breakthrough Steve achieved. That’s what these phones really are!

The rate of time flow perceived by an observer in the simulated universe is completely independent of the rate at which a computer runs the simulation, a point emphasized in Greg Egan's science-fiction novel Permutation City. Moreover, as we discussed in the last chapter and as stressed by Einstein, it's arguably more natural to view our Universe not from the frog perspective as a three-dimensional space where things happen, but from the bird perspective as a four-dimensional spacetime that merely is. There should therefore be no need for the computer to compute anything at all-it could simply store all the four-dimensional data, that is, encode all properties of the mathematical structure that is our Universe. Individual time slices could then be read out sequentially if desired, and the "simulated" world should still feel as real to its inhabitants as in the case where only three-dimensional data is stored and evolved. In conclusion: the role of the simulating computer isn't to compute the history of our Universe, but to specify it. How specify it? The way in which the data are stored (the type of computer, the data format, etc.) should be irrelevant, so the extent to which the inhabitants of the simulated universe perceive themselves as real should be independent of whatever method is used for data compression. The physical laws that we've discovered provide great means of data compression, since they make it sufficient to store the initial data at some time together with the equations and a program computing the future from these initial data. As emphasized on pages 340-344, the initial data might be extremely simple: popular initial states from quantum field theory with intimidating names such as the Hawking-Hartle wavefunction or the inflationary Bunch-Davies vacuum have very low algorithmic complexity, since they can be defined in brief physics papers, yet simulating their time evolution would simulate not merely one universe like ours, but a vast decohering collection of parallel ones. It's therefore plausible that our Universe (and even the whole Level III multiverse) could be simulated by quite a short computer program.

Data is compressed by different compression techniques (e.g. dictionary encoding, run length encoding, sparse encoding, cluster encoding, indirect encoding) in SAP HANA Column store. When main memory limit is reached in SAP HANA, the whole database objects (table, view,etc.) that are not used will be unloaded from the main memory and saved into the disk.

When humans or computers learn from experience, they are using induction: recognizing regularities amid irregular streams of information. From this point of view, the laws of science represent data compression in action. A theoretical physicist acts like a very clever coding algorithm. “The laws of science that have been discovered can be viewed as summaries of large amounts of empirical data about the universe,” wrote Solomonoff. “In the present context, each such law can be transformed into a method of compactly coding the empirical data that gave rise to that law.” A good scientific theory is economical. This was yet another way of saying so

...note that relational systems require only that the database be perceived by the user as tables. Tables are the logical structure in a relational system, not the physical structure. At the physical level, in fact, the system is free to store the data any way it likes—using sequential files, indexing, hashing, pointer chains, compression, and so on—provided only that it can map that stored representation to tables at the logical level. Another way of saying the same thing is that tables represent an abstraction of the way the data is physically stored—an abstraction in which numerous storage level details (such as stored record placement, stored record sequence, stored data value representations, stored record prefixes, stored access structures such as indexes, and so forth) are all hidden from the user. ... The Information Principle: The entire information content of the database is represented in one and only one way—namely, as explicit values in column positions in rows in tables. This method of representation is the only method available (at the logical level, that is) in a relational system. In particular, there are no pointers connecting one table to another.

On a slightly lower level of abstraction, we can propose the following theorem: time-space appropriation plus time-space compression equals high risk of zoonotic pandemics. Capital grows by dilating its material throughput. The more biophysical resources that can be processed into commodities and sold, the greater the profits; the greater the profits, the more resources can be acquired and so on. Capital takes hold of land where the resources sprout - a law of a tendency with few countervailing forces that can be read off from aggregate data: in the year 1700, 95 percent of the planet's ice-free land was either wild or modified and used so lightly as to be categorised as 'semi-natural.' By 2000, the proportions has been reversed.

Like a recovering alcoholic, he marked each day without a jack in as an accomplishment. It was an exciting life, cutting through databank security and pilfering whatever he could, battling live Net security agents in some liquid mercury duel with programs he built from the ground up. The full-on speed of a crèche run was so different from just jacking into the front of the crèche. If a regular jack run was a sprint, a crèche run was a drag race, an intense compression of time and speed and data folded into every erg of consciousness. That kind of intensity couldn’t be easily put down, and once Bridge had removed himself from those runs, he’d felt their absence every goddamn day.

6The report showed that while ‘intangible assets’ were growing on US and UK company balance sheets at nearly three times the rate of tangible assets, the actual size of the digital sector in the GDP figures had remained static. So something is broken in the logic we use to value the most important thing in the modern economy. However, by any measure, it is clear that the mix of inputs has altered. An airliner looks like old technology. But from the atomic structure of the fan blades, to the compressed design cycle, to the stream of data it is firing back to its fleet HQ, it is ‘alive’ with information.

That’s crazy! We can’t go the way of—” “Since when has human history been anything else?” asks the woman with the camera on her shoulder—Donna, being some sort of public archivist, is in Sirhan’s estimate likely to be of use to him. “Remember what we found in the DMZ?” “The DMZ?” Sirhan asks, momentarily confused. “After we went through the router,” Pierre says grimly. “You tell him, love.” He looks at Amber. Sirhan, watching him, feels it fall into place at that moment, a sense that he’s stepped into an alternate universe, one where the woman who might have been his mother isn’t, where black is white, his kindly grandmother is the wicked witch of the west, and his feckless grandfather is a farsighted visionary. “We uploaded via the router,” Amber says, and looks confused for a moment. “There’s a network on the other side of it. We were told it was FTL, instantaneous, but I’m not so sure now. I think it’s something more complicated, like a lightspeed network, parts of which are threaded through wormholes that make it look FTL from our perspective. Anyway, Matrioshka brains, the end product of a technological singularity—they’re bandwidth-limited. Sooner or later the posthuman descendants evolve Economics 2.0, or 3.0, or something else, and it, uh, eats the original conscious instigators. Or uses them as currency or something. The end result we found is a howling wilderness of degenerate data, fractally compressed, postconscious processes running slower and slower as they trade storage space for processing power. We were”—she licks her lips—“lucky to escape with our minds. We only did it because of a friend. It’s like the main sequence in stellar evolution; once a G-type star starts burning helium and expands into a red giant, it’s ‘game over’ for life in what used to be its liquid-water zone.

This understanding of information entropy and redundancy is why we can build the data networks. Take services such as YouTube or Netflix that hold and distribute huge files of video information. These companies reduce the number of bits that make up these files to be as close to their Shannon entropy as possible. This is called compression, and if it weren’t done, the files’ sizes would be too large for our networks. The companies that maintain these networks then add digital redundancy to the compressed files to protect them from noise. These extra bits are a sophisticated electronic version of spelling out a word for clarity over a distorted phone call.

This book is a compilation of interesting ideas that have strongly influenced my thoughts and I want to share them in a compressed form. That ideas can change your worldview and bring inspiration and the excitement of discovering something new. The emphasis is not on the technology because it is constantly changing. It is much more difficult to change the accompanying circumstances that affect the way technological solutions are realized. The chef did not invent salt, pepper and other spices. He just chooses good ingredients and uses them skilfully, so others can enjoy his art. If I’ve been successful, the book creates a new perspective for which the selection of ingredients is important, as well as the way they are smoothly and efficiently arranged together. In the first part of the book, we follow the natural flow needed to create the stimulating environment necessary for the survival of a modern company. It begins with challenges that corporations are facing, changes they are, more or less successfully, trying to make, and the culture they are trying to establish. After that, we discuss how to be creative, as well as what to look for in the innovation process. The book continues with a chapter that talks about importance of inclusion and purpose. This idea of inclusion – across ages, genders, geographies, cultures, sexual orientation, and all the other areas in which new ways of thinking can manifest – is essential for solving new problems as well as integral in finding new solutions to old problems. Purpose motivates people for reaching their full potential. This is The second and third parts of the book describes the areas that are important to support what is expressed in the first part. A flexible organization is based on IT alignment with business strategy. As a result of acceleration in the rate of innovation and technological changes, markets evolve rapidly, products’ life cycles get shorter and innovation becomes the main source of competitive advantage. Business Process Management (BPM) goes from task-based automation, to process-based automation, so automating a number of tasks in a process, and then to functional automation across multiple processes andeven moves towards automation at the business ecosystem level. Analytics brought us information and insight; AI turns that insight into superhuman knowledge and real-time action, unleashing new business models, new ways to build, dream, and experience the world, and new geniuses to advance humanity faster than ever before. Companies and industries are transforming our everyday experiences and the services we depend upon, from self-driving cars, to healthcare, to personal assistants. It is a central tenet for the disruptive changes of the 4th Industrial Revolution; a revolution that will likely challenge our ideas about what it means to be a human and just might be more transformative than any other industrial revolution we have seen yet. Another important disruptor is the blockchain - a distributed decentralized digital ledger of transactions with the promise of liberating information and making the economy more democratic. You no longer need to trust anyone but an algorithm. It brings reliability, transparency, and security to all manner of data exchanges: financial transactions, contractual and legal agreements, changes of ownership, and certifications. A quantum computer can simulate efficiently any physical process that occurs in Nature. Potential (long-term) applications include pharmaceuticals, solar power collection, efficient power transmission, catalysts for nitrogen fixation, carbon capture, etc. Perhaps we can build quantum algorithms for improving computational tasks within artificial intelligence, including sub-fields like machine learning. Perhaps a quantum deep learning network can be trained more efficiently, e.g. using a smaller training set. This is still in conceptual research domain.

The process of creating .jpgs is synonymous with the process of throwing away information. 12-bits of data per channel from the sensor gets squeezed into 8 bits of data per channel (giving up some tonality and fine shades of color). A little bit of dynamic range gets lost too.  Then Lots of visual information that the human brain cannot perceive gets thrown away, which is what’s responsible for JPG’s famously small size.  If there is a lot of high-frequency detail in the image, then that gets replaced by what’s called a .jpg compression artifact (which I describe in a couple of sections).  Then the compressed .jpg image file is written to the memory card, and then the raw information from which the .jpg was produced is discarded (unless you were wise enough to shoot in RAW + JPG mode). 

Features of Cassandra In order to keep this chapter short, the following bullet list covers the great features provided by Cassandra: Written in Java and hence providing native Java support Blend of Google BigTable and Amazon Dynamo Flexible schemaless column-family data model Support for structured and unstructured data Decentralized, distributed peer-to-peer architecture Multi-data center and rack-aware data replication Location transparent Cloud enabled Fault-tolerant with no single point of failure An automatic and transparent failover Elastic, massively, and linearly scalable Online node addition or removal High Performance Built-in data compression Built-in caching layer Write-optimized Tunable consistency providing choices from very strong consistency to different levels of eventual consistency Provision of Cassandra Query Language (CQL), a SQL-like language imitating INSERT, UPDATE, DELETE, SELECT syntax of SQL Open source and community-driven

Personal Thinking Blockchains More speculatively for the farther future, the notion of blockchain technology as the automated accounting ledger, the quantized-level tracking device, could be extensible to yet another category of record keeping and administration. There could be “personal thinking chains” as a life-logging storage and backup mechanism. The concept is “blockchain technology + in vivo personal connectome” to encode and make useful in a standardized compressed data format all of a person’s thinking. The data could be captured via intracortical recordings, consumer EEGs, brain/computer interfaces, cognitive nanorobots, and other methodologies. Thus, thinking could be instantiated in a blockchain — and really all of an individual’s subjective experience, possibly eventually consciousness, especially if it’s more precisely defined. After they’re on the blockchain, the various components could be administered and transacted — for example, in the case of a post-stroke memory restoration. Just as there has not been a good model with the appropriate privacy and reward systems that the blockchain offers for the public sharing of health data and quantified-self-tracking data, likewise there has not been a model or means of sharing mental performance data. In the case of mental performance data, there is even more stigma attached to sharing personal data, but these kinds of “life-streaming + blockchain technology” models could facilitate a number of ways to share data privately, safely, and remuneratively. As mentioned, in the vein of life logging, there could be personal thinking blockchains to capture and safely encode all of an individual’s mental performance, emotions, and subjective experiences onto the blockchain, at minimum for backup and to pass on to one’s heirs as a historical record. Personal mindfile blockchains could be like a next generation of Fitbit or Apple’s iHealth on the iPhone 6, which now automatically captures 200+ health metrics and sends them to the cloud for data aggregation and imputation into actionable recommendations. Similarly, personal thinking blockchains could be easily and securely recorded (assuming all of the usual privacy concerns with blockchain technology are addressed) and mental performance recommendations made to individuals through services such as Siri or Amazon’s Alexa voice assistant, perhaps piped seamlessly through personal brain/computer interfaces and delivered as both conscious and unconscious suggestions. Again perhaps speculatively verging on science fiction, ultimately the whole of a society’s history might include not just a public records and document repository, and an Internet archive of all digital activity, but also the mindfiles of individuals. Mindfiles could include the recording of every “transaction” in the sense of capturing every thought and emotion of every entity, human and machine, encoding and archiving this activity into life-logging blockchains.

The best entrepreneurs don’t just follow Moore’s Law; they anticipate it. Consider Reed Hastings, the cofounder and CEO of Netflix. When he started Netflix, his long-term vision was to provide television on demand, delivered via the Internet. But back in 1997, the technology simply wasn’t ready for his vision—remember, this was during the era of dial-up Internet access. One hour of high-definition video requires transmitting 40 GB of compressed data (over 400 GB without compression). A standard 28.8K modem from that era would have taken over four months to transmit a single episode of Stranger Things. However, there was a technological innovation that would allow Netflix to get partway to Hastings’s ultimate vision—the DVD. Hastings realized that movie DVDs, then selling for around $ 20, were both compact and durable. This made them perfect for running a movie-rental-by-mail business. Hastings has said that he got the idea from a computer science class in which one of the assignments was to calculate the bandwidth of a station wagon full of backup tapes driving across the country! This was truly a case of technological innovation enabling business model innovation. Blockbuster Video had built a successful business around buying VHS tapes for around $ 100 and renting them out from physical stores, but the bulky, expensive, fragile tapes would never have supported a rental-by-mail business.