Software Scalability Quotes

Concepts and patterns that your brain is sorting through and making sense of are much more scalable and universal than any specific vendor’s technology

The Global Learning XPRIZE competition, which started in 2014, offered $15 million for “open-source, scalable software that will enable children in developing countries to teach themselves basic reading, writing and arithmetic within 15 months.” Results from the winners, Kitkit School and onebillion, suggest that the goal has largely been achieved.

Technology is a powerful force in our society. Data, software, and communication can be used for bad: to entrench unfair power structures, to undermine human rights, and to protect vested interests. But they can also be used for good: to make underrepresented people’s voices heard, to create opportunities for everyone, and to avert disasters. This book is dedicated to everyone working toward the good.

Scalable Social Network Analysis. The SSNA would monitor telephone calls, conference calls, and ATM withdrawals, but it also sought to develop a far more invasive surveillance technology, one that could “capture human activities in surveillance environments.” The Activity Recognition and Monitoring program, or ARM, was modeled after England’s CCTV camera. Surveillance cameras would be set up across the nation, and through the ARM program, they would capture images of people as they went about their daily lives, then save these images to massive data storage banks for computers to examine. Using state-of-the-art facial recognition software, ARM would seek to identify who was behaving outside the computer’s pre-programmed threshold for “ordinary.” The parameters for “ordinary” remain classified.

a young Goldman Sachs banker named Joseph Park was sitting in his apartment, frustrated at the effort required to get access to entertainment. Why should he trek all the way to Blockbuster to rent a movie? He should just be able to open a website, pick out a movie, and have it delivered to his door. Despite raising around $250 million, Kozmo, the company Park founded, went bankrupt in 2001. His biggest mistake was making a brash promise for one-hour delivery of virtually anything, and investing in building national operations to support growth that never happened. One study of over three thousand startups indicates that roughly three out of every four fail because of premature scaling—making investments that the market isn’t yet ready to support. Had Park proceeded more slowly, he might have noticed that with the current technology available, one-hour delivery was an impractical and low-margin business. There was, however, a tremendous demand for online movie rentals. Netflix was just then getting off the ground, and Kozmo might have been able to compete in the area of mail-order rentals and then online movie streaming. Later, he might have been able to capitalize on technological changes that made it possible for Instacart to build a logistics operation that made one-hour grocery delivery scalable and profitable. Since the market is more defined when settlers enter, they can focus on providing superior quality instead of deliberating about what to offer in the first place. “Wouldn’t you rather be second or third and see how the guy in first did, and then . . . improve it?” Malcolm Gladwell asked in an interview. “When ideas get really complicated, and when the world gets complicated, it’s foolish to think the person who’s first can work it all out,” Gladwell remarked. “Most good things, it takes a long time to figure them out.”* Second, there’s reason to believe that the kinds of people who choose to be late movers may be better suited to succeed. Risk seekers are drawn to being first, and they’re prone to making impulsive decisions. Meanwhile, more risk-averse entrepreneurs watch from the sidelines, waiting for the right opportunity and balancing their risk portfolios before entering. In a study of software startups, strategy researchers Elizabeth Pontikes and William Barnett find that when entrepreneurs rush to follow the crowd into hyped markets, their startups are less likely to survive and grow. When entrepreneurs wait for the market to cool down, they have higher odds of success: “Nonconformists . . . that buck the trend are most likely to stay in the market, receive funding, and ultimately go public.” Third, along with being less recklessly ambitious, settlers can improve upon competitors’ technology to make products better. When you’re the first to market, you have to make all the mistakes yourself. Meanwhile, settlers can watch and learn from your errors. “Moving first is a tactic, not a goal,” Peter Thiel writes in Zero to One; “being the first mover doesn’t do you any good if someone else comes along and unseats you.” Fourth, whereas pioneers tend to get stuck in their early offerings, settlers can observe market changes and shifting consumer tastes and adjust accordingly. In a study of the U.S. automobile industry over nearly a century, pioneers had lower survival rates because they struggled to establish legitimacy, developed routines that didn’t fit the market, and became obsolete as consumer needs clarified. Settlers also have the luxury of waiting for the market to be ready. When Warby Parker launched, e-commerce companies had been thriving for more than a decade, though other companies had tried selling glasses online with little success. “There’s no way it would have worked before,” Neil Blumenthal tells me. “We had to wait for Amazon, Zappos, and Blue Nile to get people comfortable buying products they typically wouldn’t order online.

scripting language is a programming language that provides you with the ability to write scripts that are evaluated (or interpreted) by a runtime environment called a script engine (or an interpreter). A script is a sequence of characters that is written using the syntax of a scripting language and used as the source for a program executed by an interpreter. The interpreter parses the scripts, produces intermediate code, which is an internal representation of the program, and executes the intermediate code. The interpreter stores the variables used in a script in data structures called symbol tables. Typically, unlike in a compiled programming language, the source code (called a script) in a scripting language is not compiled but is interpreted at runtime. However, scripts written in some scripting languages may be compiled into Java bytecode that can be run by the JVM. Java 6 added scripting support to the Java platform that lets a Java application execute scripts written in scripting languages such as Rhino JavaScript, Groovy, Jython, JRuby, Nashorn JavaScript, and so on. Two-way communication is supported. It also lets scripts access Java objects created by the host application. The Java runtime and a scripting language runtime can communicate and make use of each other’s features. Support for scripting languages in Java comes through the Java Scripting API. All classes and interfaces in the Java Scripting API are in the javax.script package. Using a scripting language in a Java application provides several advantages: Most scripting languages are dynamically typed, which makes it simpler to write programs. They provide a quicker way to develop and test small applications. Customization by end users is possible. A scripting language may provide domain-specific features that are not available in Java. Scripting languages have some disadvantages as well. For example, dynamic typing is good to write simpler code; however, it turns into a disadvantage when a type is interpreted incorrectly and you have to spend a lot of time debugging it. Scripting support in Java lets you take advantage of both worlds: it allows you to use the Java programming language for developing statically typed, scalable, and high-performance parts of the application and use a scripting language that fits the domain-specific needs for other parts. I will use the term script engine frequently in this book. A script engine is a software component that executes programs written in a particular scripting language. Typically, but not necessarily, a script engine is an implementation of an interpreter for a scripting language. Interpreters for several scripting languages have been implemented in Java. They expose programming interfaces so a Java program may interact with them.

I am ambivalent about cluster servers. At the same time, they are marvelous and kludgy. They can add redundancy and failover to applications that weren’t designed for it. Configuring the cluster server itself is finicky, though, and applications usually have a few small glitches when failing over. The biggest drawback is probably that these run in active/passive mode. So, redundancy is achieved, but scalability is not. I consider cluster servers a Band-Aid for applications that don’t do it themselves.

If the application is event-driven, it can be decoupled into multiple self-contained components. This helps us become more scalable, because we can always add new components or remove old ones without stopping or breaking the system. If errors and failures are passed to the right component, which can handle them as notifications, the application can become more fault-tolerant or resilient. So if we build our system to be event-driven, we can more easily achieve scalability and failure tolerance, and a scalable, decoupled, and error-proof application is fast and responsive to users.

How did Facebook successfully overcome the growth limiter of operational scalability? On the technology side, one of the philosophies that helped Facebook become successful was its famous motto “Move fast and break things.” This emphasis on speed, which came directly from Mark Zuckerberg, allowed Facebook to achieve rapid product development and continuous product improvement. Even today, every new software engineer who joins Facebook is asked to make a revision to the Facebook codebase (potentially affecting millions or even billions of users) on his or her first day of work. However, as Facebook’s user base and engineering team grew to a massive size, Mark had to change the philosophy to “Move fast and break things with stable infrastructure.

Domain concern Architecture characteristics Mergers and acquisitions Interoperability, scalability, adaptability, extensibility Time to market Agility, testability, deployability User satisfaction Performance, availability, fault tolerance, testability, deployability, agility, security Competitive advantage Agility, testability, deployability, scalability, availability, fault tolerance Time and budget Simplicity, feasibility

Programming languages, their features, readability, and interoperation Code reuse across platforms (server vs web vs mobile) Early error detection (compile-time vs runtime error detection, breadth of validation) Availability and cost of hiring the right talent; learning curve for new hires Readability and refactorability of code Approach to code composition, embracing the change Datastore and general approach to data modeling Application-specific data model, and the blast radius from changing it Performance and latency in all tiers and platforms Scalability and redundancy Spiky traffic patterns, autoscaling, capacity planning Error recovery Logging, telemetry, and other instrumentation Reducing complexity User interfaces and their maintainability External APIs User identity and security Hardware and human costs of the infrastructure and its maintenance Enabling multiple concurrent development workstreams Enabling testability Fast-tracking development by adopting third-party frameworks

In the end, all vendor products and application architectures are constrained by the same fundamental principles of distributed computing and underlying physics: applications, and the products they use, run as processes on computers of limited capacity, communicating with one another via protocol stacks and links of nonzero latency. Therefore people need to appreciate that application architecture is the primary determinant of application performance and scalability.

The bugs that cause these kinds of software faults often lie dormant for a long time until they are triggered by an unusual set of circumstances. In those circumstances, it is revealed that the software is making some kind of assumption about its environment — and while that assumption is usually true, it eventually stops being true for some reason [11

We will actively manage this technical debt by ensuring that we invest at least 20% of all Development and Operations cycles on refactoring, investing in automation work and architecture and non-functional requirements (NFRs, sometimes referred to as the “ilities”), such as maintainability, manageability, scalability, reliability, testability, deployability, and security. Figure 11: Invest 20% of cycles on those that create positive, user-invisible value (Source: “Machine Learning and Technical Debt with D. Sculley,” Software Engineering Daily podcast, November 17, 2015,

However, a real implementation may still have to include code to handle the case where something happens that was assumed to be impossible, even if that handling boils down to printf("Sucks to be you") and exit(666) — i.e., letting a human operator clean up the mess [93]. (This is arguably the difference between computer science and software engineering.)

Cloud Service Any service made available to users on demand via the Internet from a cloud computing provider's servers rather than from a company's own on-premises servers is referred to as a cloud service. Cloud services are managed by a cloud services provider and are designed to enable easy, scalable access to applications, resources, and services. CLOUD SERVICES ARE SCALABLE AND DYNAMIC. A cloud service may dynamically scale to meet the needs of its customers, and because the service provider provides all of the necessary hardware and software, a corporation does not need to provision or deploy its own resources or dedicate IT people to operate the service. Online data storage and backup solutions, Web-based e-mail services, hosted office suites and document collaboration services, database processing, managed technical support services, and more are all examples of cloud services.

If each team exposes to other teams a formalized notion of “Here’s what we do, and here’s how to engage with us,” then the cost of coordination is reduced. It’s a way of standardizing those types of interactions, making them into a well-understood and scalable process. You can even put a “price tag” on the services, to aid with internal accounting and resource planning.

Matiyas is the one-stop solution for complete digital transformation. We are a highly promising ERP solution provider for business automation. We are providing world-class solutions to the small and medium scale business. Our consulting and technical expertise provides you with bespoke solutions to business concerns. Our customized enterprise resource planning assures you that there is an optimum deployment of resources which can be monitored on a real-time basis. As digital experts, we provide our esteemed corporate clientele with deep technical insights and the ability to align with the unique needs of modern businesses to achieve industry-specific goals. We offer top-notch digital solutions to Oman startups, SMEs, and established enterprises at a reasonable rate. Our customized solutions can be useful for all major industry verticals including healthcare, manufacturing, oil & gas, services, retail and distribution, trading, non-profit, and public sector. Our scalable ERP solutions are customizable to meet diverse and ever-changing business needs. Our Services: Business Consulting, Implementation, Customization, Configuration, Integration, Localization, Backup, Upgrade, Migration, Hosting, Training & Support. Our Offices: India, Oman, Kuwait, Canada, UAE, Armenia Our Digital Solutions: Inventory Management, Procurement Management, Selling Management, Production Management, EPC Software, Retail POS Management, Manufacturing Management, Project Management, Customer Relationship Management, Accounting & Finance Management, Human Capital Management, Assets Management, Quality Management, Ecommerce, Website, Hospital Management Information System HMIS, Education Management and many more…

the history of software development technology is the story of how to conveniently create plugins to establish a scalable and maintainable system architecture.

loosely coupled services. As a result, it improves the development time attributes—maintainability, testability, deployability, and so on—and enables an organization to develop better software faster. It also improves an application’s scalability, although that’s not the main goal.

the choice of an indirect route requires more well-defined processes, more formal formats, earlier announcement of activities and events, longer lead times for changes and communication and collaboration platforms that are scalable and available 24/7.

good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations

Having software shape the critical path of operational execution has substantial ramifications. Digital, AI-driven processes are more scalable than traditional processes.

The other new role that evolutionary architecture creates has enterprise architects defining enterprise-wide fitness functions. Enterprise architects are typically responsible for enterprise-wide nonfunctional requirements, such as scalability and security. Many organizations lack the ability to automatically assess how well projects perform individually and in aggregate for these characteristics. Once projects adopt fitness functions to protect parts of their architecture, enterprise architects can utilize the same mechanism to verify that enterprise-wide characteristics remain intact.

Tracking data becomes more detailed, analyses become further-reaching, and data is retained for a long time in order to build up detailed profiles of each person for marketing purposes. Now the relationship between the company and the user whose data is being collected starts looking quite different. The user is given a free service and is coaxed into engaging with it as much as possible. The tracking of the user serves not primarily that individual, but rather the needs of the advertisers who are funding the service. I think this relationship can be appropriately described with a word that has more sinister connotations: surveillance. Surveillance As a thought experiment, try replacing the word data with surveillance, and observe if common phrases still sound so good [93]. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.” This thought experiment is unusually polemic for this book, Designing Surveillance-Intensive Applications, but I think that strong words are needed to emphasize this point. In our attempts to make software “eat the world” [94], we have built the greatest mass surveillance infrastructure the world has ever seen. Rushing toward an Internet of Things, we are rapidly approaching a world in which every inhabited space contains at least one internet-connected microphone, in the form of smartphones, smart TVs, voice-controlled assistant devices, baby monitors, and even children’s toys that use cloud-based speech recognition. Many of these devices have a terrible security record [95]. Even the most totalitarian and repressive regimes could only dream of putting a microphone in every room and forcing every person to constantly carry a device capable of tracking their location and movements. Yet we apparently voluntarily, even enthusiastically, throw ourselves into this world of total surveillance. The difference is just that the data is being collected by corporations rather than government agencies [96].

When we develop predictive analytics systems, we are not merely automating a human’s decision by using software to specify the rules for when to say yes or no; we are even leaving the rules themselves to be inferred from data. However, the patterns learned by these systems are opaque: even if there is some correlation in the data, we may not know why. If there is a systematic bias in the input to an algorithm, the system will most likely learn and amplify that bias in its output

define complexity as accidental if it is not inherent in the problem that the software solves (as seen by the users) but arises only from the implementation.

A skilled software craftsperson should understand both good and bad software design practices, starting with what drives the reasoning behind design decisions.

Traditionally, the goal of architecture has been scalability, reliability, and security. But today it’s important that the architecture also enables the rapid and safe delivery of software. You’ll learn that the microservice architecture is an architecture style that gives an application high maintainability, testability, and deployability.

Working with distributed systems is fundamentally different from writing software on a single computer—and the main difference is that there are lots of new and exciting ways for things to go wrong

An individual computer with good software is usually either fully functional or entirely broken, but not something in between. This is a deliberate choice in the design of computers: if an internal fault occurs, we prefer a computer to crash completely rather than returning a wrong result, because wrong results are difficult and confusing to deal with. Thus, computers hide the fuzzy physical reality on which they are implemented and present an idealized system model that operates with mathematical perfection.

Cassandra is now 5 years old. It is an active open source project in the Apache Software Foundation and therefore it is known as Apache Cassandra as well. Cassandra can manage huge volume of structured, semi-structured, and unstructured data in a large distributed cluster across multiple data centers. It provides linear scalability, high performance, fault tolerance, and supports a very flexible data model.

what we believe to be the core competencies of Web 2.0 companies: Services, not packaged software, with cost-effective scalability Control over unique, hard-to-recreate data

What is Tableau Tableau Software is a highly scalable client-server architecture. It is an application that resides on your computer and used for individuals and publishing data sources as well as workbooks to tableau server. Which is allows for instantaneous insight by transforming data into interactive visualizations? Read from OnlineITGuru at Tableau Online Course

Everybody has an intuitive idea of what it means for something to be reliable or unreliable. For software, typical expectations include: The application performs the function that the user expected. It can tolerate the user making mistakes or using the software in unexpected ways. Its performance is good enough for the required use case, under the expected load and data volume. The system prevents any unauthorized access and abuse.