Autonomics: In search of a foundation for next-generation autonomous systems

Over the years, many definitions have been offered for autonomy (e.g., refs. 11–13). “Autonomic computing” (e.g., ref. 14) focuses on systems capable of self-management, and in particular, automating dynamic configuration. The research area of agent-based design and in particular “multiagent systems” (e.g., ref. 15) offers a perspective of autonomy in paying special attention to the issue of combining local goals with collaboration rules and distributed algorithms to achieve system-wide overall goals. Autonomy is often associated with “self-awareness” (e.g., ref. 16), which implies the system’s ability to perceive changes of the environment and use “knowledge” of its own states to react adequately, so that a set of goals is achieved. “Symbiotic computing” (e.g., ref. 17) studies how autonomous systems can interface and collaborate with humans and with complex organizations, considering the many technical, commercial, and ethical implications thereof. Also, controllers designed with patterns like Sense–Decide–Act or Monitor–Analyze–Plan–Execute (MAPE), are sometimes referred to as being autonomous; however, such control loops alone are clearly not sufficient for a system to move around, conduct itself, and function correctly in arbitrary real-world open environments.

In order to streamline the ensuing discussion, we offer, in this subsection and in the next one, definitions for some basic concepts (see also refs. 18–20), and illustrate them with an example of an AV for factory-floor and plant-yard deliveries (termed here FFAV).

“Systems” are the artifacts that development teams are out to build. A system works within, and reacts to, an external environment, and it consists of two types of components, agents and objects (the latter, as explained below, are typically not built by the development team), which, as we shall see, operate within a common internal system environment. The coordinated collective behavior of the system’s agents and objects is designed to meet some global, system-wide goals.

“Objects” are those components whose programmed behavior is not affected during system development. For example, objects of the FFAV system may include ready-made components, such as the motor, a set of cameras, or a steering mechanism whose input is, say, the desired angle of the front wheels. A system often interacts with objects that are not part of it but are part of the environment. Such are, in the case of the FFAV, machines on the factory floor (which may be mere obstacles or recipients of deliveries), and packages to be delivered. Objects have “states,” which can be changed by agents or by other objects, or can change “spontaneously,” for internal reasons. These internal and external objects become part of the system’s internal and external environments, respectively.

“Agents” are the main behavioral elements of an autonomous system. They are those designed (programmed, built) as part of the system’s development process.^† Agents have “agency”: They are proactive and pursue specific goals that may change dynamically. Agents can monitor objects from the internal and external environments and can change their states. They can also coordinate their own actions with other agents. Thus, the FFAV system may either have a single agent for all its functions, or separate agents for different tasks, such as work scheduling, route planning, travel control, gripping, and carrying. Agents can themselves be autonomous systems, and can, in turn, hierarchically contain other agents and systems.

The “internal environment” is the lower-level physical and virtual infrastructure used by the system’s agents and objects. It may include the computer/processor/memory, batteries and other power sources, the operating system, communication hardware and software, and database management software.

The “external environment” of a system (often simply called the “environment”) is the collection of all entities with which the system might interact. It may include other systems (with their objects and agents) and stand-alone objects, and any other physical or virtual entities that may affect, or be affected by, the system’s behavior. Key entities in any system’s external environment are of course humans—with their unpredictability, initiative, and power and authority to modify system behavior.

We say that a system or an agent (for simplicity, we shall stick to system below) manifests “autonomous behavior” if it embodies the following five behavioral functions, which are carried out with little or no intervention from humans or from other systems.

Two functions are combined to enable the system to build for itself a useful representation of the state of the external environment. “Perception” is the function that inputs stimuli, interprets their basic meaning, and removes ambiguity, yielding relevant information. Often perception has to deal with multimodal inputs, such as vision, sound, heat, touch, radar, and data communication from other systems, all obtained using mode-specific sensors and input devices, and has to then amalgamate the received information. The second function is “Model Update,” which uses the information provided by Perception to create and constantly update an integrated run-time model representing the system’s environment and its states. This model will then be used in ongoing decision making.

Two other functions constitute the system’s adaptive decision process. This means that decisions consider many possibly conflicting goals, in a way that depends on the system’s current state and that of the environment. “Goal Management” chooses from among the set of goals the ones that are relevant to the current state. “Planning” computes a plan to achieve the set of goals produced by Goal Management, subject to state-dependent constraints; this is the agent’s action in response to the current environment state, and may consist of a sequence of commands to be executed by actuators. Both Goal Management and Planning may take part in addressing conflicting goals; they do so by, for example, prioritization, constraint resolution, proceeding on multiple concurrent plans (deferring the decision), and consultation with humans or with other systems.

The fifth function that characterizes autonomous behavior is “Self-Adaptation,” which caters for dynamic adjustment over time of the system’s goals and the goal management and planning processes, through learning and reasoning, based on the evolving state of the system and its environment. Such adaptivity could come in many forms: very near term, e.g., using trial and error and recent experiences to go around an unfamiliar and not-previously specified obstacle; “life-long learning” of the system, constantly reevaluating its entire history of sensor information, its actions, and its successes and failures, in order to better achieve its goals in a dynamic unpredictable environment; even achieving certain independence, where the autonomous system studies the entire changing environment, far beyond its task scope, and is able to adjust not only its operation, but also its goals (consider an AV that upon discovering that a key user cannot travel, arranges on its own to fetch people or objects to that user, or facilitates a teleconference). The question of when should system adaptivity be curtailed and humans should be called upon is a broad one, and requires much thought and further research.

Next-generation autonomous systems, both those that are already beginning to emerge and definitely those of the future, differ from existing systems in the several key aspects. These aspects, which we now discuss, position the “next generation” adjective we attach to the systems we are talking about here beyond the realm of the ones that are being developed now and will be operational in the very near future. Rather, our subject matter are those systems we feel will be prevalent only a good number of years henceforth.

Behavioral specification is needed in virtually all stages and activities of the development process: requirements, design, simulation, testing, verification, and validation. Behavior should preferably be specified in some rigorously defined language with agreed-upon dynamic semantics, but, at the very least, can also be done in a way that can be mapped directly to precise and technically oriented natural language descriptions. Although numerous diverse computer languages have been developed for this purpose—procedural, declarative logic-based, state-based, scenario-based, and more—we argue that in next-generation autonomous systems the very specification of behavior introduces new issues that call for extensive research. Such specifications should cover both classical desired and undesired sequences of events and actions, related goals and their mutual relationships, actual behavior, as observed and interpreted by machines and by people, and inferred behaviors that may have not been fully observed. The specification should serve to build systems, to test them, to enable communication between systems, between humans, and between systems and humans, and to enable the meta-analysis of such specification.

When it comes to complex autonomous systems, the specification of even a single simple goal is hard. Assume that, for the first time, an employer wants to completely automate the floor-cleaning process. What kinds of specification are we after? Should it be focused on actions (e.g., where and how to sweep), on environment objects and entities (e.g., what kinds of dirt should be removed, and from where), or on states and results (e.g., what should the floors and shelves look like once the job is done)? How should one tell developers (and the system) about the need to move small objects or unplug devices that are in the way, or about dealing with such risks as breaking something?

We believe that what we need here are ways to describe the relevant “world” and its associated behaviors. For this, we propose to develop domain-specific ontologies of objects, properties, actions, and relations. This direction may extend or learn from current ontologies like the CYC project (26), Google’s Knowledge Graph, the OWL web ontology language and others, but may also take on different design directions. For example, the very essence of entities may be associated with action-related information about what these entities do, what can be done with them, or how other systems are expected to react thereto. The goal in this case is not to achieve comprehensive (yet elusive) knowledge, but to enable decisions within the system’s action repertoire. For example, if the FFAV sees in its path an object it cannot recognize, it should be able to present a picture and perhaps sensor information to other facilities, such as a server in the cloud. The server might then inform the FFAV as to what the object is, so it can apply its existing rules, or it may instruct the FFAV to take certain actions based on the server’s knowledge and logic.

In the context of autonomous systems, some progress along these lines can be found in, e.g., traffic sequence charts (27), US National Highway Traffic Safety Administration scenarios (28), autoware software for AVs (29), and open source simulators like CARLA (3). Each of these uses its own terms and concepts as building blocks in a bottom-up tool construction.

Beyond the issue of specifying single goals lies the extreme difficulty of specifying how the system should balance, prioritize, or weigh several, often competing goals under a bewildering multitude of circumstances. Even in a single given situation, and even if we allow the use of natural language for specification, it is often almost impossible to state what the system should or should not do. Many future generation autonomous systems will have to make complex decisions involving major human and business risks, and we doubt that stakeholders can prescribe in advance what the system should do in each case.

Furthermore, of course, in addition to such technical issues of specification, there are also weighty ethical issues. Courts of law rule on whether a decision made by a human was right or wrong, negligent or not, in line with what is expected of a reasonable person. This will become significantly harder in the realm of autonomous systems, which, for example, will have to determine where exactly to make the initial incision during a surgical procedure, or to decide in a split second between two very bad alternatives in an emergency driving situation.

During simulations, stakeholders often encounter emergent properties and unexpected behaviors that were not mentioned in the development process. For example, when observing the behavior of an AV, such as an FFAV or a golf cart, one may notice that it sometimes repeats a path with unusual precision, creating unexpected wear on the floor, ground, or grass. New requirements may be desired as a result, such as randomizing paths, documenting a limited set of supported surfaces, or even taking advantage of this kind of predictability in other parts of the system.

Dynamic changes in specifications constitute yet another complicating aspect. It is not unusual for a human operating on a certain task to receive new information or instructions about changes in goals, in means to achieve them, or in assumptions about the environment. Humans most often deal reasonably well with such updates. Autonomous systems will have to support such modes of communication and of reactive behavior (16).

Making this happen is not easy. It is not even clear how to specify a formal version of a “trouble ticket,” which describes an event, property, or pattern that was noticed by the human observer but was not part of the original specification. Furthermore, one would want to automate the detection and articulation of emergent properties, since testing and simulation are likely to be highly automated, with limited opportunity for human observation. For anticipated behavior (desired or undesired), this would be quite similar to testing; but, for unexpected behavior, automating the capturing itself in a formal, yet succinct and intuitive way, would appear to be a major challenge, requiring substantial extension of current research in fields such as specification and mining anomaly detection (see, e.g., refs. 30–32). In fact, we believe that Knuth’s famous quote, “Beware of bugs in the above code; I have only proved it correct, not tried it,” goes beyond recognizing the importance of testing given the limitations of formal methods and correctness proofs. It can be used to support our belief that testing is a must also because just observing the system in operation yields totally new insights about what the system does and does not do, and what it should or should not do.

Explainability and interpretability are especially relevant to emergent properties, particularly for those parts of the solutions based on neural nets and other “black-box” approaches. In a way, explanations induce a model on the seemingly model-less machine-learning solution. Additionally, here too, summarizing such execution patterns automatically is a challenging problem that is the subject of active research. We discuss this issue somewhat in Challenge III: Combining “Model-Based” and “Data-Driven” Approaches below.

The task of specifying and explaining behavior that must dynamically reconcile multiple goals can be aided by initially adding weights, priorities, and mutual constraints to goals, as well as just observing the system in operation and endowing the components responsible for the various goals with dynamic internal negotiation capabilities. A key role will also be played by mechanisms for specifying and handling contingent behavior, which reacts to and handles negative conditions directly related to the system’s actual behavior. The ability to provide concise explanations of the system’s decisions, both in real time and after the fact, will be of great value, allowing developers, and the system itself, to judge the programmed decisions and adjust them as needed.

By analysis, we mean simulation, testing, formal verification, and system validation against the tacit needs of the stakeholders (STV&V for short). While these techniques will be of paramount importance for next-generation autonomous systems, it is well known that none of them provides complete assurances even for current systems, so they will have to be used in ways that complement each other.

The various techniques comprising STV&V all involve one manner or another of executing a system or a model thereof in a controlled fashion, and/or traversing or analyzing the resulting states. Simulation is perhaps the most “hands on” of these, and facilitates observing emergent behaviors—desired, undesired, and not yet specified—under a variety of conditions. However, the simulated environment will always be an abstraction and simplification of reality. There are numerous simulation tools relevant to AVs; see, e.g., refs. 4, 8, 11, and 17. While these are effective and provide important features—albeit, spread across different tools—the foundation proposed here calls for additional important capabilities, such as far greater user control over environment variability and the ability to automatically detect and evaluate new emergent properties that were not in the original test specifications.

One of the main reasons that satisfactory STV&V analysis calls for foundational work, is the vast number of objects and variables involved in complex autonomous systems, and the even greater number and intricacy of the interactions between them. This is what we now discuss.

Autonomous systems will typically have to deal with numerous new elements, which are often ignored or simplified, or are controlled by other systems. Just think of an FFAV deployed at a busy outdoor factory yard, with people, equipment, and vehicles moving around, possessing distinct shapes, colors, reflection types, textures, sizes, locations, positioning, and routes. Faithful simulation and effective testing of these systems and environments is a formidable task. Furthermore, as to interactions thereof, here the problem becomes alarmingly worse. The system’s “intentional” activities may be reasonably controlled, but the number and complexity of the possible interactions and indirect effects between all objects in the environment and the system, are mind-boggling. Consider testing a very small FFAV working its way through a throng of humans and machines, perhaps even other living creatures—as in a livestock show. The interactions it has to deal with are not limited to the obvious goals of, e.g., reaching a destination while avoiding collisions and abiding by traffic laws. How about the interactions between objects that arise when the FFAV tries to avoid splashing passersby when it crosses a puddle? How should it deal with cases where it might inadvertently startle humans when quietly and suddenly showing up at their side, or when it gets entangled in a cable or in loose fabric?

As stated earlier, the fact that humans inhabit the external environment of autonomous systems further complicates matters. For example, a human’s ability to learn and adapt to new conditions, is likely to surpass that of most systems for a long time to come. Such human adaptivity may be relied upon for certain systems, where developers might allow systems to modify their own behavior even without first coordinating with affected humans, trusting that the humans will “figure things out” on their own. This adaptivity is also what often allows humans to correct or override the behavior of systems they control. Either way, it is useful to analyze the boundaries of environment assumptions, by taking into account also the whims of humans.

As we did for the specification challenge, we list below some of the issues that the autonomics foundation should address regarding analysis. One, which is a precondition to any kind of analysis, has to do with the “modeling of environments.” We envision using domain-specific libraries for various kinds of systems and tasks, in order to deal with the physical three-dimensional space of real-world objects and their mobility. These libraries will be different for different application areas. Just think of the very different kinds of environments relevant to a medical system and a transportation system. The environment would have to be modeled using languages and tools that are able to describe knowledge of the environment and assumptions thereof, and to achieve a desired level of realism by controlling abstraction levels and simulation granularity.

The second issue the foundation will have to address, which is particularly important for analysis, involves the “infrastructure” needed for STV&V. This would have to include mechanisms that orchestrate and control executions; set up the physical or virtual environments; play the role of the environment when needed; observe, record, analyze, and act upon the system’s actual behavior; and interface with engineers for these and related functions. We want to be able to test and simulate autonomous agents in interaction with the complex cyber physical environment for which they are being built. The infrastructure should be “state-aware” and transparent, being able to communicate with engineers using natural interfaces and logs that describe intuitively the state of the external environment, the internal state of the system and its agents, and the state of agents’ perception of the external environment.

For example, assume you want to figure out what the FFAV will do when it faces an obstacle consisting of two posts placed at a distance apart that is barely more than the width of the FFAV. Will it move between them or bypass them? Standard testing and simulation techniques call for actually placing the obstacles and observing the system’s behavior. However, unless we also carefully check the feedback from the tested system as to what it “thinks” it saw, one cannot be sure if it perceived the conditions as intended. Thus, the FFAV may indeed pass between the posts, but for the wrong reason: It might have misclassified one or both of them.

Such perception control is a particular case of state awareness, where during STV&V the infrastructure is able to report the system’s state and that of the environment; can guide the operation of system components based on the states of others; and can report, and react to, things the system does and does not do, its execution paths, etc. The complexity of all this is amplified by the unpredictability of behavior. Even the relatively simple problem of determining which of the agent’s states and interactions may occur in parallel with which others is extremely difficult.

The third major challenge of the analysis part of the foundation has to do with controlling and measuring the “behavioral coverage” achieved via testing, whether virtual/in silico or by deployment of the autonomous system in the real physical world. Such coverage refers to the space of all composite systems states, across multiple components, as well as of the paths and scenarios for reaching these states. Current testing methods focus on a variety of relevant aspects, including coverage of development entities (like statements, components, program changes, and requirements/assertions), automated test execution and evaluation, and automated generation of scenarios (33, 30). Makers of AVs sometimes present the number of miles (real and simulated) that their products have driven (see, e.g., refs. 34 and 35) as an indicator of behavioral coverage.

For the kinds of systems we have in mind, which will be deployed in large numbers worldwide, these approaches are inadequate. Even what appears to be the bare minimum here—a practical approach to measuring overall composite state coverage for both system and environment—is already hard enough (see discussion in ref. 36). We envision having to develop techniques to automatically generate rich sets of scenarios, subject to criteria that can be external, i.e., from the environment and the real world, or internal, such as intricate behavioral combinations of specification and implementation entities. Furthermore, given the inability to exhaustively cover all run-time possibilities, we need support for accelerated metamorphic testing in physical environments; i.e., checking thoroughly that the system behaves correctly for a given scenario, and then quickly providing assurances for many other scenarios that differ from the basic scenario only by small physical changes. We also need fitting criteria for evaluating the testing process itself.

Finally, the autonomics foundation will have to address “formal verification.” Even the best current verification methods can be used successfully only for single components or for greatly simplified models of the entire system. Also, not only is the behavioral specification of the system itself very hard, but it is no easier to specify the assertions that describe the behavior we want to verify in terms that are readily aligned with the expectations of the human users and engineers. This is further complicated by the fact that whether some behavior is desired or not may not by a binary decision but a quantitative one, spanning multiple scales (37).

Since many next-generation autonomous systems will have components based on machine learning (ML), the formal verification of neural nets, and the ability to supply adequate explanations of their internal behavior, will become increasingly important. These problems are long recognized as being very difficult, and there is an emerging field of research around them, whose initial results look very promising (38, 39). Accordingly, the next section takes a closer look at the challenge involved in incorporating such AI-based learning techniques into the foundation.

In the ensuing discussion we use the term model-based to include classical software development approaches, all of which employ traditional programming languages and prescribe step-by-step processes and/or rules that are meticulously handcrafted and organized by humans. This also includes model-drive engineering (MDE) techniques that use languages like those that have been made part of the broad unified modeling language (UML). However, we do not restrict ourselves to MDE. We have decided to use the term model-based in order to emphasize the fact that the designer is required to build and provide a thorough technical description (a model) of the problem, its inputs, its outputs, and the required processing and behavior, in terms that are aligned with the problem domain.

In contrast, we use the term data-driven to encompass all techniques that involve ML (including, but not restricted to, deep neural networks), statistical analysis, pattern recognition, and all related forms of computing in which the system’s behavior is derived from supervised or unsupervised observation. The latter can include observing input and output events and occurrences in the real world or in the processing of other systems, even that of earlier versions of the system under development. The desired behavior of the system we are developing is thus inferred; not prescribed as in model-based approaches.

There is a growing call to find ways to combine the two techniques, leveraging their relative advantages to complement each other (40–42). Nevertheless, there is still no agreement on how to do this, the combination being very different from integration practices in classical engineering. In addition, of course, due to the new challenges involved, the problem is further exacerbated for next-generation autonomous systems.

There are several differences between traditional software development and constructing solutions based on ML, which must be taken into account when trying to integrate the two. To better concentrate on the integration issue in this subsection, we disregard the still-open research problems in each of them.

The first difference involves the “general life cycle.” Traditional software engineering—in any of a number of classical life cycle methodologies—calls for requirements elicitation and specification, design, code, testing, and so on. In contrast, developing a module or system based on ML involves totally different stages, such as the collection, validation, and sampling of training data, the actual training, evaluation, revision, and retraining, etc.

The second difference concerns “specifying requirements.” Consider even very simple cases, such as requiring that an electrical switch must turn off when the temperature reaches 80°, or that the brakes must be activated when a stationary obstacle is sensed and the stopping time at the current speed is less than 1 s. These are well defined, and engineers can translate them easily into working components, but for a system trained to handle excessive heat or avoid collision based on positive and negative examples, it is not at all clear how to use the requirements or how to incorporate them into the respective ML components.

Related to specifying requirements is the issue of after-the-fact “explainability” (also referred to “interpretability”). This calls for the ability to justify, or rationalize, a particular system decision using problem-related parameters and arguments. More generally, we want to be able to describe in this way what the system does and the underlying rules, algorithms, and computations it uses. Despite the remarkable success of neural nets in performing many kinds of tasks, their internal workings are often a mystery. Current ideas addressing this problem are still a far cry from the situation with traditional programming, for which engineering practices recommend producing code that is easy to read and understand, and to enhance it with ample comments. Moreover, even if explainability and interpretability tools are eventually able to extract the tacit rules behind the operation of large neural nets, the way these rules relate to the net’s actual mechanisms will be very different from the relation between natural language descriptions and source code in classical programming languages.

In addition, the difficulty in specifying the behavior of neural nets and explaining what they do and why they do it, makes it very difficult to analyze their behavior. While some initial work has been done on checking properties thereof (see, e.g., ref. 39), there is still much to be done on the testing and verification of systems based on learning.

An important related difference involves “decomposability,” which is crucial in most stages of development, e.g., for understanding and anticipating system behavior, finding and fixing errors, carrying out enhancements, and assessing the impact of changes. In model-based designs, most system artifacts can be hierarchically decomposed into well-understood functional and structural elements, the role they play in the full system being more or less clear. In contrast, the design of data-based ML solutions is typically accompanied by an end-to-end mindset—system-based or problem-based. Being able to decompose a ML solution into meaningful parts appears to be an interesting challenge, which will, of course, bear upon explainability and verification.

Finally, we mention the differences between the two approaches with regard to their “trustworthiness and certification,” which are clearly related to testing and verification. Many kinds of autonomous systems are highly critical; failure to meet their expected behavior can be disastrous. Critical system design calls for providing appropriate trustworthiness guarantees for the system’s functions and reliability. These are often specified in standards, like DO178B for avionic systems and ISO 26262 for electronic components in the automotive industry. In principle, model-based techniques give rise to predictability at design time, but components based on ML are not engineered in the same way. Achieving for them an accepted level of trustworthiness and certification requires wholly new technical solutions, since conventional testing and simulation techniques are inadequate given the complexity and unpredictability associated with next-generation autonomous systems. Complementary nontechnical measures, such as risk management, the concept of insurance, or the use of the justice system as deterrent against negligence, are separate issues altogether and are beyond of the scope of this paper.

These significant differences illustrate the magnitude of the methodological and technical integration challenge that the autonomics foundation will have to address. To give a relatively straightforward example of this, consider a proposed system that is to ultimately consist of conventional model-driven components (based, e.g., on an object model, algorithms, scenarios, rules, and decision tables) and ML data-driven components (based, e.g., on neural nets). At some point, the engineers will have to decide which subproblems should be solved using which of the two approaches. Sometimes the answer is easy: For example, reading traffic signs can be fully ML based, while the decision to remain below a known speed limit can be model based. Staying in lane on a clean, well-marked highway might be model based, while negotiating a road surface covered with sand could very well be ML based. However, in many cases, the choice between the two approaches will be a lot more difficult. Besides, designing the run-time decision process that determines which situation is the relevant one, in order to activate the proper component (e.g., whether the road surface is clean or sandy), is in itself a design challenge; the parameters of the decision may not be clear-cut, and the designers will have to choose whether to use a model-based, data-based, or hybrid approach for this subtask.

The foundation should describe and discuss various specific approaches to the integration. One example is a “pipeline” approach, where some preprocessing is done by one method, and interim results are passed on to the other method. In the “divide-and-conquer” approach, the problem is divided into subproblems, and different techniques can be applied to each; the actual division can itself be model-based or data-driven. In another approach, one method (usually rules) serves as a “protective wrapper” around the other (usually ML-based), constraining the latter to be within some decision domain. In additional variants, an approach that is mostly model based can incorporate several data-driven black boxes to enrich sensor processing or to solve particular subproblems. One can also apply both techniques to develop complete solutions, thus creating redundancy, and then use various composition or “voting” techniques to yield the final system behavior.

By definition, such approaches for composing hybrid solutions from model-based and ML-based components will also imply the first level of the decomposition of the solutions.