Reactive control

So far, this chapter has dealt with control solutions that require the presence of a specific model of the environment, so that either a reference path or a field of potentials can be formed on its basis. But there are some difficulties with this approach. Firstly, it is often difficult to obtain sufficiently accurate models, especially in a complex or remote environment, such as the surface of Mars. Secondly, even in those cases where there is an opportunity to make a model with sufficient accuracy, the computational complexity and localization errors can lead to the fact that these methods will be practically not applicable. In certain circumstances, one of the types of a reflex agent project becomes more appropriate - a project based on the so-called reactive control.

One example of such a project is a six-legged robot, or hexapod , shown in Figure a) , which is designed for walking over rough terrain. In general, the robot sensors do not allow the formation of terrain models with an accuracy sufficient for any of the path planning methods described in the previous section. In addition, even in the case of using sufficiently accurate sensors, the task of planning a path is not solvable using the available computational tools due to the presence of twelve degrees of freedom (two for each leg).

Nevertheless, it is possible to define the specification of the controller directly, without using an explicit model of the environment. For the considered example of a walking robot, after selecting the appropriate level of abstraction, the task of determining the control law turned out to be surprisingly simple. In an acceptable control law, the cyclic movement of each leg can be provided so that this leg touches the ground for some time and moves in the air at the rest of the time. Coordination of all six legs should be carried out so that three of them (located at opposite ends) are always on the ground to provide physical support. This principle of control can be easily programmed, and it fully justifies itself on level ground. And on rough terrain, the movement of legs forward can be hindered by obstacles. This difficulty can be overcome with the help of an extremely simple control rule: if the movement of a leg is blocked forward, you should take it back a little, lift it higher and make another attempt. The resulting controller is shown in Figure b) as a finite state machine; it is a reflexing agent with state support, in which the internal state is represented by the index of the current state of the automaton (from s ₁ to s ₄ ).

An example of the use of reactive control: six-legged robot (a); augmented finite state machine (Augmented Finite State Machine - AFSM) for controlling one leg (b). The AFSM automaton responds to sensory feedback: if a leg cannot move forward when performing the stage of its rotation and transfer in the forward direction, then it rises every time higher and higher

Practice has shown that the varieties of such a simple controller, acting on the basis of feedback, allow us to implement extremely reliable ways of walking, with the help of which the robot freely maneuvers on rough terrain. It is obvious that in such a controller a model is not used, moreover, an algorithmic conclusion is not carried out and no search is performed to generate control actions. During the operation of such a controller, feedback from the environment plays a crucial role in the development of robot behavior. The robot software itself, taken separately, does not determine what actually happens after the robot enters an environment. Behavior resulting from the interaction of the (simple) controller and the (complex) environment is often referred to as emergent behavior (ie, not planned behavior, but due to the situation). Strictly speaking, all the robots discussed in this chapter show emergent behavior due to the fact that none of the models used in them is ideal. But according to tradition, this term is used to refer only to such management methods that do not use explicitly specified environmental models. In addition, emergent behavior is characteristic of a significant part of biological organisms.

From a formal point of view, reactive controllers are one of the forms of policy implementation for MDP tasks (or, if they have an internal state, for POMDP tasks). In robotics, an approach that involves drawing up such a policy manually is of great practical importance, since it is often impossible to formulate an exact model. Some of the methods (such as Q-training and policy search) do not require an environment model and allow you to create high-quality controllers for robots, but instead require the provision of huge amounts of training data.