Under the Radar with Active Inference

Back to Portfolio of Projects | LearnableLoopAI.com | Blog |

0 Active Inference: Bridging Minds and Machines

In recent years, the landscape of machine learning has undergone a profound transformation with the emergence of active inference, a novel paradigm that draws inspiration from the principles of biological systems to inform intelligent decision-making processes. Unlike traditional approaches to machine learning, which often passively receive data and adjust internal parameters to optimize performance, active inference represents a dynamic and interactive framework where agents actively engage with their environment to gather information and make decisions in real-time.

At its core, active inference is rooted in the notion of agents as embodied entities situated within their environments, constantly interacting with and influencing their surroundings. This perspective mirrors the fundamental processes observed in living organisms, where perception, action, and cognition are deeply intertwined to facilitate adaptive behavior. By leveraging this holistic view of intelligence, active inference offers a unified framework that seamlessly integrates perception, decision-making, and action, thereby enabling agents to navigate complex and uncertain environments more effectively.

One of the defining features of active inference is its emphasis on the active acquisition of information. Rather than waiting passively for sensory inputs, agents proactively select actions that are expected to yield the most informative outcomes, thus guiding their interactions with the environment. This active exploration not only enables agents to reduce uncertainty and make more informed decisions but also allows them to actively shape their environments to better suit their goals and objectives.

Furthermore, active inference places a strong emphasis on the hierarchical organization of decision-making processes, recognizing that complex behaviors often emerge from the interaction of multiple levels of abstraction. At each level, agents engage in a continuous cycle of prediction, inference, and action, where higher-level representations guide lower-level processes while simultaneously being refined and updated based on incoming sensory information.

The applications of active inference span a wide range of domains, including robotics, autonomous systems, neuroscience, and cognitive science. In robotics, active inference offers a promising approach for developing robots that can adapt and learn in real-time, even in unpredictable and dynamic environments. In neuroscience and cognitive science, active inference provides a theoretical framework for understanding the computational principles underlying perception, action, and decision-making in biological systems.

In conclusion, active inference represents a paradigm shift in machine learning, offering a principled and unified framework for understanding and implementing intelligent behavior in artificial systems. By drawing inspiration from the principles of biological systems, active inference holds the promise of revolutionizing our approach to building intelligent machines and understanding the nature of intelligence itself.

1 BUSINESS UNDERSTANDING

Military operating units (for example artillery units, tank units, ships, combat and reconnaissance aircraft) can often be detected by probing enemy radar systems (for example ship search radars and ground search radars). Making use of suitable instrumentation an operating military unit can detect the presence of enemy signals. There is a tradeoff between moving the unit radially as far away from the signal source of probing radar (for safety) and, having the unit close enough to have some form of engagement capability. The problem is how far away a unit should be moved to become relatively invisible, yet not too far away to have the potential for engagement.

The current project attempts to find some traction in solving this problem. It makes use of an example problem from the RxInfer Julia package and modifies and expands it to develop a proof of concept.

We will have an active inference agent that directs a unit to relocate radially within an area with signal presence of enemy probing radar. The agent will attempt to position the unit at a desired signal strength relative to the enemy source. This desired signal strength will be presented as an input to the agent. For this proof-of-concept (POC), its value will be 5%.

versioninfo() ## Julia version

Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
  JULIA_NUM_THREADS =

import Pkg
Pkg.add(Pkg.PackageSpec(;name="RxInfer", version="3.6.0"))
Pkg.add("Random")
Pkg.add(Pkg.PackageSpec(;name="Plots"))
Pkg.add(Pkg.PackageSpec(;name="LaTeXStrings"))
Pkg.add("Distributions")

using RxInfer
using Random; Random.seed!(1957) 
using Distributions 
using Plots

import RxInfer.ReactiveMP: getrecent, messageout

   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

Pkg.status()

Status `~/.julia/environments/v1.10/Project.toml`
  [31c24e10] Distributions v0.25.111
  [b964fa9f] LaTeXStrings v1.3.1
  [91a5bcdd] Plots v1.40.8
  [86711068] RxInfer v3.6.0
  [9a3f8284] Random

2 DATA UNDERSTANDING

There is no pre-existing data to be analyzed.

3 DATA PREPARATION

There is no pre-existing data to be prepared.

4 MODELING

4.1 Narrative

Please review the narrative in section 1.

4.2 Core Elements

This section attempts to answer three important questions:

What metrics are we going to track?
What decisions do we intend to make?
What are the sources of uncertainty?

For this problem, we will only track the signal strength of an enemy radar as measured by a military unit.

Decisions will be in the form of agent-prescribed velocity actions on a unit.

The only source of uncertainty relating to the environment will be the noise in the measurements of the signal strength.

4.3 System-Under-Steer / Environment / Generative Process

The signal strength $ρ$ is a function of position $s ̃$ where position is the distance between the signal source and a military unit, relative to the signal source. It follows the profile

$ρ (s ̃) = \frac{ρ_{0}}{s ̃^{2} + 0.1 s ̃ + 1.0}$ where $ρ_{0} = 100$ represents the signal strength at the radar source (see the next code and chart).

## Signal strength at the source
_ρ₀ = 100.0

## Signal strength profile
## this function is hidden from the agent
## returns a 1-element vector
ρ(s̃) = [_ρ₀/(s̃[1]^2 + 0.1*s̃[1] + 1.0)]

ρ (generic function with 1 method)

_s̃̂ = 0.0:0.01:8.0 ## distance range
_ŷ = [ρ([s̃ₖ])[1] for s̃ₖ in _s̃̂] ## observation range
_s̃₀ = [0.7] ## initial position
_x₊ = [5.0] ## target/goal signal strength
plot(
    _s̃̂, _ŷ, color="black", 
    xlabel="Radial position [miles]", 
    ylabel="Signal strength [%]", 
    label="Signal strength (100% at radar source)") #.
scatter!(_s̃₀, [ρ(_s̃₀)], label="Initial position signal strength")

4.3.1 State variables

The state at time $t$ of the system-under-steer (sustr), also referred to as the environment (envir), or the generative process (genpr) will be given by $s ̃_{t}$ which will be the radial position of the unit relative to the radar souce.

4.3.2 Decision variables

Decisions are in the form of agent-prescribed velocity actions on a unit. An action suggests the adjustment of the radial velocity to either move away from the radar source or to move towards it. This velocity is limited to the interval $(- V^{m a x}, V^{m a x})$ . It is given by:

$\begin{aligned} V^{a} & = V^{m a x} \cdot \tanh (a) \\ = 0.05 \cdot \tanh (a) \end{aligned}$ where $a_{t}$ is the velocity action on the unit at time $t$ .

Since the mobile capabilities of units have limits, we use the $\tanh (\cdot)$ function to limit the velocity action to the interval $(- V^{m a x}, V^{m a x})$ .

Let us visualize how the velocity action on a unit, $V^{a}$ , varies with the amount of action $a$ .

_Vᵐᵃˣ = 0.5
Vᵃ = (a::Real) -> _Vᵐᵃˣ*tanh(a)

#24 (generic function with 1 method)

_a_x = range(-10, 10, length=500)
_Va_y = [ _Vᵐᵃˣ*tanh(xs) for xs in _a_x ]
plot(
    _a_x, _Va_y, title="Limits on velocity actions", 
    label="Landscape", color="black", 
    xlabel="Action, a", ylabel="Action velocity (Va)", legend=nothing,
    ylimits=(-0.5, 0.5)
)

4.3.3 Exogenous information variables

There are no exogenous information variables for this problem.

4.3.4 Transition and Observation functions

The transition function captures the dynamics of the environment/system-under-steer/generative process:

$\begin{aligned} s ̃_{t} & = s ̃_{t - 1} + V ᵃ_{t} \\ = s ̃_{t - 1} + V ᵐ ᵃ ˣ \tanh (a_{t}) \end{aligned}$

The environment generates outcomes as noisy observations of the current state with an observation noise variance

$ϑ = 10^{- 4}$

The observation function can be represented by: $y_{t} \sim N (ρ (s ̃_{t}), ϑ)$

_γ = 1e4 ## transition precision (system noise)
_ϑ = 1e-4 ## observation variance (observation noise)

0.0001

4.3.5 Objective function

The objective function is such that the Bethe free energy is minimized. This aspect will be handled by the RxInfer Julia package.

4.3.6 Implementation of the System-Under-Steer / Environment / Generative Process

The agent and the environment interact through a Markov blanket. Because states of the agent are unknown to the world, we wrap them in a comprehension that only returns functions for interacting with the agent. Internal beliefs cannot be directly observed, and interaction is only allowed through the Markov blanket of the agent (i.e. the sensors and actuators).

function create_envir(; ρ, s̃₀)
    s̃ₜ₋₁ = s̃₀
    s̃ₜ = s̃ₜ₋₁
    yₜ = ρ(s̃ₜ)[1] + sqrt(_ϑ)*randn() ## noisy signal strength
    execute = (aₜ::Float64) -> begin
        s̃ₜ = s̃ₜ₋₁ + [Vᵃ(aₜ)] ## next state
        yₜ = ρ(s̃ₜ)[1] + sqrt(_ϑ)*randn() ## noisy signal strength
        s̃ₜ₋₁ = s̃ₜ ## keep state
    end

    observe = () -> begin 
        return [yₜ]
    end

    return (execute, observe)
end

create_envir (generic function with 1 method)

4.4 Uncertainty Model

The only uncertainty we have for the environment is the noise associated with an observation as captured in the observation function above: $y_{t} \sim N (ρ (s ̃_{t}), ϑ)$

4.5 Agent / Generative Model

4.5.1 State variables

According to the agent the state of the system-under-steer/environment/generative process will be $s_{t}$ , rather than $s ̃_{t}$ , which is the distance of the agent from the signal source.

4.5.2 Decision variables

According to the agent the action on the environment at time $t$ will be represented by $u_{t}$ , also known as the control state of the agent.

4.5.3 Implementation of the Agent / Generative Model / Internal Model

We start by specifying a probabilistic model for the agent that describes the agent’s internal beliefs over the external dynamics of the environment.

The generative model is defined as follows:

$\begin{array}{r} p^{'} (x_{:}, s_{:}, u_{:}) = \underset{\begin{array}{c} Initial \\ state \\ prior \end{array}}{\underset{⏟}{p (s_{t - 1})}} \prod_{k = t}^{t + T} \underset{\begin{array}{c} Observation \\ model \end{array}}{\underset{⏟}{p (x_{k} ∣ s_{k})}} \underset{\begin{array}{c} Transition \\ model \end{array}}{\underset{⏟}{p (s_{k} ∣ s_{k - 1}, u_{k}}}) \underset{\begin{array}{c} Control \\ prior \end{array}}{\underset{⏟}{p (u_{k})}} (2019_VanDeLaar,_SAIPbMP eq 9) \end{array}$

The generative model includes future time steps.

To infer goal-driven (i.e. purposeful) behavior, we add prior beliefs $p^{+} (x)$ about desired future observations. This leads to an extended agent model:

$\begin{aligned} p (x_{:}, s_{:}, u_{:}) & = \frac{p^{'} (x, s, u) p^{+} (x)}{\int_{x} p^{'} (x, s, u) p^{+} (x) d x} \\ \propto \underset{original generative model}{\underset{⏟}{p (s_{t - 1}) \prod_{k = t}^{t + T} p (x_{k} ∣ s_{k}) p (s_{k} ∣ s_{k - 1}, u_{k}) p (u_{k})}} \underset{\begin{array}{c} extension \\ Goal \\ prior \end{array}}{\underset{⏟}{p^{+} (x_{k})}} \\ \propto \underset{\begin{array}{c} Initial \\ state \\ prior \end{array}}{\underset{⏟}{p (s_{t - 1})}} \prod_{k = t}^{t + T} \underset{\begin{array}{c} Observation \\ model \end{array}}{\underset{⏟}{p (x_{k} ∣ s_{k})}} \underset{\begin{array}{c} Transition \\ model \end{array}}{\underset{⏟}{p (s_{k} ∣ s_{k - 1}, u_{k}}}) \underset{\begin{array}{c} Control \\ prior \end{array}}{\underset{⏟}{p (u_{k})}} \underset{\begin{array}{c} Goal \\ prior \end{array}}{\underset{⏟}{p^{+} (x_{k})}} \end{aligned}$

Even though the states and observations are implemented as 1-element vectors, we will represent them in the mathematical expressions as scalar random variables. The controls are implemented as scalar random random variables. The previous expression can consequently be modified to become:

The factors are defined as:

observation:

$\begin{aligned} p (x_{k} ∣ s_{k}) & = N (x_{k} ∣ - s_{k}, ϑ) \\ = N (x_{k} ∣ - s_{k}, 10^{- 2}) \end{aligned}$ where $x_{k}$ denotes observations of the agent after interacting with the environment. Note that we hampered the observation model. Instead of the actual signal strength profile, we specify that the observed signal strength decreases with position linearly.

state transition:

$\begin{aligned} p (s_{k} ∣ s_{k - 1}, u_{k}) & = N (s_{k} ∣ s_{k - 1} + u_{k}, ϑ) \\ p (s_{t - 1}) & = N (s_{t - 1} ∣ m_{t - 1}, V_{t - 1}) \end{aligned}$

The current state is a linear function of the previous state and action. We have endowed the agent with an accurate model of the system dynamics.

control:

$\begin{aligned} p (u_{t}) & = \prod_{k = t}^{t + T} N (u_{k} ∣ 0, Ξ) \\ = \prod_{k = t}^{t + T} N (u_{k} ∣ 0, ξ \cdot I) \\ = \prod_{k = t}^{t + T} N (u_{k} ∣ 0, 0.5) \\ = N (u_{t} ∣ m_{u}, V_{u}) \end{aligned}$

This represents the control priors.

goal/target/preference:

$\begin{aligned} p^{+} (x_{t}) & = \prod_{k = t}^{t + (T - 1)} N (x_{k} ∣ 0, σ^{h u g e}) \cdot N (x_{T} ∣ x_{+}, Σ) \\ = \prod_{k = t}^{t + (T - 1)} N (x_{k} ∣ 0, σ^{h u g e}) \cdot N (x_{T} ∣ x_{+}, σ \cdot I) \\ = \prod_{k = t}^{t + (T - 1)} N (x_{k} ∣ 0, 10^{12}) \cdot N (x_{T} ∣ 5.0, 10^{- 4} \cdot I) \\ = N (x_{t} ∣ m_{x}, V_{x}) \end{aligned}$

This represents the target/goal/preference priors and encodes a belief about a preferred signal strength $x ₊ = 5.0$ .

initial state:

Setting $t = 1$ , $p (s_{0}) = N (s_{0} ∣ 0, 10^{- 12})$

This means we set a tight prior for the initial state.

4.5.3.1 Generative Model for the Signal Strength

The code in the next block defines the agent’s internal beliefs over the external dynamics and its probabilistic model of the environment, which correspond accurately by directly using the functions defined above. We use the @model macro from RxInfer to define the probabilistic model and the meta block to define approximation methods for the nonlinear state-transition functions.

In the model specification we include the beliefs over its future states (up to T steps ahead), in addition to the current state of the agent.

@model function signalstrength_model(mᵤ, Vᵤ, mₓ, Vₓ, mₛ₍ₜ₋₁₎, Vₛ₍ₜ₋₁₎, T, ρ)
    g = (sₜ₋₁::AbstractVector) -> begin
        sₜ = similar(sₜ₋₁) ## Next state
        sₜ = ρ(sₜ₋₁)
        return sₜ
    end
    
    Γ = _γ*diageye(1) ## Transition precision
    𝚯 = _ϑ*diageye(1)  ## Observation variance
    
    ## mₛ₍ₜ₋₁₎ = datavar(Vector{Float64})
    ## Vₛ₍ₜ₋₁₎ = datavar(Matrix{Float64})

    sₜ₋₁ ~ MvNormal(mean=mₛ₍ₜ₋₁₎, cov=Vₛ₍ₜ₋₁₎)
    sₖ₋₁ = sₜ₋₁
    
    ## mᵤ = datavar(Vector{Float64}, T)
    ## Vᵤ = datavar(Matrix{Float64}, T)
    
    ## mₓ = datavar(Vector{Float64}, T)
    ## Vₓ = datavar(Matrix{Float64}, T)
    
    ## u = randomvar(T)
    ## s = randomvar(T)
    ## x = randomvar(T)
    
    ## hIuI = randomvar(T)
    
    local s
    
    for k in 1:T
        ## Control
        u[k]    ~ MvNormal(mean=mᵤ[k], cov=Vᵤ[k])
        hIuI[k] ~ MvNormal(mean=sₖ₋₁ + u[k], precision=Γ)
        
        ## State transition
        s[k] ~ g(hIuI[k]) where { meta=DeltaMeta(method=Unscented(alpha=1.9)) }
        
        ## Likelihood of future observations
        x[k] ~ MvNormal(mean=s[k], cov=𝚯)
        
        ## Target/Goal prior
        x[k] ~ MvNormal(mean=mₓ[k], cov=Vₓ[k])
        
        sₖ₋₁ = s[k]
    end
    return (s, )
end

Next, we define the agent:

function create_agent(; T=20, ρ, x₊, s₀, ξ=0.5, σ=1e-4) ## ρ is a function
    ## Set control priors
    Ξ  = fill(ξ, 1, 1) ##Control prior variance
    mᵤ = Vector{Float64}[ [0.0] for k=1:T ] ##Set control priors
    Vᵤ = Matrix{Float64}[ Ξ for k=1:T ]

    ## Set target/goal priors
    Σ       = σ*diageye(1) ##Target/Goal prior variance
    mₓ      = [zeros(1) for k=1:T]
    mₓ[end] = x₊ ##Set prior mean to reach goal at t=T
    Vₓ      = [huge*diageye(1) for k=1:T]
    Vₓ[end] = Σ ##Set prior variance to reach goal at t=T

    ## Set initial brain state prior
    mₛ₍ₜ₋₁₎ = s₀
    Vₛ₍ₜ₋₁₎ = tiny*diageye(1)
    
    ## Set current inference results
    result = nothing

    ## Bayesian inference by message passing
    ## The `infer` function is the heart of the agent
    compute = (υₜ::Float64, ŷₜ::Vector{Float64}) -> begin
        mᵤ[1] = [υₜ] ## Register action with the generative model
        Vᵤ[1] = fill(tiny, 1, 1) ## Clamp control prior to performed action

        mₓ[1] = ŷₜ ## Register observation with the generative model
        Vₓ[1] = tiny*diageye(1) ## Clamp target/goal prior to observation

        result = infer(
            model=signalstrength_model(T=T, ρ=ρ),
            data=Dict(
                :mᵤ     => mᵤ, 
                :Vᵤ     => Vᵤ, 
                :mₓ     => mₓ, 
                :Vₓ     => Vₓ,
                :mₛ₍ₜ₋₁₎ => mₛ₍ₜ₋₁₎,
                :Vₛ₍ₜ₋₁₎ => Vₛ₍ₜ₋₁₎))
    end
    
    ## The `act` function returns the inferred best possible action
    act = () -> begin
        if result !== nothing
            return mode(result.posteriors[:u][2])[1]
        else
            return 0.0 ## Without inference result we return some 'random' action
        end
    end
    
    ## The `future` function returns the inferred future states
    future = () -> begin 
        if result !== nothing 
            return getindex.(mode.(result.posteriors[:s]), 1)
        else
            return zeros(T)
        end
    end
    
    ## The `slide` function modifies the `(mₛ₍ₜ₋₁₎, Vₛ₍ₜ₋₁₎` for the next step
    ## and shifts (or slides) the array of future goals `(mₓ, Vₓ)` 
    ## and inferred actions `(mᵤ, Vᵤ)`
    slide = () -> begin
        model  = RxInfer.getmodel(result.model)
        (s, )  = RxInfer.getreturnval(model)
        varref = RxInfer.getvarref(model, s) 
        var    = RxInfer.getvariable(varref)
        
        slide_msg_idx = 3 ##This index is model dependent
        (mₛ₍ₜ₋₁₎, Vₛ₍ₜ₋₁₎) = mean_cov(getrecent(messageout(var[2], slide_msg_idx)))

        mᵤ = circshift(mᵤ, -1)
        mᵤ[end] = [0.0]
        Vᵤ = circshift(Vᵤ, -1)
        Vᵤ[end] = Ξ

        mₓ = circshift(mₓ, -1)
        mₓ[end] = x₊
        Vₓ = circshift(Vₓ, -1)
        Vₓ[end] = Σ
    end

    return (act, future,   compute, slide)
end

create_agent (generic function with 1 method)

4.6 Agent Policy Evaluation

4.6.1 Training/Tuning

4.6.1.1 Naive approach

In this simulation we are going to perform a naive action policy. In this case, with limited velocity actions, the agent should not be able to achieve its target/goal:

_Nⁿᵃⁱᵛᵉ  = 100 ## Total simulation time
_πⁿᵃⁱᵛᵉ = 0.5*ones(_Nⁿᵃⁱᵛᵉ) ## Naive policy with full-positive action only
(execute_naive, observe_naive) = create_envir(;
    ρ=ρ,
    s̃₀=_s̃₀
);
_yⁿᵃⁱᵛᵉ = Vector{Vector{Float64}}(undef, _Nⁿᵃⁱᵛᵉ)
for t = 1:_Nⁿᵃⁱᵛᵉ
    execute_naive(_πⁿᵃⁱᵛᵉ[t]) ## Execute environmental process
    _yⁿᵃⁱᵛᵉ[t] = observe_naive() ## Observe external states
end

_yⁿᵃⁱᵛᵉ

100-element Vector{Vector{Float64}}:
 [51.025631139609395]
 [40.53499161645107]
 [32.47511444630898]
 [26.31444483053452]
 [21.614293161258527]
 [17.985455673169426]
 [15.152938219149133]
 [12.898459860403587]
 [11.107287673389216]
 [9.642046099271342]
 ⋮
 [0.1992771324252401]
 [0.1878571389854612]
 [0.19284980893479814]
 [0.18698770085538735]
 [0.1917329935426001]
 [0.18519512893652412]
 [0.18016678485212392]
 [0.16196139777174545]
 [0.19465215909381703]

_pa=plot(
    map(x -> x[1], _πⁿᵃⁱᵛᵉ), label="actions", 
    xlabel="t", ylabel="velocity",
    ## ylimits=(-0.6, 0.6)
    )
_py=plot(
    map(x -> x[1], _yⁿᵃⁱᵛᵉ), label="observations", 
    color="red", xlabel="t", ylabel="temperature", 
    ## ylimits=(0, 60)
    )
_py=plot!(
    [0, _Nⁿᵃⁱᵛᵉ], [_x₊[1], _x₊[1]],
    label="target/goal", color="green", xlabel="t", ylabel="signal strength")
plot(_pa, _py, layout=@layout([ a; b ]))

4.6.1.1 Active inference approach

In the active inference approach we are going to create an agent that models the environment around itself as well as the best possible actions in a probabilistic manner. That should help the agent to understand that the brute-force approach is not the most efficient one and hopefully to realise that a little bit of swing is necessary to achieve its goal.

### Simulation parameters
## Total simulation time
_Nᵃⁱ = 100

## Lookahead time horizon
_Tᵃⁱ = 20

## Initial state (position)
_s₀ = [0.7]

## Control prior variance value
_ξ = 0.5

## Target prior variance value
_σ = 1e-4

## Target/Goal signal strength
_x₊ = [5.0]

1-element Vector{Float64}:
 5.0

(execute_ai, observe_ai) = create_envir(
    ρ=ρ,
    s̃₀=_s₀
)
(act_ai, future_ai,   compute_ai, slide_ai) = create_agent(;
    T =_Tᵃⁱ,
    ρ=ρ,
    x₊=_x₊,
    s₀ = _s₀,
    ξ = _ξ,
    σ = _σ
) 

## Step through experimental protocol
_as = Vector{Float64}(undef, _Nᵃⁱ)         ## Actions
_fs = Vector{Vector{Float64}}(undef, _Nᵃⁱ) ## Predicted future
_ys = Vector{Vector{Float64}}(undef, _Nᵃⁱ) ## Observations
for t = 1:_Nᵃⁱ
    ## 1. Act-Execute-Observe: #.execute() & observe() from create_world()    
    _as[t] = act_ai()            ## Invoke an action from the agent
    _fs[t] = future_ai()         ## Fetch the predicted future states
             execute_ai(_as[t]) ## The action influences hidden external states
    _ys[t] = observe_ai()        ## Observe the current environmental outcome (update p)
    ## 2. Infer:    
             compute_ai(_as[t], _ys[t]) ## Infer beliefs from current model state (update q)
    ## 3. Slide:
             slide_ai() ## Prepare for next iteration
end

_as

100-element Vector{Float64}:
   0.0
 -21.36693936153308
   3.0844941227780266e-10
  -1.1918746306556778e-14
  -1.722071340263603e-14
  -3.0138542363982357e-15
 -31.452425900704522
   1.6074350769364204e-10
  -8.3420908010661e-14
 -31.446772692202252
   ⋮
   0.025881850832354857
   0.007653542075123006
  -0.0063834975023143664
  -0.012522202864476677
  -0.00014329476910728152
  -0.007284710352820595
   0.010620506709198467
   0.01181222275839934
  -0.004284750132064466

_fs

100-element Vector{Vector{Float64}}:
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
 [64.1050914724714, 0.024295900016316145, 81.9952229102532, 0.02326461014243788, 81.99806039627605, 0.023282594724625006, 81.99785776754386, 0.023282703209851985, 81.9978566504563, 0.023282703575680037, 81.99785662813623, 0.023282703596969337, 81.997844434632, 0.023282717002393875, 81.98991973518139, 0.023291429347476626, 76.83956349025844, 0.028953685838688913, -3270.4380985931484, 3.7089213946170507]
 [0.26021807567801597, 76.51925797232008, 0.024588284561821157, 81.98602856756122, 0.023282400892067664, 81.9978657140906, 0.023282686253346622, 81.9978568364462, 0.023282703487297916, 81.99785662903548, 0.02328270359669967, 81.99784443413607, 0.023282717002941582, 81.98991941118395, 0.023291429703676588, 76.83935292046723, 0.028953917337270783, -3270.5749504000423, 3.7090718482934713, 5.000049900596623]
 [78.45150761662177, 0.016228540418750714, 82.07048733038336, 0.023250706181025784, 81.99817991788056, 0.023282612509532636, 81.99785751105236, 0.02328270347076279, 81.99785662889671, 0.023282703597490663, 81.99784443412724, 0.023282717002946058, 81.9899194111522, 0.023291429703711494, 76.83935289984423, 0.028953917359943536, -3270.5749638031775, 3.7090718630287607, 4.999822723877539, 4.999861213496321]
 [94.37013230877719, 0.011217479921043444, 82.11296745966081, 0.023242904871372275, 81.99824688694304, 0.023282622714842653, 81.99785736469268, 0.023282703618305377, 81.99785664567494, 0.023282703577973612, 81.99785636641897, 0.02328270388468357, 81.99767435356523, 0.023282903988027164, 81.87938203095233, 0.023412953530179893, 4.999797348102333, 4.999634221244921, 4.999634234374716, 4.999861288251819]
 [94.38212854399477, 0.011214629903962969, 82.11299068542773, 0.02324290061511117, 81.99824692343306, 0.023282622720486805, 81.99785734582152, 0.02328270363904456, 81.99784443331787, 0.023282717004131294, 81.98991941113931, 0.023291429703717344, 76.83935289984885, 0.028953917359938484, -3270.574963800136, 3.7090718630254185, 4.999822775404278, 4.999634211165357, 4.999634234378711, 4.999861288251818]
 [94.36356818729371, 0.01121755478301197, 82.11296684957067, 0.0232429049831749, 81.99824688598311, 0.02328262271469594, 81.99785736426385, 0.023282703618777004, 81.99785636561724, 0.023282703885866615, 81.99767435355238, 0.02328290398803298, 81.87938203095224, 0.023412953530179935, 4.999797348102336, 4.99963422123323, 4.999634285889269, 4.999634285875338, 4.999634234349101, 4.999861288251829]
 [0.10530350361415308, 80.78624610612391, 0.02350207822557235, 81.99599083459704, 0.023282358552201603, 81.99786140435666, 0.023282699111549565, 81.99785641412981, 0.023282703864909528, 81.99767435376368, 0.023282903987974658, 81.87938203095284, 0.02341295353017984, 4.999797348102336, 4.99963422123323, 4.999634285889271, 4.999634285863645, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [79.79624682114766, 0.01568647127138887, 82.07524084668127, 0.023249831565799503, 81.9981874287653, 0.02328261364601782, 81.99785721426943, 0.023282703795600265, 81.99767435415534, 0.023282903988623177, 81.87938203094538, 0.023412953530184206, 4.999797348102336, 4.99963422123323, 4.999634285889271, 4.999634285863648, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [94.34660743114513, 0.01122158515499835, 82.1129339840157, 0.023242911023192224, 81.99823462140365, 0.023282636133500886, 81.98992012865332, 0.02329142974608166, 76.83935289842654, 0.028953917362276142, -3270.5749637992258, 3.7090718630257395, 4.999822775404281, 4.9996342111536665, 4.999634285893268, 4.999634285863647, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 ⋮
 [4.447675712031228, 4.999777316900034, 4.999634229173657, 4.999634285886126, 4.99963428586365, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.468165388678463, 4.999774221360361, 4.999634230400698, 4.999634285885639, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.483732667326656, 4.999771777632537, 4.999634231369364, 4.999634285885255, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.490484433347434, 4.999770692642443, 4.999634231799438, 4.999634285885085, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.476834592210739, 4.999772870411554, 4.9996342309361985, 4.999634285885427, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.484726027284328, 4.9997716189665375, 4.999634231432256, 4.9996342858852305, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.464851881406653, 4.999774731221687, 4.999634230198593, 4.999634285885721, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.463518510739242, 4.999774935357221, 4.999634230117679, 4.99963428588575, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]
 [4.481416551896328, 4.999772146306981, 4.999634231223226, 4.999634285885313, 4.9996342858636496, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.9996342858636575, 4.999634285863655, 4.999634285875348, 4.999634234349101, 4.999861288251829]

_ys

100-element Vector{Vector{Float64}}:
 [64.10936504759323]
 [94.32155860152815]
 [94.35950835426148]
 [94.33926086982841]
 [94.34579432967536]
 [94.33339793643168]
 [94.35788964496521]
 [94.34506074900816]
 [94.3213489202805]
 [64.0823668187312]
 ⋮
 [4.589496750260439]
 [4.600911678908157]
 [4.603673994810185]
 [4.585660072382549]
 [4.5992361929589345]
 [4.573617509466697]
 [4.578482534763504]
 [4.599738809785967]
 [4.59197989520212]

_pa = plot(
    0.5*tanh.(_as), label="actions", xlabel="t", ylabel="velocity", 
    ## ylimits=(-0.6, 0.6)
    )
_py = plot(
    map(x -> x[1], _ys), label="observations",
    color="red", xlabel="t", ylabel="signal strength", 
    ## ylimits=(0, 60)
    )
_py = plot!(
    [0, _Nᵃⁱ], [_x₊[1], _x₊[1]], 
    label="target/goal", color="green", xlabel="t", ylabel="signal strength")
plot(_pa, _py, layout=@layout([ a; b ]))

_s₀, ρ(_s₀), _x₊

([0.7], [64.1025641025641], [5.0])

In this scenario a military unit suddenly finds itself $0.7$ miles away from a radar source. Because $s_{0} = [0.7]$ , the signal strength starts at $ρ ([0.7]) = 64.1$ . However, the target signal strength is $x_{+} = [5.0]$ . The agent then issues actions in the form of velocity adjustments until the target signal strength is reached.