Control a Roomba using Active Inference and RxInfer
Kobus Esterhuysen
May 26, 2024
October 22, 2026
This is the fifth part of an effort to add control to the Hidden Markov Model RxInfer example. Eventually this work will be used in a more elaborate scheduling project making use of active inference. It follows Parts 1–4 of this series. Terms introduced in those parts are listed there and are not repeated here.
The names of model variables are written in italic lower case, such as name, and their values in double quotes, such as “value”. Section 4.1 describes these conventions, and Appendix A gives their motivation.
In the previous parts, the agent first watched a Roomba move through a 3-room apartment, a bedroom, a living room and a bathroom, inferring which room it was in (Parts 1 and 2) and whether that room was occupied (Part 3). In Part 4, the agent took control, with a single goal: to avoid disturbing people. It succeeded, with about a third less disturbance than a Roomba moving at random, but it did so by largely avoiding the living room, the room most often occupied. A Roomba that does not clean the busiest room is not much of a cleaning robot.
In this part, the agent gets a second goal, cleaning. It has to balance two goals that can conflict: cleaning where it is needed, and not disturbing anyone. The sharpest case is the living room. It gets dirty fastest, and it is also the room most often in use. The agent should clean it when it is quiet, rather than avoiding it altogether.
Several state factors. The model of the environment grows from one state factor to five. Each factor is one aspect of the world:
In Parts 3 and 4, the room and the occupancy were combined into a single state factor, station. Splitting the state into factors is the standard way of building larger active inference models, and gaining experience with it is a main purpose of this part. Each factor’s transitions depend only on a few factors. The dirt in a room depends on itself and on the whereabouts. Each sensor depends only on a few factors. The motion sensor depends on the whereabouts and the occupant. These dependencies are the core of multi-factor modeling.
The Roomba has three sensors, and three observation modalities:
The light sensor from the previous parts added little and is dropped.
Two preferences. As before, goals are preferences over observations. The agent prefers to observe “no motion”, as in Part 4, and now also prefers to observe “dirt picked up”, which draws it toward dirty rooms. A preference for a clean floor would backfire: the agent would prefer to sit in rooms that are already clean. How strongly the agent holds each preference sets the balance between cleaning and avoiding people.
Planning. As in Part 4, the agent looks ahead, scores each candidate sequence of moves by its expected free energy, and carries out the first move of the best sequences. Getting from the bedroom to the living room takes two moves, and cleaning takes at least one more, so the agent looks three steps ahead.
To keep the focus on multi-factor modeling and on balancing two goals, the agent is given its model of the world, rather than learning it. Internally it combines the five factors into their joint states, \(3 \cdot 4 \cdot 2 \cdot 2 \cdot 2 = 96\) in total, so that perception and planning stay exact. A later part can replace this with a factorized approximation, which is what makes much larger models feasible.
Terms added in this part.
Assumptions. The transition probabilities and the sensor error rates in 4.3.4 are assumed, not measured from a Roomba. Appendix B records why those numbers were chosen. When measured numbers arrive, 4.3.4 and Appendix B are the places to change.
Structure of the code. This notebook uses the series protocol. One tick is one step of the Roomba. The cycle is execute, observe, perceive, plan, act, slide.
create_envir(...; seed) returns execute, observe and state. The environment draws its randomness from MersenneTwister(seed).
execute(Lûₜ₋₁) carries out the chosen heading for one tick: it updates the whereabouts, the occupant moves, and bed dust, lounge grime and bath soil accumulate or are cleaned.observe() returns the observation tuple \(\hat{\mathbf{o}}^{(t)}\), the modalities surface, commotion and pickup.state() returns the true state tuple \(\hat{\mathbf{s}}^{\ast(t)}\), for evaluation only. The agent never receives it.create_agent(; LC) returns act, future, compute and slide.
compute(Lôₜ) perceives and plans: it updates the joint belief \(\boldsymbol{q}^{(t)}\) from the observation, then scores the policies by \(\boldsymbol{G}_{\Sigma}\).act() returns the chosen heading, the most probable first action.future(p) returns the predicted marginal beliefs under policy \(p\), for inspection.slide() moves one tick forward: the posterior becomes the next prior.As in the previous parts, and as in the active inference literature, \(\mathbf{A}\) holds the observation tensors and \(\mathbf{B}\) the state transition tensors, swapped relative to the original RxInfer example, which follows the engineering convention of using \(A\) for state transitions. With several factors, these arrays are tensors, with one axis for each factor they depend on.
Julia Version 1.12.7
Commit 6d172b025e4 (2026-08-15 08:05 UTC)
Build Info:
Official https://julialang.org release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 12 × Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
WORD_SIZE: 64
LLVM: libLLVM-18.1.7 (ORCJIT, skylake)
GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 12 virtual cores)import Pkg
Pkg.add(Pkg.PackageSpec(;name="RxInfer"))
Pkg.add(Pkg.PackageSpec(;name="Random"))
Pkg.add(Pkg.PackageSpec(;name="Plots"))
Pkg.add(Pkg.PackageSpec(;name="XLSX"))
Pkg.add(Pkg.PackageSpec(;name="DataFrames"))
Pkg.add(Pkg.PackageSpec(;name="BenchmarkTools"))
Pkg.add(Pkg.PackageSpec(;name="Distributions"))
Pkg.add(Pkg.PackageSpec(;name="OffsetArrays"))
using RxInfer, Random, BenchmarkTools, Distributions, Plots, XLSX, DataFrames, OffsetArrays, Statistics Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`
Project No packages added to or removed from `~/.julia/environments/v1.12/Project.toml`
Manifest No packages added to or removed from `~/.julia/environments/v1.12/Manifest.toml`Status `~/.julia/environments/v1.12/Project.toml`
[6e4b80f9] BenchmarkTools v1.8.0
[a93c6f00] DataFrames v1.8.2
[31c24e10] Distributions v0.25.132
[682c06a0] JSON v1.11.0
[6fe1bfb0] OffsetArrays v1.17.0
[91a5bcdd] Plots v1.41.7
[86711068] RxInfer v6.1.0
[10745b16] Statistics v1.11.5
[fdbf4ff8] XLSX v0.13.2
[9a3f8284] Random v1.11.0The data is simulated. One tick is one step of the Roomba. A run covers \(T + 1 = 600\) ticks, \(t = 0, \ldots, T\) with \(T = 599\). At each tick the agent observes, chooses an action, and the environment responds, so the path depends on the agent’s choices. The same records would, in a real apartment, come from the Roomba’s camera, motion sensor and dirt sensor, and from its commanded heading. The true states would not be in that log.
Three kinds of one-hot encoded vectors are recorded. In each vector, the component that is 1 marks the value that occurred.
The state tuple has entries of different lengths, 3, 4 and 2. That is why the states form a tuple rather than one vector.
The agent works with the observations only, and with the actions it chose itself. The true states are kept for evaluation: they show whether the marginal beliefs match the factors, how often the whereabouts and the occupant are the same room, and how often each room is dirty.
The simulation is used directly. There is no cleaning or reshaping before inference. Data arrive tick by tick: at tick \(t\) the agent receives only \(\hat{\mathbf{o}}^{(t)}\), updates its beliefs, and chooses the next action.
Trajectories are indexed from 0, so index \(t\) is time \(t\). A run records four of them, each re-indexed with offset0:
Lôː, the observations \(\hat{\mathbf{o}}^{(0:T)}\), for \(t = 0, \ldots, T\),Lûː, the actions \(\hat{\mathbf{u}}^{(0:T-1)}\), for \(t = 0, \ldots, T-1\),Lsː, the marginal beliefs \(\boldsymbol{s}^{(t,n)}\), for \(t = 0, \ldots, T\), one vector per factor, andLŝˣː, the true states \(\hat{\mathbf{s}}^{\ast(0:T)}\), for evaluation only.The agent itself filters a joint belief \(\boldsymbol{q}^{(t)}\) over the 96 combinations of the five factors, so the filtering is exact. Lsː stores the five marginals of that joint, not the joint. The marginal \(\boldsymbol{s}^{(t,n)}\) is the joint added up over every factor other than \(n\). The marginals have the same shape as the true state tuple, so each factor can be compared on its own. They drop the relations between factors. The joint can say that the whereabouts and the occupant are probably the same room. The two marginals cannot.
Index constants, defined in the next cell: whereabouts_n, occupant_n, bed_dust_n, lounge_grime_n, bath_soil_n for the factors, heading_n for the control factor, and surface_m, commotion_m, pickup_m for the modalities. Label vectors in that cell hold the values, in the same order as the indices. The same cell defines tuple0 and offset0, the plot helpers step_x and step_y, and the default font sizes.
This part does not call RxInfer. The agent’s computations are written in Julia, so nothing is converted at an RxInfer boundary.
## tuple0(x...): a 0-based tuple (x^{(0)}, ..., x^{(N)}). Each argument is one entry.
## offset0(v): re-index a vector so that index t is time t. It does not wrap v.
tuple0(x...) = OffsetVector(collect(x), 0:length(x)-1)
offset0(v) = OffsetVector(v, 0:length(v)-1)
## step_x(n): n + 1 edges, from -0.5 in steps of 1. Tick t spans t - 0.5 to t + 0.5.
## step_y(v): repeat the last value so the step covers the last tile.
step_x(n) = range(-0.5, step=1.0, length=n + 1)
step_y(v) = vcat(collect(v), v[end])
default(titlefontsize=13, guidefontsize=12, legendfontsize=10, tickfontsize=9)
## ---------- index constants and label vectors ----------
whereabouts_n = 0
occupant_n = 1
bed_dust_n = 2
lounge_grime_n = 3
bath_soil_n = 4
heading_n = 0
surface_m = 0
commotion_m = 1
pickup_m = 2
whereabouts_labels = ["bedroom", "living room", "bathroom"]
occupant_labels = ["bedroom", "living room", "bathroom", "away"]
dust_labels = ["clean", "dirty"]
heading_labels = ["bedroom", "living room", "bathroom"]
surface_labels = ["hardwood", "carpet", "tiles"]
commotion_labels = ["no motion", "motion"]
pickup_labels = ["nothing picked up", "dirt picked up"]
## listing cells still use these names
room_labels = whereabouts_labels
person_labels = occupant_labels
dirt_labels = dust_labels
action_labels = heading_labels3-element Vector{String}:
"bedroom"
"living room"
"bathroom"Why tuple0() rather than OffsetVector() directly?
OffsetVector, it is easy to offset the wrong level: OffsetVector([1.0, 0.0, 0.0], 0:2) ## the one-hot vector itself, re-indexed 0:2 (wrong level)
OffsetVector([[1.0, 0.0, 0.0]], 0:0) ## a tuple containing the one-hot vector (intended)
tuple0([1.0, 0.0, 0.0]) ## the same as the second line, with no way to get it wrongThe first line is valid code but means something else: the one-hot vector $\hat{\boldsymbol{s}}^{\ast(0,0)}$ itself, now indexed from 0, rather than the tuple $\hat{\mathbf{s}}^{\ast(0)} = \big(\hat{\boldsymbol{s}}^{\ast(0,0)}\big)$. Nothing would complain, and later code that indexes `[0]` would get a single number instead of a vector. `tuple0` always wraps its arguments as *entries*, so this mistake cannot happen.The index range is computed automatically. With OffsetVector, the range is written by hand and has to agree with the number of entries: adding a second modality to LAˣ while forgetting to change 0:0 to 0:1 gives an error at best. tuple0 derives the range from the number of arguments, so adding or removing an entry is a one-line change.
It reads like the math. tuple0(A0, A1) looks like \(\big(\boldsymbol{A}^{(0)}, \boldsymbol{A}^{(1)}\big)\): entries separated by commas, nothing else. OffsetVector([A0, A1], 0:1) adds square brackets and a range that have nothing to do with the math.
It states the intent. OffsetVector says “an array with shifted indices”, which could be anything. tuple0 says “a tuple of factors or modalities, as in the math”, the specific concept from (10.33). And since all tuples are built the same way, a search for tuple0( finds every one of them.
There is one place to change things. Changing how tuples work, e.g. switching to an immutable 0-based type, or adding a check that every column of every \(\boldsymbol{A}\) and \(\boldsymbol{B}\) sums to 1, means changing one line instead of every place a tuple is built.
Entries of different shapes just work. collect builds a vector of whatever is passed in, so tuple0(A0, A1) with a \(3\times 3\) and a \(2\times 3\) matrix gives a vector of matrices, which is exactly what a tuple of likelihood matrices with different numbers of observations needs.
A symbol’s typeface shows what it is. Upright bold is a tuple, italic bold is an array, and a plain letter is a scalar or one random variable. Round brackets pick a member, time first, then the factor or the modality. Square brackets pick a position along an array. An entry, every index fixed, is plain. A sum over one or more dimensions is marked \(\Sigma\).
Text conventions.
Italics and bold are not used for emphasis. Appendix A gives the motivation.
Symbols.
| Math | Code | Meaning |
|---|---|---|
| \(s^{(t,0)}\), \(\boldsymbol{s}^{(t,0)}\), \(\hat{s}^{\ast(t,0)}\), \(\hat{\boldsymbol{s}}^{\ast(t,0)}\) | Lsₜ[whereabouts_n], Lŝˣₜ[whereabouts_n] |
whereabouts: “bedroom”, “living room”, “bathroom” |
| \(s^{(t,1)}\), \(\boldsymbol{s}^{(t,1)}\), \(\hat{s}^{\ast(t,1)}\) | Lsₜ[occupant_n], Lŝˣₜ[occupant_n] |
occupant: “bedroom”, “living room”, “bathroom”, “away” |
| \(s^{(t,2)}\), \(\boldsymbol{s}^{(t,2)}\), \(\hat{s}^{\ast(t,2)}\) | Lsₜ[bed_dust_n] |
bed dust, dirt in the bedroom: “clean”, “dirty” |
| \(s^{(t,3)}\), \(\boldsymbol{s}^{(t,3)}\), \(\hat{s}^{\ast(t,3)}\) | Lsₜ[lounge_grime_n] |
lounge grime, dirt in the living room: “clean”, “dirty” |
| \(s^{(t,4)}\), \(\boldsymbol{s}^{(t,4)}\), \(\hat{s}^{\ast(t,4)}\) | Lsₜ[bath_soil_n] |
bath soil, dirt in the bathroom: “clean”, “dirty” |
| \(o^{(t,0)}\), \(\hat{o}^{(t,0)}\), \(\hat{\boldsymbol{o}}^{(t,0)}\) | Lôₜ[surface_m] |
surface: “hardwood”, “carpet”, “tiles” |
| \(o^{(t,1)}\), \(\hat{o}^{(t,1)}\), \(\hat{\boldsymbol{o}}^{(t,1)}\) | Lôₜ[commotion_m] |
commotion: “no motion”, “motion” |
| \(o^{(t,2)}\), \(\hat{o}^{(t,2)}\), \(\hat{\boldsymbol{o}}^{(t,2)}\) | Lôₜ[pickup_m] |
pickup: “nothing picked up”, “dirt picked up” |
| \(u^{(t,0)}\), \(\hat{u}^{(t,0)}\), \(\hat{\boldsymbol{u}}^{(t,0)}\) | Lûₜ[heading_n], heading_labels |
heading: “bedroom”, “living room”, “bathroom” |
| \(\hat{\mathbf{s}}^{\ast(t)}\), \(\hat{\mathbf{o}}^{(t)}\), \(\hat{\mathbf{u}}^{(t)}\) | Lŝˣₜ, Lôₜ, Lûₜ |
realized tuples at tick \(t\) |
| \(\hat{\mathbf{s}}^{\ast(0:T)}\), \(\hat{\mathbf{o}}^{(0:T)}\), \(\hat{\mathbf{u}}^{(0:T-1)}\) | Lŝˣː, Lôː, Lûː |
trajectories |
| \(\mathbf{s}^{(t)}=\big(\boldsymbol{s}^{(t,0)},\ldots,\boldsymbol{s}^{(t,4)}\big)\) | Lsₜ, Lsː |
tuple of marginal beliefs, and its trajectory |
| \(\boldsymbol{q}^{(t)}\) | _qₜ |
joint belief over the 96 combinations. A project symbol, so it does not clash with the marginals \(\boldsymbol{s}^{(t,n)}\) |
| \(\boldsymbol{A}^{\ast(m)}\), \(\mathbf{A}^{\ast}\) | _Aˣı0ı and so on, LAˣ |
environment likelihood tensors, and the tuple |
| \(\boldsymbol{B}^{\ast(n)}\), \(\mathbf{B}^{\ast}\) | _Bˣı0ı and so on, LBˣ |
environment transition tensors, and the tuple |
| \(\boldsymbol{D}^{\ast(n)}\), \(\mathbf{D}^{\ast}\) | Lŝˣ₀ |
environment initial state, here a one-hot tuple, not a distribution the agent is shown |
| \(\boldsymbol{A}^{(m)}\), \(\mathbf{A}\) | LA[m] |
agent’s likelihood tensors, equal to the environment’s |
| \(\boldsymbol{B}^{(n)}\), \(\mathbf{B}\) | LB[n] |
agent’s transition tensors, equal to the environment’s |
| \(\log\boldsymbol{C}^{(m)}\), \(\mathbf{C}\) | LC[m], LC |
log-preferences over modality \(m\). surface is \(\boldsymbol{0}\) |
| \(\boldsymbol{D}^{(n)}\), \(\mathbf{D}\) | LD[n], LD |
agent’s initial prior, uniform on each factor |
| \(\mathcal{D}_A^{(m)}\), \(\mathcal{D}_B^{(n)}\) | LdepsA[m], LdepsB[n] |
factors that modality \(m\), or the transition of factor \(n\), depends on |
| \(\boldsymbol{H}^{(m)}\) | LH[m] |
entropy of each column of \(\boldsymbol{A}^{(m)}\) |
| \(p_{\text{move}}\), \(p_{\text{clean}}\), \(p_{\text{dirty}}^{[r]}\) | p_move, p_clean, p_dirty |
move success, cleaning success, and the dirt rate of room \(r\) |
| \(H\), \(P\), \(\boldsymbol{V}\), \(\boldsymbol{V}^{[p]}\), \(V^{[p,\tau]}\) | H, P, _V |
horizon, number of policies, the decision matrix, policy \(p\), the action of policy \(p\) at step \(\tau\) |
| \(\boldsymbol{G}\), \(G^{[p,\tau]}\) | (accumulated, not stored) | per-step expected free energy, policies by steps |
| \(\boldsymbol{G}_{\Sigma}\), \(G_{\Sigma}^{[p]}\) | _GΣ |
sum of \(\boldsymbol{G}\) over the steps, one number per policy |
| \(\boldsymbol{E}\) | (none) | no habit prior. Policies are scored by \(\boldsymbol{G}_{\Sigma}\) only |
| \(\boldsymbol{\pi}\), \(\pi^{[p]}\) | _qπ |
policy posterior \(Q(\pi^{(t)})\), and the probability of policy \(p\) |
| \(Q(u^{(t,0)})\) | _qu |
posterior over the next heading |
| \(\boldsymbol{q}^{(\tau)}_{\pi^{[p]}}\) | inside efe |
predicted joint belief at step \(\tau\) under policy \(p\) |
| \(\boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}}\) | o inside efe |
predicted observation of modality \(m\) |
The code stores beliefs, not the random variables. Lsₜ holds \(\big(\boldsymbol{s}^{(t,0)}, \ldots, \boldsymbol{s}^{(t,4)}\big)\). Inside \(P\), \(Q\) and \(\mathcal{Cat}\), the same letter without bold is the random variable, \(\mathbf{s}^{(t)}=\big(s^{(t,0)},\ldots,s^{(t,4)}\big)\). A realized category is plain, \(\hat{s}^{\ast(t,n)}\) and \(\hat{o}^{(t,m)}\). Its one-hot encoding is bold, \(\hat{\boldsymbol{s}}^{\ast(t,n)}\) and \(\hat{\boldsymbol{o}}^{(t,m)}\), with \(\hat{o}^{[i]}=\delta_{i\hat{o}}\). The environment’s asterisk sits before the time index.
A named member uses ı (dotless i). _Bˣı0ı is \(\boldsymbol{B}^{\ast(0)}\). A position along an array uses I…I, with _ for a free index. Indexing a tuple, LBˣ[whereabouts_n], is the same array as the name _Bˣı0ı. Julia’s own [ ] is the indexing syntax. It is not the math’s bracket.
In Python, ˣ is normalized to x and ₜ to t, so LAˣ and LAx are the same name. This notebook is Julia. Do not reuse LAx for something else.
Dependencies. Each tensor depends on only some factors. \(\mathcal{D}_A^{(m)}\) is the set of factors modality \(m\) depends on. \(\mathcal{D}_B^{(n)}\) is the set of factors whose previous values the transition of factor \(n\) depends on, always including factor \(n\). The calligraphic \(\mathcal{D}\) is not the initial-state tuple \(\mathbf{D}\).
| tensor | depends on | shape | axes |
|---|---|---|---|
| \(\boldsymbol{A}^{\ast(0)}\), surface | \(\mathcal{D}_A^{(0)}=\{0\}\) | \(3\times 3\) | texture, whereabouts |
| \(\boldsymbol{A}^{\ast(1)}\), commotion | \(\mathcal{D}_A^{(1)}=\{0,1\}\) | \(2\times 3\times 4\) | stir, whereabouts, occupant |
| \(\boldsymbol{A}^{\ast(2)}\), pickup | \(\mathcal{D}_A^{(2)}=\{0,2,3,4\}\) | \(2\times 3\times 2\times 2\times 2\) | pickup, whereabouts, the three dirt factors |
| \(\boldsymbol{B}^{\ast(0)}\), whereabouts | \(\mathcal{D}_B^{(0)}=\{0\}\), and the heading | \(3\times 3\times 3\) | next whereabouts, previous whereabouts, heading |
| \(\boldsymbol{B}^{\ast(1)}\), occupant | \(\mathcal{D}_B^{(1)}=\{1\}\) | \(4\times 4\) | next occupant, previous occupant |
| \(\boldsymbol{B}^{\ast(n)}\), \(n=2,3,4\) | \(\mathcal{D}_B^{(n)}=\{n,0\}\) | \(2\times 2\times 3\) | next dirt, previous dirt, previous whereabouts |
For example, \(\boldsymbol{A}^{\ast(1)}[\bullet, r, p]\) is the distribution over commotion when the whereabouts is \(r\) and the occupant is \(p\). The dirt sensor depends on all three dirt factors because the whereabouts selects which room’s dirt it senses.
Joint belief. The agent’s joint belief \(\boldsymbol{q}^{(t)}\in[0,1]^{96}\) is one probability for each combination of the five factors, stored as an array of size \(3\times 4\times 2\times 2\times 2\). It is a project symbol: the chapters’ belief for these factors is the tuple of marginals \(\mathbf{s}^{(t)}=\big(\boldsymbol{s}^{(t,0)},\ldots,\boldsymbol{s}^{(t,4)}\big)\), and \(\boldsymbol{q}^{(t)}\) is the joint those marginals are taken from. The marginals are what Lsː records. The joint is what perception and planning use, because commotion and pickup each tie factors together.
What is new, compared with Part 4.
Five factors, with dependencies. Part 4 had one combined factor, station. This part has whereabouts, occupant, bed dust, lounge grime and bath soil. Only the whereabouts is controlled. The sets \(\mathcal{D}_A^{(m)}\) and \(\mathcal{D}_B^{(n)}\) say which factors each tensor depends on. In code they are LdepsA and LdepsB.
A second preference. \(\log\boldsymbol{C}^{(1)}\) prefers “no motion”, as in Part 4. \(\log\boldsymbol{C}^{(2)}\) prefers “dirt picked up”. surface is used for perception only, so \(\log\boldsymbol{C}^{(0)}=\boldsymbol{0}\). The light sensor is dropped. commotion is modality 1 and pickup is modality 2.
Exact joint filtering, marginals for evaluation. With 96 combinations, the agent filters \(\boldsymbol{q}^{(t)}\) exactly. The recorded beliefs are the marginals \(\boldsymbol{s}^{(t,n)}\).
A horizon of three. \(H=3\) gives \(P=3^3=27\) policies. From the bedroom, the living room is two moves away, and the cleaning shows up only after the Roomba is there.
The reasons, including alternatives that were not taken, are in Appendix B.
Changes in the symbol conventions compared to Part 4.
A_dependencies and B_dependencies in pymdp. The calligraphic \(\mathcal{D}\) for dependency sets is distinct from the upright \(\mathbf{D}\) for the initial-state tuple._sₜ and the tuple of marginals is Lsₜ.RxInfer.LdepsA and LdepsB, both preferences, and the joint and marginal beliefs. The rows that were specific to the light sensor and to the combined station factor are removed.Metrics. Three measures, judged with the true states. The formulas are in 4.3.5.
The baselines are strategies a person could follow without this model: head for the bedroom and stay there, cycle the three rooms on a fixed schedule, and choose the heading at random. The Part 4 agent, which prefers only “no motion”, is the fourth comparison. It shows what avoiding people does to cleanliness. A good agent of this part should keep the rooms, and the living room in particular, cleaner than that agent, and still disturb the person less than the random Roomba.
Decisions. At each tick the agent chooses the heading: “bedroom”, “living room” or “bathroom”. Heading for the current room means staying. It looks \(H=3\) steps ahead and scores each of the 27 policies by \(G_{\Sigma}^{[p]}\). The score favors “no motion” and “dirt picked up”, and it favors observations that reduce uncertainty about the state. The two preferences can pull apart. The living room gets dirty fastest, and it is also where the person spends the most time.
Sources of uncertainty.
In the environment: each factor has its own stochastic dynamics, written in \(\mathbf{B}^{\ast}\), and each sensor misreads, written in \(\mathbf{A}^{\ast}\).
The run starts from a fixed state, so the initial state is not a source of uncertainty in the environment: the whereabouts is “bedroom”, the occupant is “living room”, and all three dirt factors are “clean”. The action itself adds no uncertainty. The agent knows which heading it chose.
On the agent’s side: the agent is given \(\mathbf{A}=\mathbf{A}^{\ast}\) and \(\mathbf{B}=\mathbf{B}^{\ast}\), so it is not uncertain about how the environment works. It is uncertain about the state, now and over the horizon. It does not know the starting state, and begins from a uniform prior over the 96 combinations. Dirt in a room the Roomba is not in is never observed, so that uncertainty grows until the Roomba goes back. Going back to check a room has value of its own, besides the chance of finding dirt.
Changes in the core elements compared to Part 4.
First, before letting the agent take control of the Roomba, we describe the environment it acts in, i.e. the generative process that produces the data. In this part, the environment’s state has five aspects, each described by its own state factor:
At time \(t\), the true state is the realized state tuple \[ \hat{\mathbf{s}}^{\ast(t)} = \big(\hat{\boldsymbol{s}}^{\ast(t,0)}, \hat{\boldsymbol{s}}^{\ast(t,1)}, \hat{\boldsymbol{s}}^{\ast(t,2)}, \hat{\boldsymbol{s}}^{\ast(t,3)}, \hat{\boldsymbol{s}}^{\ast(t,4)}\big) \] whose entries are the one-hot encoded values of the five factors. The entries differ in size, 3, 4 and 2 values, which is why the state is a tuple rather than a single vector.
In Parts 3 and 4, occupancy was a property of the whereabouts, combined with the room into a single factor. With several factors, it is more natural to describe where the person is. The whereabouts is then occupied exactly when the Roomba and the person are in the same room.
The initial state is given by the tuple of initial state distributions \(\mathbf{D}^{\ast} = \big(\boldsymbol{D}^{\ast(0)}, \ldots, \boldsymbol{D}^{\ast(4)}\big)\). In our simulation, the run always starts from the same state: the Roomba in the bedroom, the person in the living room, and all rooms clean, so each \(\boldsymbol{D}^{\ast(n)}\) is a one-hot vector.
Actions. As in Part 4, at each time step the agent chooses an action, the room the Roomba heads for,
and the environment carries it out. Heading for the room it is already in means staying put. The action chosen at time \(t-1\) determines how the Roomba moves from \(t-1\) to \(t\).
Transitions, one per factor. The true transition probabilities are captured by the tuple \(\mathbf{B}^{\ast} = \big(\boldsymbol{B}^{\ast(0)}, \ldots, \boldsymbol{B}^{\ast(4)}\big)\), with one tensor per factor. Given the previous state and action, each factor changes independently of the others, and each depends only on a few factors:
The dirt factors are where the dependencies between factors appear in the transitions: a room’s dirt depends on its own previous value and on where the Roomba was. The action influences the dirt only indirectly, through where it takes the Roomba.
Our Roomba is equipped with three sensors. The first is a small camera that tracks the surface it is moving over, since there are
However, this method is not foolproof, and sometimes the Roomba will mistake one floor type for another. Don’t be too hard on the little guy, it’s just a Roomba after all.
The second is a motion sensor that reports whether it detects movement. It is the most direct evidence that the person is in the whereabouts: it usually detects motion when they are, but sometimes misses them when they are sitting still, and occasionally detects motion when they are not.
The third, new sensor is a dirt sensor that reports whether the Roomba’s brushes are picking up dirt. It usually does so when the room it is in is dirty, but sometimes misses dirt, and occasionally reports dirt in a clean room. The light sensor of the previous parts is dropped.
At time \(t\), the received observation tuple is \(\hat{\mathbf{o}}^{(t)} = \big(\hat{\boldsymbol{o}}^{(t,0)}, \hat{\boldsymbol{o}}^{(t,1)}, \hat{\boldsymbol{o}}^{(t,2)}\big)\), whose entries are the one-hot encoded values of three observation modalities:
Observations, each depending on a few factors. The true mapping from the state to the observations is \(\mathbf{A}^{\ast} = \big(\boldsymbol{A}^{\ast(0)}, \boldsymbol{A}^{\ast(1)}, \boldsymbol{A}^{\ast(2)}\big)\), with one likelihood tensor per modality:
The tensors also encode the probability that a sensor gives a misleading reading. Observations do not depend on the action. This leaves us with the following generative process specification, where a realized value without bold, such as \(\hat{s}^{\ast(t-1,0)}\), is used as an index into a tensor:
\[ \mathcal{E} \;\triangleq\; \begin{cases} \hat{\boldsymbol{s}}^{\ast(0,n)} \sim \mathcal{Cat}\big(\boldsymbol{D}^{\ast(n)}\big), \quad n = 0, \ldots, 4 & \text{INITIAL STATE}\\[4pt] \hat{\boldsymbol{s}}^{\ast(t,0)} \sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(0)}[\bullet, \hat{s}^{\ast(t-1,0)}, \hat{u}^{(t-1,0)}]\big), \quad t>0 & \text{ROOM}\\[2pt] \hat{\boldsymbol{s}}^{\ast(t,1)} \sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(1)}[\bullet, \hat{s}^{\ast(t-1,1)}]\big), \quad t>0 & \text{PERSON}\\[2pt] \hat{\boldsymbol{s}}^{\ast(t,n)} \sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(n)}[\bullet, \hat{s}^{\ast(t-1,n)}, \hat{s}^{\ast(t-1,0)}]\big), \quad t>0,\ n = 2, 3, 4 & \text{DIRT}\\[4pt] \hat{\boldsymbol{o}}^{(t,0)} \sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(0)}[\bullet, \hat{s}^{\ast(t,0)}]\big) & \text{TEXTURE}\\[2pt] \hat{\boldsymbol{o}}^{(t,1)} \sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(1)}[\bullet, \hat{s}^{\ast(t,0)}, \hat{s}^{\ast(t,1)}]\big) & \text{MOTION}\\[2pt] \hat{\boldsymbol{o}}^{(t,2)} \sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(2)}[\bullet, \hat{s}^{\ast(t,0)}, \hat{s}^{\ast(t,2)}, \hat{s}^{\ast(t,3)}, \hat{s}^{\ast(t,4)}]\big) & \text{DIRT SENSOR} \end{cases} \]
Each line picks out, from a tensor, the probability vector for the current values of exactly the factors it depends on, as listed in 4.1. The specification makes the structure of the environment visible at a glance: each factor and each sensor looks at only a few others.
With actions and several state factors, this process has the structure of a factorized partially observable Markov decision process: five hidden Markov chains, one per factor, coupled through the dependencies in their transitions, with the Room chain driven by the agent’s actions, and three sensors emitting noisy observations of them. The agent never sees the true state directly. In this part, it is given its model of the environment, \(\mathbf{A} = \mathbf{A}^{\ast}\) and \(\mathbf{B} = \mathbf{B}^{\ast}\), and its goal is to choose actions that keep the rooms clean while staying out of the person’s way.
Changes in the environment compared to Part 4.
## This is the first time the tuple has entries of different sizes: 3, 4 and 2 values.
## That's exactly what tuple0 is for, and why the state isn't a single vector.
## factor 0 (site): 1 bedroom 2 living room 3 bathroom
## factor 1 (occupant): 1 bedroom 2 living room 3 bathroom 4 away
## factors 2–4 (Dirt in bedroom, living room, bathroom): 1 clean 2 dirty
Lŝˣ₀ = tuple0(
[1.0, 0.0, 0.0], ## ŝ^{*(0,0)}: Roomba in the bedroom
[0.0, 1.0, 0.0, 0.0], ## ŝ^{*(0,1)}: person in the living room
[1.0, 0.0], ## ŝ^{*(0,2)}: bedroom clean
[1.0, 0.0], ## ŝ^{*(0,3)}: living room clean
[1.0, 0.0], ## ŝ^{*(0,4)}: bathroom clean
) ## ŝ^{*(0)}: the initial state tuple5-element OffsetArray(::Vector{Vector{Float64}}, 0:4) with eltype Vector{Float64} with indices 0:4:
[1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0](["bedroom", "living room", "bathroom"], ["bedroom", "living room", "bathroom", "away"], ["clean", "dirty"])The state variables represent what we need to know. The true state at time \(t\) is given by a (compound) tuple \(\hat{\mathbf{s}}^{\ast(t)}\) whose components are the one-hot encoded state factors (here five factors):
\[ \hat{\mathbf{s}}^{\ast(t)} = \big(\hat{\boldsymbol{s}}^{\ast(t,0)}, \hat{\boldsymbol{s}}^{\ast(t,1)}, \hat{\boldsymbol{s}}^{\ast(t,2)}, \hat{\boldsymbol{s}}^{\ast(t,3)}, \hat{\boldsymbol{s}}^{\ast(t,4)}\big) \]
where the state factors are:
The factors have different numbers of values, 3, 4 and 2, so their one-hot vectors differ in length. This is why the state is a tuple rather than a single vector. All combinations together give \(3 \cdot 4 \cdot 2 \cdot 2 \cdot 2 = 96\) possible states.
The observation variables represent what we can sense. Similarly, the (compound) observation tuple received by the Roomba at time \(t\) is given by
\[ \hat{\mathbf{o}}^{(t)} = \big(\hat{\boldsymbol{o}}^{(t,0)}, \hat{\boldsymbol{o}}^{(t,1)}, \hat{\boldsymbol{o}}^{(t,2)}\big) \]
where the observation modalities are:
Changes in the state and observation variables compared to Part 4.
The decision variables represent what we can control. As in Part 4, the agent chooses an action at each time step. The action at time \(t\) is given by a (compound) tuple \(\hat{\mathbf{u}}^{(t)}\) whose components are the one-hot encoded control factors (here a single factor):
\[ \hat{\mathbf{u}}^{(t)} = \big(\hat{\boldsymbol{u}}^{(t,0)}\big) \]
where the control factors are:
Heading for the room the Roomba is already in means staying put. The action chosen at time \(t\) takes effect at time \(t+1\): it selects which of the transition matrices \(\boldsymbol{B}^{\ast(0)}[\bullet, \bullet, u]\) the environment uses for the whereabouts in the next step.
Of the five state factors, the action controls only one, the whereabouts, but its effects reach further. Through where it takes the Roomba, the action also determines which room gets cleaned: the dirt factors depend on the whereabouts, so the action influences them indirectly. The occupant, on the other hand, is beyond the agent’s control: it can choose its room, but not its company.
The agent’s choice is therefore always a bet on two things at once. It heads for a room it expects to be quiet and in need of cleaning, and its sensors then tell it whether those expectations came true: the motion sensor whether the person is there, the dirt sensor whether there is dirt to pick up.
Changes in the decision variables compared to Part 4.
Exogenous forces represent what changes on its own: the influences on the environment that the agent cannot control, arriving between its decisions. In an active inference model, they appear as the stochastic dynamics of the state factors, in so far as these dynamics do not depend on the agent’s actions.
Forces versus factors. A state factor and an exogenous force are not the same thing. In the sequential-decision framework, the state \(S_t\), the decision \(x_t\) and the exogenous information \(W_{t+1}\) are separate, and the next state follows from all three:
\[ S_{t+1} = S^M\big(S_t, x_t, W_{t+1}\big) \]
The occupant, for example, is a state; the person deciding to walk to the bathroom is the exogenous force acting on it. In an active inference model, such a force has no variable of its own: it is contained in the randomness of the transition tensors \(\mathbf{B}^{\ast}\).
Kinds of state factors. Not every factor the agent cannot control is driven by exogenous forces alone. With the factors of this part, four kinds can be distinguished:
| kind of factor | example | exogenous? |
|---|---|---|
| controlled: the action selects its transition | Room | only the chance that a move fails |
| indirectly influenced: not controlled, but its transition depends on a controlled factor | Dirt (cleaning depends on where the Roomba is) | partly: dirt accumulates exogenously, but is removed by the agent |
| fully exogenous: its transition depends on nothing the agent does | Person | yes |
| static: never changes at all | the layout of the apartment | no force acts on it: it is a hidden fact, not a force |
The exogenous forces in this part. There are two:
In the previous parts, the changes in occupancy were an exogenous force as well, but they were folded into the combined station factor. With a separate factor for the person, they are now explicit. Further exogenous forces could be added in later parts, for example doors that are closed at random.
Changes in the exogenous forces compared to Part 4.
Starting from \(\hat{\boldsymbol{s}}^{\ast(0,n)} \sim \mathcal{Cat}\big(\boldsymbol{D}^{\ast(n)}\big)\), \(n = 0, \ldots, 4\), with one-hot vectors for the Roomba in the bedroom, the person in the living room, and all rooms clean, the environment evolves according to
\[ \begin{aligned} \hat{\boldsymbol{s}}^{\ast(t,0)} &\sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(0)}[\bullet, \hat{s}^{\ast(t-1,0)}, \hat{u}^{(t-1,0)}]\big), \quad t>0 && \text{ROOM}\\ \hat{\boldsymbol{s}}^{\ast(t,1)} &\sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(1)}[\bullet, \hat{s}^{\ast(t-1,1)}]\big), \quad t>0 && \text{PERSON}\\ \hat{\boldsymbol{s}}^{\ast(t,n)} &\sim \mathcal{Cat}\big(\boldsymbol{B}^{\ast(n)}[\bullet, \hat{s}^{\ast(t-1,n)}, \hat{s}^{\ast(t-1,0)}]\big), \quad t>0,\ n = 2, 3, 4 && \text{DIRT}\\ \hat{\boldsymbol{o}}^{(t,0)} &\sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(0)}[\bullet, \hat{s}^{\ast(t,0)}]\big) && \text{TEXTURE}\\ \hat{\boldsymbol{o}}^{(t,1)} &\sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(1)}[\bullet, \hat{s}^{\ast(t,0)}, \hat{s}^{\ast(t,1)}]\big) && \text{MOTION}\\ \hat{\boldsymbol{o}}^{(t,2)} &\sim \mathcal{Cat}\big(\boldsymbol{A}^{\ast(2)}[\bullet, \hat{s}^{\ast(t,0)}, \hat{s}^{\ast(t,2)}, \hat{s}^{\ast(t,3)}, \hat{s}^{\ast(t,4)}]\big) && \text{DIRT SENSOR} \end{aligned} \]
where \(\hat{u}^{(t-1,0)}\) is the action chosen at time \(t-1\). With one tensor per factor, each table below stays small: the 96 joint states never have to be written out.
One table per action \(u\). If the Roomba heads for the room it is in, it stays. If it heads for an adjacent room, it gets there with \(p_{\text{move}} = 0.9\), and otherwise stays where it is. There is no door between the bedroom and the living room, so when it heads from one to the other, it first moves to the bathroom. Each column is a probability distribution over the next room, given the previous room, so each column sums to 1.
With the heading “bedroom”.
| next room previous room | bedroom | living room | bathroom |
|---|---|---|---|
| bedroom | 1.0 | 0.0 | 0.9 |
| living room | 0.0 | 0.1 | 0.0 |
| bathroom | 0.0 | 0.9 | 0.1 |
With the heading “living room”.
| next room previous room | bedroom | living room | bathroom |
|---|---|---|---|
| bedroom | 0.1 | 0.0 | 0.0 |
| living room | 0.0 | 1.0 | 0.9 |
| bathroom | 0.9 | 0.0 | 0.1 |
With the heading “bathroom”.
| next room previous room | bedroom | living room | bathroom |
|---|---|---|---|
| bedroom | 0.1 | 0.0 | 0.0 |
| living room | 0.0 | 0.1 | 0.0 |
| bathroom | 0.9 | 0.9 | 1.0 |
The bathroom is the hub of the apartment: from the bedroom, heading for the living room leads to the bathroom first, and vice versa.
The person moves independently of the Roomba and of the action. Each column is a probability distribution over the person’s next location, given the previous one, so each column sums to 1.
| next previous | bedroom | living room | bathroom | away |
|---|---|---|---|---|
| bedroom | 0.80 | 0.03 | 0.20 | 0.00 |
| living room | 0.13 | 0.90 | 0.30 | 0.05 |
| bathroom | 0.05 | 0.04 | 0.50 | 0.00 |
| away | 0.02 | 0.03 | 0.00 | 0.95 |
The living room, with its abundance of natural light, is where the person spends most time (staying with 0.90). Bathroom visits are short (0.50). When the person leaves the apartment, they mostly stay away for a while (0.95), and they come back through the living room.
Each dirt factor depends on its own previous value and on where the Roomba was. If the Roomba was in that room, a dirty room becomes clean with \(p_{\text{clean}} = 0.8\), and a clean room stays clean. If the Roomba was elsewhere, a clean room becomes dirty with \(p_{\text{dirty}}(n)\), and a dirty room stays dirty. Each column is a probability distribution over the next value, given the previous one, so each column sums to 1.
With the Roomba in this room.
| next previous | clean | dirty |
|---|---|---|
| clean | 1.0 | 0.8 |
| dirty | 0.0 | 0.2 |
With the Roomba elsewhere.
| next previous | clean | dirty |
|---|---|---|
| clean | \(1 - p_{\text{dirty}}(n)\) | 0.0 |
| dirty | \(p_{\text{dirty}}(n)\) | 1.0 |
with a rate of dirt accumulation that differs per room:
| bedroom | living room | bathroom | |
|---|---|---|---|
| \(p_{\text{dirty}}\) | 0.02 | 0.05 | 0.03 |
The living room, the room in most use, gets dirty fastest: on average after about 20 ticks, against 50 for the bedroom and about 33 for the bathroom. Since the whereabouts has three possible values, \(\boldsymbol{B}^{\ast(n)}\) consists of three \(2 \times 2\) matrices: the cleaning table for the value matching room \(n\), and the accumulating table for the other two.
The texture depends only on the whereabouts. Each column is a probability distribution over the observed surface, so each column sums to 1.
| observation \(o^{(t,0)}\) room | bedroom | living room | bathroom |
|---|---|---|---|
| hardwood | 0.9 | 0.05 | 0.05 |
| carpet | 0.05 | 0.9 | 0.05 |
| tiles | 0.05 | 0.05 | 0.9 |
The off-diagonal entries are the camera’s mistakes: in each room, it misreads the floor with probability 0.1, split equally between the two other textures.
Motion depends on the whereabouts and the occupant, but only through one question: are they in the same room? The table gives the probability of detecting “motion”; the probability of “no motion” is 1 minus each entry.
| room person | bedroom | living room | bathroom | away |
|---|---|---|---|---|
| bedroom | 0.8 | 0.05 | 0.05 | 0.05 |
| living room | 0.05 | 0.8 | 0.05 | 0.05 |
| bathroom | 0.05 | 0.05 | 0.8 | 0.05 |
The motion sensor detects the person with probability 0.8 when they are in the whereabouts; it misses them when they are sitting still with probability 0.2. Otherwise, it raises a false alarm with probability 0.05.
The dirt reading depends on the whereabouts and on the dirt in all three rooms, but again only through one question: is the room the Roomba is in dirty? Each column is a probability distribution over the dirt reading, so each column sums to 1.
| observation \(o^{(t,2)}\) dirt in the whereabouts | clean | dirty |
|---|---|---|
| nothing picked up | 0.95 | 0.1 |
| dirt picked up | 0.05 | 0.9 |
The dirt sensor picks up dirt in a dirty room with probability 0.9, and misses it with probability 0.1. In a clean room, it falsely reports dirt with probability 0.05. Although \(\boldsymbol{A}^{\ast(2)}\) has \(2 \times 3 \times 2 \times 2 \times 2 = 48\) entries, they are all filled from this small table: for each room, the column is selected by the dirt factor of that room.
Changes in the transition and observation functions compared to Part 4.
## factor 0 (site): 1 bedroom 2 living room 3 bathroom
## factor 1 (occupant): 1 bedroom 2 living room 3 bathroom 4 away
## factors 2–4 (Dirt in bedroom, living room, bathroom): 1 clean 2 dirty
## actions (target room): 1 bedroom 2 living room 3 bathroom
## ---------- factor 0: site (3×3×3: next room, previous room, action) ----------
p_move = 0.9 ## probability that a move succeeds
next_room(r, u) = r == u ? r : (r == 3 || u == 3) ? u : 3 ## no door between bedroom 1 and living room 2
_Bˣı0ı = zeros(3, 3, 3)
for u in 1:3, r in 1:3
r′ = next_room(r, u)
if r′ == r
_Bˣı0ı[r, r, u] = 1.0 ## heading for the current room: stay
else
_Bˣı0ı[r′, r, u] = p_move ## move succeeds
_Bˣı0ı[r, r, u] = 1 - p_move ## move fails: stay
end
end
## ---------- factor 1: occupant (4×4: next location, previous location) ----------
_Bˣı1ı = [0.80 0.03 0.20 0.00; ## to bedroom (columns: from bedroom, living room, bathroom, away)
0.13 0.90 0.30 0.05; ## to living room
0.05 0.04 0.50 0.00; ## to bathroom
0.02 0.03 0.00 0.95] ## to away
## ---------- factors 2–4: pickup (2×2×3: next dirt, previous dirt, previous room of the Roomba) ----------
p_clean = 0.8 ## probability that a dirty room is cleaned, per tick
p_dirty = [0.02, 0.05, 0.03] ## rate of dirt accumulation: bedroom, living room, bathroom
function dirt_tensor(room, p_dirty_room)
_B = zeros(2, 2, 3)
for r in 1:3
if r == room ## the Roomba was in this room: cleaning
_B[:, :, r] = [1.0 p_clean;
0.0 1 - p_clean]
else ## the Roomba was elsewhere: dirt accumulating
_B[:, :, r] = [1 - p_dirty_room 0.0;
p_dirty_room 1.0]
end
end
_B
end
_Bˣı2ı = dirt_tensor(1, p_dirty[1])
_Bˣı3ı = dirt_tensor(2, p_dirty[2])
_Bˣı4ı = dirt_tensor(3, p_dirty[3])
LBˣ = tuple0(
_Bˣı0ı, ## B^{*(0)}: site
_Bˣı1ı, ## B^{*(1)}: occupant
_Bˣı2ı, ## B^{*(2)}: bed dust
_Bˣı3ı, ## B^{*(3)}: lounge grime
_Bˣı4ı, ## B^{*(4)}: bath soil
)
## D_B^{(n)}: the factors whose previous values each transition depends on (own factor first)
LdepsB = tuple0([whereabouts_n], [occupant_n], [bed_dust_n, whereabouts_n], [lounge_grime_n, whereabouts_n], [bath_soil_n, whereabouts_n])
size.(LBˣ) ## (3,3,3), (4,4), (2,2,3), (2,2,3), (2,2,3)5-element OffsetArray(::Vector{Tuple{Int64, Int64, Vararg{Int64}}}, 0:4) with eltype Tuple{Int64, Int64, Vararg{Int64}} with indices 0:4:
(3, 3, 3)
(4, 4)
(2, 2, 3)
(2, 2, 3)
(2, 2, 3)Changes in the transition code compared to Part 4.
idx, the occupancy rules p_occ and p_stay, and the combined \(6 \times 6 \times 3\) array _B_joint are removed. Each factor gets its own tensor, built directly from the tables in 4.3.4._B_room is unchanged from Part 4, now as the tensor of the whereabouts only._B_person, the \(4 \times 4\) transition matrix of the occupant, is entered directly from its table.dirt_tensor builds the \(2 \times 2 \times 3\) tensor for one room’s dirt: the cleaning matrix in the slice where the Roomba was in that room, and the accumulating matrix in the other two slices.LBˣ is a tuple of five tensors, one per state factor, with different shapes: \(3 \times 3 \times 3\), \(4 \times 4\), and three times \(2 \times 2 \times 3\).LdepsB records the dependencies \(\mathcal{D}_B^{(n)}\) of each transition tensor, using 0-based factor indices. Each list starts with the factor itself, and the order of the list matches the order of the tensor’s axes. The action axis of the whereabouts is implicit, since it is the only controlled factor.## factor 0 (site): 1 bedroom 2 living room 3 bathroom
## factor 1 (occupant): 1 bedroom 2 living room 3 bathroom 4 away
## factors 2–4 (Dirt in bedroom, living room, bathroom): 1 clean 2 dirty
## ---------- modality 0: texture (3×3: texture, room) ----------
_Aˣı0ı = [0.9 0.05 0.05; ## hardwood (columns: room bedroom, living room, bathroom)
0.05 0.9 0.05; ## carpet
0.05 0.05 0.9 ] ## tiles
## ---------- modality 1: stir (2×3×4: motion, room, person) ----------
p_detect, p_false_motion = 0.8, 0.05 ## motion if the person is in the site / otherwise
_Aˣı1ı = zeros(2, 3, 4)
for r in 1:3, p in 1:4
p_motion = p == r ? p_detect : p_false_motion ## same room? (person 4 = away is never the same room)
_Aˣı1ı[:, r, p] = [1 - p_motion, p_motion] ## (no motion, motion)
end
## ---------- modality 2: pickup (2×3×2×2×2: dirt reading, room, bed dust, lounge grime, bath soil) ----------
p_pickup, p_false_pickup = 0.9, 0.05 ## dirt picked up in a dirty room / in a clean room
_Aˣı2ı = zeros(2, 3, 2, 2, 2)
for r in 1:3, d₁ in 1:2, d₂ in 1:2, d₃ in 1:2
dirty = (d₁, d₂, d₃)[r] == 2 ## is the site dirty?
p_dirt = dirty ? p_pickup : p_false_pickup
_Aˣı2ı[:, r, d₁, d₂, d₃] = [1 - p_dirt, p_dirt] ## (nothing picked up, dirt picked up)
end
LAˣ = tuple0(
_Aˣı0ı, ## A^{*(0)}: texture
_Aˣı1ı, ## A^{*(1)}: stir
_Aˣı2ı, ## A^{*(2)}: pickup
)
## D_A^{(m)}: the factors each modality depends on, in the order of the tensor's axes after the first
LdepsA = tuple0([whereabouts_n], [whereabouts_n, occupant_n], [whereabouts_n, bed_dust_n, lounge_grime_n, bath_soil_n])
size.(LAˣ) ## (3,3), (2,3,4), (2,3,2,2,2)3-element OffsetArray(::Vector{Tuple{Int64, Int64, Vararg{Int64}}}, 0:2) with eltype Tuple{Int64, Int64, Vararg{Int64}} with indices 0:2:
(3, 3)
(2, 3, 4)
(2, 3, 2, 2, 2)Changes in the observation code compared to Part 4.
repeat constructions that spread each room’s column over the six joint states are no longer needed._A_texture, the \(3 \times 3\) table of the Texture modality, is entered directly from its table._A_motion is a \(2 \times 3 \times 4\) tensor over the whereabouts and the occupant, filled by a loop: motion is detected with probability 0.8 when the two are in the same room, and falsely reported with 0.05 otherwise._A_dirt is a \(2 \times 3 \times 2 \times 2 \times 2\) tensor over the whereabouts and the dirt in all three rooms, filled by a loop that picks out the dirt factor of the whereabouts, (d₁, d₂, d₃)[r], to decide whether dirt is picked up.LdepsA records the dependencies \(\mathcal{D}_A^{(m)}\) of each likelihood tensor, using 0-based factor indices, in the order of the tensor’s axes after the observation axis.3×3 Matrix{Float64}:
0.9 0.05 0.05
0.05 0.9 0.05
0.05 0.05 0.93×4 Matrix{Float64}:
0.8 0.05 0.05 0.05
0.05 0.8 0.05 0.05
0.05 0.05 0.8 0.05The objective function is the designer’s measure of performance, judged with the true states \(\hat{\mathbf{s}}^{\ast(t)}\). The environment does not pursue it, and the agent does not minimize it. The agent’s own objective, the expected free energy, is in 4.5.6.
Two measures are primary, one per goal. Disturbance is the fraction of ticks the Roomba spends in the same room as the person,
\[ J_{\text{disturb}} = \frac{1}{T+1} \sum_{t=0}^{T} \mathbb{1}\big[\hat{s}^{\ast(t,0)} = \hat{s}^{\ast(t,1)}\big]. \]
The occupant value “away” has a different index from every whereabouts value, so the indicator is 0 whenever the person is out. Lower is better.
Cleanliness is the fraction of ticks each dirt factor is “dirty”, and the average of the three,
\[ J_{\text{dirty}}^{(n)} = \frac{1}{T+1} \sum_{t=0}^{T} \mathbb{1}\big[\hat{s}^{\ast(t,n)} = \text{dirty}\big], \qquad J_{\text{dirty}} = \frac{1}{3}\sum_{n=2}^{4} J_{\text{dirty}}^{(n)}, \]
for \(n=2,3,4\), that is bed dust, lounge grime and bath soil. Lower is better. lounge grime is where the goals conflict: the living room gets dirty fastest, and the person uses it most.
Tracking is secondary. It is how often the most probable marginal equals the true value,
\[ J_{\text{track}}^{(n)} = \frac{1}{T+1} \sum_{t=0}^{T} \mathbb{1}\big[\arg\max \boldsymbol{s}^{(t,n)} = \hat{s}^{\ast(t,n)}\big], \qquad n = 0, \ldots, 4. \]
Higher is better, at most 1. The agent can avoid the person and find dirt only if the marginals are usually right.
The baselines in 4.2, the Part 4 agent, and the agent with both preferences are compared on \(J_{\text{disturb}}\) and \(J_{\text{dirty}}\). If the cleaning preference works, both preferences should beat the motion-only agent on \(J_{\text{dirty}}\), especially for lounge grime, at some cost in \(J_{\text{disturb}}\). The single detailed run in 4.6 is for reading the beliefs. The comparison that the conclusion uses is the average over 30 seeds. One run can be misleading.
The agent does not optimize these measures. It never sees \(\hat{\mathbf{s}}^{\ast(t)}\). It acts on \(\log\boldsymbol{C}^{(1)}\) and \(\log\boldsymbol{C}^{(2)}\). The measures say, from outside, how far those preferences go toward the behavior we want.
Changes in the objective function compared to Part 4.
## returns a one-hot encoding of a random sample from a categorical distribution
function rand_1hot_vec(rng, distribution::Categorical)
K = ncategories(distribution)
sample = zeros(K)
drawn_category = rand(rng, distribution)
sample[drawn_category] = 1.0
return sample
endrand_1hot_vec (generic function with 1 method)function create_envir(; LAˣ, LBˣ, LdepsA, LdepsB, Lŝˣ₀, seed=42)
rng = MersenneTwister(seed)
## the current values (1-based indices) of the factors in deps, read from a tuple of one-hot vectors
values_of(Lŝ, deps) = Tuple(argmax(Lŝ[n]) for n in deps)
## ô^{(t,m)} ~ Cat(A^{*(m)}[•, values of the factors in D_A^{(m)}]), for every modality m
generate_obs(Lŝ) = tuple0((rand_1hot_vec(rng, Categorical(LAˣ[m][:, values_of(Lŝ, LdepsA[m])...]))
for m in eachindex(LAˣ))...)
## t = 0
Lŝˣₜ = Lŝˣ₀ ## ŝ^{*(0)}
Lôₜ = generate_obs(Lŝˣₜ) ## ô^{(0)}
execute = (Lûₜ₋₁) -> begin ## Lûₜ₋₁ = û^{(t-1)}: the action chosen at t-1, a tuple of one-hot vectors
u = argmax(Lûₜ₋₁[0]) ## û^{(t-1,0)} as an index: 1 bedroom, 2 living room, 3 bathroom
Lŝˣₜ₋₁ = Lŝˣₜ
## ŝ^{*(t,n)} ~ Cat(B^{*(n)}[•, values of the factors in D_B^{(n)}, (action, for the controlled factor 0)])
Lŝˣₜ = tuple0((begin
action = n == whereabouts_n ? (u,) : () ## only factor 0 (site) has an action axis
p = LBˣ[n][:, values_of(Lŝˣₜ₋₁, LdepsB[n])..., action...]
rand_1hot_vec(rng, Categorical(p))
end for n in eachindex(LBˣ))...)
Lôₜ = generate_obs(Lŝˣₜ) ## ô^{(t)}
nothing
end
observe = () -> Lôₜ ## ô^{(t)} = (ô^{(t,0)}, ô^{(t,1)}, ô^{(t,2)})
state = () -> Lŝˣₜ ## ŝ^{*(t)} = (ŝ^{*(t,0)}, ..., ŝ^{*(t,4)}), true state (for evaluation only)
return (execute, observe, state)
endcreate_envir (generic function with 1 method)Changes in create_envir compared to Part 4.
values_of turns the one-hot vectors of the factors a tensor depends on into indices, in the order of the tensor’s axes as listed in the dependency tuples.generate_obs draws one observation per modality in the same way, both at \(t = 0\) and after each step. The intermediate probability vectors of Part 4 are no longer stored.LdepsA and LdepsB are new arguments, passed in from the cells that define the likelihood and transition tensors. The dependency lists must match the axis order of the tensors exactly: own factor first for the transitions, then the other dependencies in the listed order.In order to generate data to mimic the observations of the Roomba, we need to specify four things: the initial state (i.e., where the Roomba and the person start, and which rooms are dirty), the actual transition probabilities \(\mathbf{B}^{\ast}\), one tensor per state factor (i.e., how likely the Roomba is to reach the room it heads for, how the person moves, and how dirt accumulates and is cleaned), the observation distributions \(\mathbf{A}^{\ast}\), one tensor per sensor (i.e., what texture the Roomba will observe in each room, whether it will detect motion, depending on where the person is, and whether it will pick up dirt, depending on whether its room is dirty), and a way to choose the actions. We can then use these specifications to generate observations from the generative process, which has the structure of a factorized partially observable Markov decision process (POMDP).
In this section, we test the environment on its own, with a Roomba that chooses its destination at random. This also gives the reference point for the measures from 4.3.5: how often a Roomba without preferences ends up in the same room as the person, and how dirty the rooms get. In 4.6, the agent’s choices take the place of the random ones.
To generate our data, we’ll follow these steps: 1. Assume an initial state \(\hat{\mathbf{s}}^{\ast(0)}\), one value per factor: the Roomba in the bedroom, the person in the living room, and all rooms clean. 2. Determine the observations in the starting state, one per sensor \(m = 0, 1, 2\) (Texture, Motion, Dirt), by drawing from the categorical distribution that \(\boldsymbol{A}^{\ast(m)}\) gives for the current values of the factors that sensor depends on. 3. Choose an action \(\hat{u}^{(t-1,0)}\), the room the Roomba heads for. Here, it is drawn at random from the three rooms. 4. Determine the next state, one factor at a time: for each factor \(n\), draw its next value from the categorical distribution that \(\boldsymbol{B}^{\ast(n)}\) gives for the previous values of the factors it depends on, and, for the whereabouts, the chosen action. The Roomba moves, the person moves, and the dirt in each room grows or is cleaned. 5. Determine the observations in the new state, as in step 2. 6. Repeat steps 3-5 for as many time steps as we want.
We will generate 600 data points (\(t = 0, \ldots, T\) with \(T = 599\)) to simulate 600 ticks of the Roomba moving through the apartment. Lôː will contain the Roomba’s observations at each tick, \(\hat{\mathbf{o}}^{(0:T)}\), where each entry is a tuple holding the texture of the floor it’s on, whether it detects motion, and whether it picks up dirt. Lûː will contain the action chosen at each tick, \(\hat{\mathbf{u}}^{(0:T-1)}\). Lŝˣː will contain the true state at each tick, \(\hat{\mathbf{s}}^{\ast(0:T)}\): where the Roomba and the person actually were, and which rooms were actually dirty.
The following code implements this process and generates our data:
Changes in the data generation compared to Part 4.
T = 599 ## 600 ticks: t = 0, ..., T
## factor 0 (site): 1 bedroom 2 living room 3 bathroom
## factor 1 (occupant): 1 bedroom 2 living room 3 bathroom 4 away
## factors 2–4 (Dirt in bedroom, living room, bathroom): 1 clean 2 dirty
Lŝˣ₀ = tuple0([1.0, 0.0, 0.0], ## Roomba in the bedroom
[0.0, 1.0, 0.0, 0.0], ## person in the living room
[1.0, 0.0], ## bedroom clean
[1.0, 0.0], ## living room clean
[1.0, 0.0]) ## bathroom clean → ŝ^{*(0)}
(execute_sim, observe_sim, state_sim) = create_envir(; LAˣ, LBˣ, LdepsA, LdepsB, Lŝˣ₀)
## random Roomba: chooses its destination at random (reference point for 4.3.5)
rng_act = MersenneTwister(7) ## separate random stream for the actions
random_action(rng) = tuple0(Float64.(1:3 .== rand(rng, 1:3))) ## a random one-hot action tuple
Lŝˣː = offset0(Vector{Any}(undef, T+1)) ## ŝ^{*(0:T)}: true states, each a tuple (Room, Person, Dirt ×3)
Lôː = offset0(Vector{Any}(undef, T+1)) ## ô^{(0:T)}: observations, each a tuple (Texture, Motion, Dirt)
Lûː = offset0(Vector{Any}(undef, T)) ## û^{(0:T-1)}: chosen actions, each a tuple (heading)
for t = 0:T
if t > 0
Lûː[t-1] = random_action(rng_act) ## choose û^{(t-1)} (here at random)
execute_sim(Lûː[t-1]) ## the environment carries it out: move, update person and dirt, observe
end
Lŝˣː[t] = state_sim()
Lôː[t] = observe_sim()
endChanges in the random Roomba simulation compared to Part 4.
create_envir receives the dependency tuples LdepsA and LdepsB, which tell the environment which factors each tensor depends on.600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
⋮
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0][Lŝˣ[1] for Lŝˣ in Lŝˣː] ## ŝ^{*(t,1)} for t = 0, ..., T: the Person factor (one-hot over 4 locations)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[1.0, 0.0, 0.0, 0.0]
[1.0, 0.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
⋮
[0.0, 0.0, 0.0, 1.0]
[0.0, 0.0, 0.0, 1.0]
[0.0, 1.0, 0.0, 0.0]
[1.0, 0.0, 0.0, 0.0]
[0.0, 0.0, 1.0, 0.0]
[0.0, 0.0, 1.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0][Lŝˣ[3] for Lŝˣ in Lŝˣː] ## ŝ^{*(t,3)} for t = 0, ..., T: the Dirt factor of the living room (one-hot: clean, dirty)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
⋮
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]## t, site, occupant, bed dust, lounge grime, bath soil
[(t,
room_labels[argmax(Lŝˣː[t][0])],
person_labels[argmax(Lŝˣː[t][1])],
(dirt_labels[argmax(Lŝˣː[t][n])] for n in 2:4)...)
for t in 0:T]600-element Vector{Tuple{Int64, Vararg{String, 5}}}:
(0, "bedroom", "living room", "clean", "clean", "clean")
(1, "bedroom", "living room", "clean", "clean", "clean")
(2, "bathroom", "living room", "clean", "clean", "clean")
(3, "bedroom", "living room", "clean", "clean", "clean")
(4, "bathroom", "living room", "clean", "clean", "clean")
(5, "bathroom", "bedroom", "clean", "clean", "clean")
(6, "living room", "bedroom", "clean", "clean", "clean")
(7, "living room", "living room", "clean", "clean", "clean")
(8, "bathroom", "living room", "clean", "clean", "clean")
(9, "bathroom", "living room", "clean", "dirty", "clean")
⋮
(591, "bathroom", "away", "clean", "clean", "clean")
(592, "bedroom", "away", "clean", "clean", "clean")
(593, "bathroom", "living room", "clean", "clean", "clean")
(594, "bathroom", "bedroom", "clean", "clean", "clean")
(595, "bedroom", "bathroom", "clean", "clean", "clean")
(596, "bathroom", "bathroom", "clean", "clean", "clean")
(597, "bathroom", "living room", "clean", "clean", "clean")
(598, "bathroom", "living room", "clean", "clean", "clean")
(599, "bathroom", "living room", "clean", "clean", "clean")599-element OffsetArray(::Vector{Vector{Float64}}, 0:598) with eltype Vector{Float64} with indices 0:598:
[1.0, 0.0, 0.0]
[0.0, 1.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 1.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
⋮
[1.0, 0.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]599-element OffsetArray(::Vector{String}, 0:598) with eltype String with indices 0:598:
"bedroom"
"living room"
"bedroom"
"bathroom"
"bathroom"
"living room"
"living room"
"bedroom"
"bathroom"
"bathroom"
⋮
"bedroom"
"bedroom"
"living room"
"bathroom"
"bedroom"
"bathroom"
"bathroom"
"bathroom"
"living room"600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
⋮
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]
[0.0, 0.0, 1.0]600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
⋮
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0][Lô[2] for Lô in Lôː] ## ô^{(t,2)} for t = 0, ..., T: the Dirt modality (nothing picked up, dirt picked up)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
⋮
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]rooms = [argmax(Lŝˣ[0]) for Lŝˣ in parent(Lŝˣː)] ## site ŝ^{*(t,0)}: 1 bedroom, 2 living room, 3 bathroom
persons = [argmax(Lŝˣ[1]) for Lŝˣ in parent(Lŝˣː)] ## occupant ŝ^{*(t,1)}: 1 bedroom, 2 living room, 3 bathroom, 4 away
same_room = rooms .== persons ## the Roomba disturbs the person
dirty = [[argmax(Lŝˣ[n]) == 2 for Lŝˣ in parent(Lŝˣː)] for n in 2:4] ## bed dust ŝ^{*(t,2)}, lounge grime ŝ^{*(t,3)}, bath soil ŝ^{*(t,4)}
destinations = [argmax(Lû[0]) for Lû in parent(Lûː)] ## chosen destination û^{(t,0)}, t = 0, ..., T-1
p_room = plot(0:T, rooms, linetype=:steppost, leg=:topright, label="site ŝ*(t,0)",
title="True site ŝ*(t,0)",
yticks=(1:3, whereabouts_labels), xlims=(0, T))
scatter!(p_room, 0:T-1, destinations, markersize=2, markerstrokewidth=0, color=:gray, alpha=0.6,
label="destination chosen û(t,0)")
p_person = plot(0:T, persons, linetype=:steppost, leg=:topright, label="occupant ŝ*(t,1)", color=:red,
yticks=(1:4, occupant_labels), xlims=(0, T))
scatter!(p_person, (0:T)[same_room], persons[same_room], markersize=2, markerstrokewidth=0, color=:black,
label="same room as Roomba")
p_dirt = plot(leg=:topright, xlabel="Time t", yticks=(1:3, whereabouts_labels),
ylims=(0.5, 3.5), xlims=(0, T))
for (r, room) in enumerate(["bedroom", "living room", "bathroom"])
scatter!(p_dirt, (0:T)[dirty[r]], fill(r, count(dirty[r])), markersize=2, markerstrokewidth=0,
color=:brown, label=(r == 1 ? "dirty ŝ*(t,2), ŝ*(t,3), ŝ*(t,4)" : ""))
end
plot(p_room, p_person, p_dirt, layout=@layout([a; b; c]), size=(900, 650), link=:x)
Changes in the states plot compared to Part 4.
same_room: the whereabouts equals the occupant, which is the disturbance condition from 4.3.5.textures = [argmax(Lô[0]) for Lô in parent(Lôː)] ## surface index (1, 2, 3) of ô^{(t,0)} for t = 0, ..., T
plot(title="Observed texture")
plot!(
0:T, textures, linetype=:steppost, linestyle=:solid, leg=:right,
label="observed surface", xlabel="Time t", yticks=(1:3, surface_labels))
pickups = [argmax(Lô[2]) for Lô in parent(Lôː)] ## dirt index (1 = nothing, 2 = dirt picked up) of ô^{(t,2)} for t = 0, ..., T
plot(title="Observed pickup")
plot!(
0:T, pickups, linetype=:steppost, linestyle=:solid, leg=:right,
label="dirt reading", xlabel="Time t", yticks=(1:2, pickup_labels))
motions = [argmax(Lô[1]) for Lô in parent(Lôː)] ## motion index (1 = no motion, 2 = motion) of ô^{(t,1)} for t = 0, ..., T
plot(title="Observed stir")
plot!(
0:T, motions, linetype=:steppost, linestyle=:solid, leg=:right,
label="observed commotion", xlabel="Time t", yticks=(1:2, commotion_labels))
rooms = [argmax(Lŝˣ[0]) for Lŝˣ in parent(Lŝˣː)] ## site ŝ^{*(t,0)}: 1 bedroom, 2 living room, 3 bathroom
persons = [argmax(Lŝˣ[1]) for Lŝˣ in parent(Lŝˣː)] ## occupant ŝ^{*(t,1)}: 1 bedroom, 2 living room, 3 bathroom, 4 away
same_room = (rooms .== persons) .+ 1 ## 1 no, 2 yes: the person is in the site, ŝ^{*(t,0)} = ŝ^{*(t,1)}
dirt_here = [argmax(Lŝˣː[t][rooms[t+1] + 1]) for t in 0:T] ## 1 clean, 2 dirty: the dirt factor of the site, ŝ^{*(t, 1+ŝ^{*(t,0)})}
textures = [argmax(Lô[0]) for Lô in parent(Lôː)] ## surface ô^{(t,0)}: 1 hardwood, 2 carpet, 3 tiles
motions = [argmax(Lô[1]) for Lô in parent(Lôː)] ## motion ô^{(t,1)}: 1 no motion, 2 motion
pickups = [argmax(Lô[2]) for Lô in parent(Lôː)] ## dirt reading ô^{(t,2)}: 1 nothing, 2 dirt picked up
destinations = [argmax(Lû[0]) for Lû in parent(Lûː)] ## destination û^{(t,0)}: 1 bedroom, 2 living room, 3 bathroom (t = 0, ..., T-1)
window = (0, 150) ## zoom in so individual ticks are visible; use (0, T) for the full run
panel(ts, y, ticks, label, color) = plot(ts, y, linetype=:steppost, label=label, color=color,
yticks=ticks, xlims=window, legend=:topright)
plot(
panel(0:T-1, destinations, (1:3, whereabouts_labels), "destination û(t,0)", :gray),
panel(0:T, rooms, (1:3, whereabouts_labels), "site ŝ*(t,0)", :blue),
panel(0:T, textures, (1:3, surface_labels), "surface ô(t,0)", :steelblue),
panel(0:T, persons, (1:4, occupant_labels), "occupant ŝ*(t,1)", :red),
panel(0:T, same_room, (1:2, ["no", "yes"]), "same room", :red),
panel(0:T, motions, (1:2, commotion_labels), "motion ô(t,1)", :darkred),
panel(0:T, dirt_here, (1:2, dust_labels), "site dirty ŝ*(t, 1+ŝ*(t,0))", :brown),
panel(0:T, pickups, (1:2, pickup_labels), "dirt reading ô(t,2)", :sienna),
layout=(8, 1), size=(900, 1150), link=:x, xlabel=["" "" "" "" "" "" "" "Time t"],
plot_title="Random heading û(t,0): actions, states and sensors",
)
plot(title="True site and observed surface")
plot!(0:T, rooms, linetype=:steppost, label="true room",
yticks=(1:3, [whereabouts_labels[i] * " / " * surface_labels[i] for i in eachindex(whereabouts_labels)]))
plot!(0:T, textures, linetype=:steppost, linestyle=:dash, label="observed surface",
xlabel="Time t", leg=:right)
plot(title="Person in the room, and observed stir")
plot!(0:T, same_room, linetype=:steppost, label="person in the site", color=:red,
yticks=(1:2, ["no / " * commotion_labels[1], "yes / " * commotion_labels[2]]))
plot!(0:T, motions, linetype=:steppost, linestyle=:dash, label="observed commotion", color=:darkred,
xlabel="Time t", leg=:right)
mistakes = count(rooms .!= textures)
println("camera mistakes: $mistakes of $(T+1) ticks ($(round(100*mistakes/(T+1), digits=1))%)")camera mistakes: 49 of 600 ticks (8.2%)present = same_room .== 2
misses = count(present .& (motions .== 1)) ## person in the room, no motion detected
false_alarms = count(.!present .& (motions .== 2)) ## person elsewhere, motion detected
println("motion sensor misses: $misses of $(count(present)) ticks with the person present ($(round(100*misses/count(present), digits=1))%)")
println("motion sensor false alarms: $false_alarms of $(count(.!present)) ticks without ($(round(100*false_alarms/count(.!present), digits=1))%)")motion sensor misses: 16 of 95 ticks with the person present (16.8%)
motion sensor false alarms: 13 of 505 ticks without (2.6%)## Empirical rate of dirt accumulation per room, compared with p_dirty from 4.3.4.
## Counts the ticks at which a room was clean and the Roomba was elsewhere,
## and how often the room was dirty at the next tick.
dirty = [[argmax(Lŝˣ[n]) == 2 for Lŝˣ in parent(Lŝˣː)] for n in 2:4] ## bed dust, lounge grime, bath soil
for (r, room) in enumerate(whereabouts_labels)
chances = [t for t in 1:T if !dirty[r][t] && rooms[t] != r] ## clean, and the Roomba elsewhere (1-based positions)
rate = mean(dirty[r][t+1] for t in chances)
println("$room: got dirty at $(round(rate, digits=3)) of $(length(chances)) chances (p_dirty: $(p_dirty[r]))")
endbedroom: got dirty at 0.021 of 423 chances (p_dirty: 0.02)
living room: got dirty at 0.05 of 341 chances (p_dirty: 0.05)
bathroom: got dirty at 0.035 of 284 chances (p_dirty: 0.03)## ---------- random Roomba: reference values for 4.3.5 ----------
rooms_random = [argmax(Lŝˣ[0]) for Lŝˣ in parent(Lŝˣː)]
persons_random = [argmax(Lŝˣ[1]) for Lŝˣ in parent(Lŝˣː)]
dirty_random = [[argmax(Lŝˣ[n]) == 2 for Lŝˣ in parent(Lŝˣː)] for n in 2:4] ## bed dust, lounge grime, bath soil
disturbance_random = mean(rooms_random .== persons_random) ## J_disturb: fraction of ticks in the person's room
dirtiness_random = [mean(dirty_random[r]) for r in 1:3] ## J_dirty(r): fraction of ticks each room is dirty
coverage_random = [mean(rooms_random .== r) for r in 1:3] ## fraction of ticks in each room, for reference
println("random Roomba, disturbance: $(round(100*disturbance_random, digits=1))% of ticks in the person's room")
println("random Roomba, dirty: ",
join(["$(r): $(round(100*dirtiness_random[i], digits=1))%" for (i, r) in enumerate(["bedroom", "living room", "bathroom"])], ", "),
" (average $(round(100*mean(dirtiness_random), digits=1))%)")
println("random Roomba, coverage: ",
join(["$(r): $(round(100*coverage_random[i], digits=1))%" for (i, r) in enumerate(["bedroom", "living room", "bathroom"])], ", "))random Roomba, disturbance: 15.8% of ticks in the person's room
random Roomba, dirty: bedroom: 8.7%, living room: 20.7%, bathroom: 3.2% (average 10.8%)
random Roomba, coverage: bedroom: 22.5%, living room: 25.8%, bathroom: 51.7%Changes in the random Roomba’s reference values compared to Part 4.
disturbance_random, dirtiness_random and coverage_random, so that they survive the agent’s run in 4.6, which overwrites the recorded states. This cell replaces the separate reference-value cell of Part 4.The uncertainty in the environment is captured in the tuple of state transition tensors \(\mathbf{B}^{\ast} = \big(\boldsymbol{B}^{\ast(0)}, \ldots, \boldsymbol{B}^{\ast(4)}\big)\), one per state factor, and the tuple of observation likelihood tensors \(\mathbf{A}^{\ast} = \big(\boldsymbol{A}^{\ast(0)}, \boldsymbol{A}^{\ast(1)}, \boldsymbol{A}^{\ast(2)}\big)\), for surface, commotion and pickup. Their entries are the probabilities of the Roomba’s moves, including moves that fail, of the person’s movements, of dirt accumulating and being cleaned, and of the readings of the camera, the motion sensor and the dirt sensor. With several factors, the uncertainty is spread over several small tensors, each describing one aspect of the environment. The actions themselves add no uncertainty: the agent knows which action it chose, only not exactly what will come of it. The initial state adds no uncertainty either, because the run always starts from the same state: the Roomba in the bedroom, the person in the living room, and all rooms clean.
On the agent’s side, there is no uncertainty about how the environment works: in this part, the agent is given \(\mathbf{A} = \mathbf{A}^{\ast}\) and \(\mathbf{B} = \mathbf{B}^{\ast}\). What remains is uncertainty about the state: where the Roomba is, where the person is, and which rooms are dirty, and, when planning, where its actions will lead, whether it will meet the person there, and whether there will be dirt to clean. The uncertainty differs per factor. The whereabouts is well tracked, since the agent knows where it was heading and the camera confirms it. The person is only noticed when they are in the whereabouts. And the dirt in other rooms is never observed at all: the agent can only predict it, so its uncertainty about those rooms grows the longer the Roomba stays away. The agent handles all this uncertainty with its beliefs, and takes it into account when choosing actions.
Changes in the uncertainty model compared to Part 4.
On the agent’s side, the state of the environment at time \(t\) is not known: the agent represents it by a (compound) tuple of categorical random variables, one per state factor (here five factors):
\[ \mathbf{s}^{(t)} = \big(s^{(t,0)}, s^{(t,1)}, s^{(t,2)}, s^{(t,3)}, s^{(t,4)}\big) \]
where the state factors are:
The agent never observes these variables directly. What it holds instead is a belief. Internally, this is the joint belief \(\boldsymbol{q}^{(t)} \in [0,1]^{96}\), with one probability per combination of the five factors. It is convenient to arrange it as an array with one axis per factor, of size \(3 \times 4 \times 2 \times 2 \times 2\), in the factor order used for the axes of the tensors in 4.1. From it, the agent’s belief about a single factor, the marginal \(\boldsymbol{s}^{(t,n)}\), follows by adding up over all other axes. For example, \(\boldsymbol{s}^{(t,1)} \in [0,1]^{4}\) is the agent’s belief about where the person is.
The agent needs the joint belief, rather than five separate marginals, because its sensors tie the factors together. When the motion sensor detects motion, the agent learns little about the whereabouts or the occupant on their own, but a lot about the two together: they are probably in the same room. Likewise, when the dirt sensor picks up dirt, the agent learns that whichever room the Roomba is in is dirty. Only the joint belief can hold such statements; the marginals lose them, as noted in 4.1.
The initial state prior is given by
\[ P_D\big(\mathbf{s}^{(0)}\big) = \prod_{n=0}^{4} \mathcal{Cat}\big(s^{(0,n)};\, \boldsymbol{D}^{(n)}\big) \]
where \(\boldsymbol{D}^{(n)}\) parameterizes the categorical distribution of \(s^{(0,n)}\). Unlike the environment, the agent does not know the starting state, so each factor gets a uniform prior: \(\boldsymbol{D}^{(0)} = [\tfrac{1}{3}, \tfrac{1}{3}, \tfrac{1}{3}]^{\top}\) for the room, \(\boldsymbol{D}^{(1)} = [\tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4}]^{\top}\) for the person, and \(\boldsymbol{D}^{(n)} = [\tfrac{1}{2}, \tfrac{1}{2}]^{\top}\) for the dirt in each room, \(n = 2, 3, 4\). Together, they give the uniform joint prior over all 96 combinations, \(\tfrac{1}{96}\) each. The prior is the only point at which the factors are independent: once the first observations arrive, the joint belief in general no longer splits into a product of its marginals.
The (compound) observation tuple received by the agent at time \(t\) is the same one the environment generates:
\[ \hat{\mathbf{o}}^{(t)} = \big(\hat{\boldsymbol{o}}^{(t,0)}, \hat{\boldsymbol{o}}^{(t,1)}, \hat{\boldsymbol{o}}^{(t,2)}\big) \]
where the observation modalities are:
As in Part 4, the agent also chooses actions, and it prefers some observations over others. These are described in 4.5.2 and 4.5.3.
Changes in the agent’s state and observation variables compared to Part 4.
As in Part 4, the agent makes decisions. At each time step, it chooses an action: the room the Roomba heads for. The agent represents its action at time \(t\) by a (compound) tuple of categorical random variables, one per control factor (here a single factor):
\[ \mathbf{u}^{(t)} = \big(u^{(t,0)}\big) \]
where the control factors are:
Heading for the room the Roomba is already in means staying put. Unlike the state, the agent always knows which action it has taken: once chosen, the action is a realized value \(\hat{\boldsymbol{u}}^{(t,0)}\), which the agent hands to the environment and also uses itself, to predict how the state will change. What the agent does not know is the outcome: whether the move succeeds, whether the person is in the room it reaches, and whether there is dirt there to clean.
The action selects the transition matrix of the whereabouts only. Its effect on the other factors runs through their dependencies: the dirt factors depend on where the Roomba is, so in its predictions the agent’s choice of room also determines which room gets cleaned. The person’s movements, on the other hand, are predicted the same way whatever the agent does.
Policies. To choose an action, the agent looks \(H\) steps ahead. It considers candidate policies, i.e. sequences of \(H\) actions, collected as the rows of a matrix \(\boldsymbol{V}\): policy \(p\) is \(\boldsymbol{V}^{[p]}\), and \(V^{[p,\tau]}\) is the action it prescribes at step \(\tau = 0, \ldots, H-1\). With three possible actions per step, there are \(3^H\) policies. In this part, \(H = 3\), which gives 27 policies. Three steps are needed for cleaning to pay off: from the bedroom, reaching the living room takes two moves, through the bathroom, and the dirt is picked up and removed only once the Roomba is there. A policy such as head for the bathroom, then the living room, then stay lets the agent see that pay-off. The agent scores each policy by its expected free energy (see 4.5.6), turns the scores into a probability for each policy, and from those derives a probability for each possible next action. Only the first action of the chosen policy is carried out; at the next time step, the agent observes the result and plans again.
Changes in the agent’s decision variables compared to Part 4.
Goals are preferences over observations. The agent prefers to receive the observations that go with the situation it wants. There is one log-preference vector per modality, \(\log\boldsymbol{C}^{(m)}\), collected in the tuple \(\mathbf{C}\). These are log-preferences, given directly, not the logarithm of a distribution that is then normalized. Adding a constant to every entry of \(\log\boldsymbol{C}^{(m)}\), which is what normalizing would do, adds the same constant to every \(G_{\Sigma}^{[p]}\) and changes no decision.
The agent cannot prefer a hidden state directly, because it never observes the occupant or the dirt factors. It prefers the observations that go with what it wants, and chooses actions it expects to produce them.
For cleaning, the observation has to be chosen with care. Preferring “nothing picked up” would backfire: the agent would be drawn to rooms that are already clean, where it has nothing to do. Preferring “dirt picked up” draws it to dirty rooms, which are the rooms where being there changes the state. Once a room is clean, that preference is no longer met there, and the agent is pulled toward wherever dirt is most likely to have built up: the living room, which gets dirty fastest, and rooms it has not visited for a while.
The ratio of the two log-preferences sets the balance. Here, “motion” counts more against the agent than “dirt picked up” counts for it, so the agent should clean the living room when it expects the room to be quiet, rather than going wherever dirt is likely. A zero \(\log\boldsymbol{C}^{(2)}\) is the Part 4 agent, the motion-only baseline in 4.6. There is no habit vector \(\boldsymbol{E}\). Nothing in this part prices the number of moves.
The zero preference on surface does not make the camera useless. The camera still tells the agent the whereabouts, which it needs in order to predict where a heading leads, and which room’s dirt pickup is sensing. It does not make one room more attractive than another.
Changes in the preferences compared to Part 4.
The state transition functions, one per state factor, are given by
\[ \begin{aligned} P_{B^{(0)}}\big(s^{(t,0)} \mid s^{(t-1,0)}, u^{(t-1,0)}\big) &= \mathcal{Cat}\big(s^{(t,0)};\, \boldsymbol{B}^{(0)}[\bullet, s^{(t-1,0)}, u^{(t-1,0)}]\big) && \text{ROOM}\\ P_{B^{(1)}}\big(s^{(t,1)} \mid s^{(t-1,1)}\big) &= \mathcal{Cat}\big(s^{(t,1)};\, \boldsymbol{B}^{(1)}[\bullet, s^{(t-1,1)}]\big) && \text{PERSON}\\ P_{B^{(n)}}\big(s^{(t,n)} \mid s^{(t-1,n)}, s^{(t-1,0)}\big) &= \mathcal{Cat}\big(s^{(t,n)};\, \boldsymbol{B}^{(n)}[\bullet, s^{(t-1,n)}, s^{(t-1,0)}]\big), \quad n = 2, 3, 4 && \text{DIRT} \end{aligned} \]
where, as in 4.3, the values of the random variables are used as indices into the tensors. Each line picks out the column of \(\boldsymbol{B}^{(n)}\) for the previous values of exactly the factors in \(\mathcal{D}_B^{(n)}\), and, for the whereabouts, the action chosen at \(t-1\). Given the previous state and action, the factors change independently of each other, so the transition of the whole state is the product
\[ P_{\mathbf{B}}\big(\mathbf{s}^{(t)} \mid \mathbf{s}^{(t-1)}, u^{(t-1,0)}\big) = \prod_{n=0}^{4} P_{B^{(n)}}\big(s^{(t,n)} \mid \ldots\big) \]
where the dots stand for the conditioning variables of each line above.
The observation generation functions, one per sensor, are given by
\[ \begin{aligned} P_{A^{(0)}}\big(o^{(t,0)} \mid s^{(t,0)}\big) &= \mathcal{Cat}\big(o^{(t,0)};\, \boldsymbol{A}^{(0)}[\bullet, s^{(t,0)}]\big) && \text{TEXTURE}\\ P_{A^{(1)}}\big(o^{(t,1)} \mid s^{(t,0)}, s^{(t,1)}\big) &= \mathcal{Cat}\big(o^{(t,1)};\, \boldsymbol{A}^{(1)}[\bullet, s^{(t,0)}, s^{(t,1)}]\big) && \text{MOTION}\\ P_{A^{(2)}}\big(o^{(t,2)} \mid s^{(t,0)}, s^{(t,2)}, s^{(t,3)}, s^{(t,4)}\big) &= \mathcal{Cat}\big(o^{(t,2)};\, \boldsymbol{A}^{(2)}[\bullet, s^{(t,0)}, s^{(t,2)}, s^{(t,3)}, s^{(t,4)}]\big) && \text{DIRT SENSOR} \end{aligned} \]
where each line picks out the column of \(\boldsymbol{A}^{(m)}\) for the current values of exactly the factors in \(\mathcal{D}_A^{(m)}\). It parameterizes the categorical distribution of \(o^{(t,m)}\): over surface readings for \(m = 0\), over motion readings for \(m = 1\), and over dirt readings for \(m = 2\). Observations do not depend on the action.
In this part, the agent is given its tensors, \(\boldsymbol{A}^{(m)} = \boldsymbol{A}^{\ast(m)}\) and \(\boldsymbol{B}^{(n)} = \boldsymbol{B}^{\ast(n)}\), and it knows their dependencies, which are the same as the environment’s. The tensors are known parameters, not random variables, so they appear as subscripts.
An entry of \(\boldsymbol{A}^{(m)}\) is the probability of a specific observation given specific values of the factors it depends on, and each column, obtained by fixing all axes except the first, is a categorical distribution over observations. The same holds for each \(\boldsymbol{B}^{(n)}\), with the next value of factor \(n\) in place of the observation. If the state were known, the right column could simply be looked up. The agent, however, only has its joint belief \(\boldsymbol{q}^{(t)}\), so it takes the weighted average of the columns over all 96 combinations, each weighted by its probability under the belief. To write this down, let \(\mathbf{j} = (j_0, \ldots, j_4)\) denote one combination of factor values, so that \(\boldsymbol{q}^{(t)}[\mathbf{j}]\) is its probability, and let \(j_{\mathcal{D}}\) denote the values of the factors in a dependency set \(\mathcal{D}\), in order. For example, \(j_{\mathcal{D}_A^{(1)}} = (j_0, j_1)\), the whereabouts and the occupant.
Inference. After carrying out action \(\hat{u}^{(t-1,0)}\), the agent first predicts the new state, by moving the probability of each combination \(\mathbf{i}\) in its previous belief to the combinations \(\mathbf{j}\) it can lead to:
\[ \sum_{\mathbf{i}} P_{\mathbf{B}}\big(\mathbf{j} \mid \mathbf{i}, \hat{u}^{(t-1,0)}\big)\,\boldsymbol{q}^{(t-1)}[\mathbf{i}] \]
Then it updates this prediction with the observations it receives. Given the state, the three sensors are independent of each other, so it combines them by multiplying their likelihoods:
\[ \boldsymbol{q}^{(t)}[\mathbf{j}] \;\propto\; \Big(\prod_{m=0}^{2} \boldsymbol{A}^{(m)}\big[\hat{o}^{(t,m)}, j_{\mathcal{D}_A^{(m)}}\big]\Big) \cdot \sum_{\mathbf{i}} P_{\mathbf{B}}\big(\mathbf{j} \mid \mathbf{i}, \hat{u}^{(t-1,0)}\big)\,\boldsymbol{q}^{(t-1)}[\mathbf{i}] \]
where \(\hat{o}^{(t,m)}\), the received observation without bold, is used as an index, and the result is normalized to sum to 1 over all 96 combinations. At \(t = 0\), the prediction is replaced by the uniform initial prior from 4.5.1. A combination that explains all readings well gains belief, while one that explains only some of them loses some. Each sensor contributes differently:
This is exact Bayesian filtering over the 96 joint states, which is feasible because the number of combinations is small.
Planning. To evaluate policy \(p\), the agent applies the same functions to the future, step by step, starting from its current belief. Its predicted joint beliefs are
\[ \boldsymbol{q}^{(0)}_{\pi^{[p]}}[\mathbf{j}] = \sum_{\mathbf{i}} P_{\mathbf{B}}\big(\mathbf{j} \mid \mathbf{i}, V^{[p,0]}\big)\,\boldsymbol{q}^{(t)}[\mathbf{i}], \qquad \boldsymbol{q}^{(\tau)}_{\pi^{[p]}}[\mathbf{j}] = \sum_{\mathbf{i}} P_{\mathbf{B}}\big(\mathbf{j} \mid \mathbf{i}, V^{[p,\tau]}\big)\,\boldsymbol{q}^{(\tau-1)}_{\pi^{[p]}}[\mathbf{i}] \quad (\tau > 0) \]
and the observations it then expects are
\[ \boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}} = \sum_{\mathbf{j}} \boldsymbol{A}^{(m)}\big[\bullet, j_{\mathcal{D}_A^{(m)}}\big]\,\boldsymbol{q}^{(\tau)}_{\pi^{[p]}}[\mathbf{j}] \]
Here \(\boldsymbol{q}^{(\tau)}_{\pi^{[p]}}\) is the predicted joint belief about time \(t + \tau + 1\) if the agent follows policy \(p\), and \(\boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}}\) is the observation of sensor \(m\) it then expects. Unlike the example in 4.1, which treated the room and the person as independent, the expected observation is computed from the joint belief, so it also accounts for any relation between the factors a sensor depends on. No real observations are available for the future, so these predictions are never updated; they are what the agent compares with its preferences when it scores policy \(p\) (see 4.5.6).
The two preferences make the corresponding predictions the ones that matter most:
Each policy is predicted over all 96 joint states, but with 27 policies and three steps this remains a small computation.
Changes in the agent’s transition and observation functions compared to Part 4.
We start by specifying a probabilistic model for the agent that describes the agent’s internal beliefs about the external dynamics of the environment. As in Part 4, this model has two parts. Up to the current time \(t\), it describes what has happened: the states the environment went through, the observations the Roomba received, and the actions \(\hat{u}^{(0:t-1,0)}\) the agent actually took. Beyond \(t\), it describes what could happen over the next \(H\) steps, depending on which policy \(\pi^{(t)}\) the agent chooses. With several factors and sensors, we write the model in terms of the tuples \(\mathbf{s}^{(t')}\) and \(\mathbf{o}^{(t')}\). The generative model at time \(t\) is defined as follows:
\[ \begin{aligned} &P\big(\mathbf{o}^{(0:t+H)}, \mathbf{s}^{(0:t+H)}, \pi^{(t)} \mid \hat{u}^{(0:t-1,0)}\big) \\[4pt] &\quad= \underbrace{P\big(\pi^{(t)}\big)}_{\substack{\text{policy}\\ \text{prior}}} \,\underbrace{P_D\big(\mathbf{s}^{(0)}\big)}_{\substack{\text{state}\\ \text{prior}}} \prod_{t'=0}^{t+H} \underbrace{P_{\mathbf{A}}\big(\mathbf{o}^{(t')} \mid \mathbf{s}^{(t')}\big)}_{\substack{\text{observation-generation}\\ \text{model}}} \\[4pt] &\qquad\times \underbrace{\prod_{t'=1}^{t} P_{\mathbf{B}}\big(\mathbf{s}^{(t')} \mid \mathbf{s}^{(t'-1)}, \hat{u}^{(t'-1,0)}\big)}_{\text{past: actions taken}} \;\underbrace{\prod_{\tau=0}^{H-1} P_{\mathbf{B}}\big(\mathbf{s}^{(t+\tau+1)} \mid \mathbf{s}^{(t+\tau)}, V^{[\pi^{(t)},\tau]}\big)}_{\text{future: actions under policy } \pi^{(t)}} \end{aligned} \]
where \(P_D\) is the agent’s uniform initial state prior from 4.5.1, and \(P_{\mathbf{A}}\) and \(P_{\mathbf{B}}\) combine the functions from 4.5.4, one per sensor and one per factor:
\[ P_{\mathbf{A}}\big(\mathbf{o}^{(t')} \mid \mathbf{s}^{(t')}\big) = \prod_{m=0}^{2} P_{A^{(m)}}\big(o^{(t',m)} \mid \ldots\big), \qquad P_{\mathbf{B}}\big(\mathbf{s}^{(t')} \mid \mathbf{s}^{(t'-1)}, u\big) = \prod_{n=0}^{4} P_{B^{(n)}}\big(s^{(t',n)} \mid \ldots\big) \]
with, as in 4.5.4, the dots standing for exactly the factors in \(\mathcal{D}_A^{(m)}\) and \(\mathcal{D}_B^{(n)}\), and, for the whereabouts, the action \(u\). The tensors \(\mathbf{A} = \mathbf{A}^{\ast}\) and \(\mathbf{B} = \mathbf{B}^{\ast}\) and the prior \(\mathbf{D}\) are known parameters, so they appear as subscripts. As in Part 4, there are no parameter priors: the agent learns nothing about its tensors in this part.
Written out, the model is a product of many small pieces, each involving only a few variables: one prior per factor, one observation function per sensor and time step, and one transition function per factor and time step. This is what a factorized model means, and it is why each tensor can stay small. The action appears in only one of these pieces, the transition of the whereabouts; its influence on the dirt runs through the dirt transitions’ dependence on the Roomba’s previous room.
The two transition products differ only in where the action comes from. In the past, the action is the one the agent actually took, \(\hat{u}^{(t'-1,0)}\), a known value. In the future, it is the action \(V^{[\pi^{(t)},\tau]}\) that the policy prescribes at step \(\tau\); since the policy is not yet chosen, it is a random variable. Step \(\tau\) predicts time \(t + \tau + 1\), as in 4.5.4.
The policy prior \(P\big(\pi^{(t)}\big)\) is where the agent’s preferences enter the model. In active inference, it is not chosen freely: policies are a priori more probable the lower their expected free energy,
\[ P(\pi^{(t)}=p)=\sigma(-\boldsymbol{G}_{\Sigma})^{[p]} \]
where \(\boldsymbol{G}_{\Sigma}\) collects the expected free energies \(G_{\Sigma}^{[p]}\) of the 27 policies, which depend on both preferences in \(\mathbf{C}\) from 4.5.3 (see 4.5.6), and \(\sigma\) is the softmax. The agent thus expects to act in ways that lead to its preferred observations: no motion, and dirt picked up.
The product inside \(P_{\mathbf{A}}\) is where the three sensors are combined. Each sensor depends on its own subset of the factors, and given the whole state, the three readings are independent. For the future, these observations are not yet available: the agent uses their expected values, \(\boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}}\) from 4.5.4, to evaluate each policy.
Note that the factorization is a property of the model, not of the agent’s beliefs. The model is a product of small pieces, but once observations come in, the posterior over the state no longer factorizes, as explained in 4.5.1. This is why the agent works with the joint belief over all 96 combinations. In larger models, where the joint states become too many, the belief itself is approximated by a product of per-factor beliefs. That approximation is left for a later part.
Changes in the generative model compared to Part 4.
Now it is time to build our agent. As in Part 4, the agent does not learn anything: it is given its model of the world, i.e. the observation tensors \(\mathbf{A} = \mathbf{A}^{\ast}\) with their dependencies \(\mathcal{D}_A^{(m)}\), the transition tensors \(\mathbf{B} = \mathbf{B}^{\ast}\) with their dependencies \(\mathcal{D}_B^{(n)}\), the uniform initial priors \(\mathbf{D}\) from 4.5.1, and its preferences \(\mathbf{C}\) from 4.5.3. What it has to do is use this model, at every tick, in three steps.
1. Update its belief (perception). After the environment has carried out the previous action \(\hat{u}^{(t-1,0)}\) and returned the new observations \(\hat{\mathbf{o}}^{(t)}\), the agent first predicts the new joint state, then corrects that prediction with what its sensors report, as in 4.5.4:
\[ \boldsymbol{q}^{(t)}[\mathbf{j}] \;\propto\; \Big(\prod_{m=0}^{2} \boldsymbol{A}^{(m)}\big[\hat{o}^{(t,m)}, j_{\mathcal{D}_A^{(m)}}\big]\Big) \cdot \sum_{\mathbf{i}} P_{\mathbf{B}}\big(\mathbf{j} \mid \mathbf{i}, \hat{u}^{(t-1,0)}\big)\,\boldsymbol{q}^{(t-1)}[\mathbf{i}] \]
With the joint belief stored as a \(3 \times 4 \times 2 \times 2 \times 2\) array, both parts are array operations:
The result is normalized to sum to 1. At \(t = 0\), the prediction is replaced by the uniform joint prior, the product of the \(\boldsymbol{D}^{(n)}\). With known tensors and only 96 joint states, this is exact Bayesian filtering: no variational approximation is needed. For recording and evaluation, the agent also computes the marginals \(\boldsymbol{s}^{(t,n)}\) from the joint belief, by adding up over the other axes.
2. Evaluate the policies (planning). For each of the \(3^H = 27\) policies \(\pi^{[p]}\), the agent predicts its joint beliefs and expected observations over the next \(H = 3\) steps, as in 4.5.4, and scores the policy by its expected free energy \(G_{\Sigma}^{[p]}\) (see 4.5.6). The score favors policies that are expected to lead to preferred observations, i.e. “no motion” and “dirt picked up”, and policies that are expected to reduce the agent’s uncertainty about the state, e.g. by going back to check on a room it has not seen for a while.
3. Choose an action (decision). The scores are turned into a posterior over policies, \(Q(\pi^{(t)}=p)=\sigma(-\boldsymbol{G}_{\Sigma})^{[p]}\), and from it into a distribution over the next action, \(Q\big(u^{(t,0)}\big)\), by adding up the probabilities of all policies that start with that action. The agent then carries out the most probable action, \(\hat{u}^{(t,0)}\), and the cycle repeats at the next tick.
The model is larger than in Part 4, with 96 joint states instead of six, and 27 policies instead of 9, but still small enough for all three steps to be computed directly in Julia, without RxInfer. As in create_envir, the code is generic over factors and modalities: the dependency tuples LdepsA and LdepsB tell it which axes of the joint belief each tensor acts on, so the same few functions handle every factor and sensor. This keeps every quantity of Algorithm 18 visible in the code. RxInfer also supports active inference, as planning formulated as inference, which becomes attractive for larger models, where neither the joint states nor the policies can be enumerated; this is a possible direction for a later part. Let’s build the agent!
Changes in the agent’s three steps compared to Part 4.
create_envir: the dependency tuples tell each function which axes of the joint belief a tensor acts on.## ---------- the agent's model: given, not learned ----------
LA = LAˣ ## A^{(m)} = A^{*(m)}, m = 0, 1, 2 (dependencies D_A^{(m)} in LdepsA)
LB = LBˣ ## B^{(n)} = B^{*(n)}, n = 0, ..., 4 (dependencies D_B^{(n)} in LdepsB)
LD = tuple0(fill(1/3, 3), ## D^{(0)}: site, uniform
fill(1/4, 4), ## D^{(1)}: occupant, uniform
fill(1/2, 2), fill(1/2, 2), fill(1/2, 2)) ## D^{(2)}, D^{(3)}, D^{(4)}: pickup, uniform
LC = tuple0(zeros(3), ## log C^{(0)} = 0: texture is for perception only
log.([0.9, 0.1]), ## log C^{(1)}: prefers "no motion"
log.([0.2, 0.8])) ## log C^{(2)}: prefers "dirt picked up"
LC_motion = tuple0(zeros(3), log.([0.9, 0.1]), zeros(2)) ## Part 4 agent: log C^{(2)} = 0
## ---------- the joint state: an array with one axis per factor ----------
sizes = Tuple(size(LB[n], 1) for n in eachindex(LB)) ## number of values per factor: (3, 4, 2, 2, 2), 96 combinations
## ---------- policies: all action sequences of length H ----------
H = 3 ## planning horizon
_V = [p[τ] for p in vec(collect(Iterators.product(ntuple(_ -> 1:3, H)...))), τ in 1:H] ## P×H; column τ+1 is step τ
P = size(_V, 1) ## number of policies, 3^H = 27
## ---------- helpers ----------
normalize(v) = v ./ sum(v)
log_stable(x) = log.(max.(x, 1e-16)) ## avoids log(0)
softmax(x) = normalize(exp.(x .- maximum(x)))
## the marginal of the joint belief over the factors in deps, with its axes in the order of deps
function marginal(_q, deps)
others = Tuple(k for k in 1:ndims(_q) if (k-1) ∉ deps) ## axes to add up over (axis k is factor k-1)
m = dropdims(sum(_q, dims=others), dims=others) ## axes now in increasing factor order
permutedims(m, invperm(sortperm(deps))) ## reorder to the order of deps
end
marginals(_q) = tuple0((marginal(_q, [n]) for n in eachindex(LB))...) ## (s^{(t,0)}, ..., s^{(t,4)}), for recording
## an array over the factors in deps (axes in the order of deps), spread out over the joint state's other axes
function spread(arr, deps)
a = permutedims(arr, sortperm(deps)) ## axes in increasing factor order
reshape(a, ntuple(k -> (k-1) in deps ? sizes[k] : 1, length(sizes))) ## size 1 on the other axes: broadcasts
end
_D = reduce((a, b) -> a .* b, (spread(LD[n], [n]) for n in eachindex(LD))) ## joint prior: ∏_n D^{(n)}, 1/96 each
## H[A^{(m)}]: entropy of each column of A^{(m)}, an array over the factors in D_A^{(m)}
LH = tuple0((dropdims(-sum(LA[m] .* log_stable(LA[m]), dims=1), dims=1) for m in eachindex(LA))...)
## ---------- prediction: Σ_i P_B(j | i, u) s[i], one factor at a time ----------
## each factor must use the *previous* values of the factors it depends on, so a factor is updated
## before the factors it depends on: the dirt factors (which depend on the Roomba previous room)
## before the site
update_order = [bath_soil_n, lounge_grime_n, bed_dust_n, occupant_n, whereabouts_n]
@assert all(findfirst(==(n), update_order) < findfirst(==(d), update_order)
for n in eachindex(LB) for d in LdepsB[n] if d != n)
## moves the probability of each combination along the axis of factor n, using B^{(n)}
function transition(_q, n, u)
_q′ = zero(_q)
action = n == whereabouts_n ? (u,) : () ## only factor 0 (site) has an action axis
for I in CartesianIndices(_q)
p = _q[I]
p == 0 && continue
i = Tuple(I) ## the combination (1-based values of all factors)
col = view(LB[n], :, (i[d+1] for d in LdepsB[n])..., action...) ## B^{(n)}[•, values of D_B^{(n)}, (u)]
for (j, b) in enumerate(col)
_q′[Base.setindex(i, j, n+1)...] += b * p ## factor n takes value j, the others keep theirs
end
end
_q′
end
function predict(_q, u)
for n in update_order
_q = transition(_q, n, u)
end
_q
end
## ---------- step 1: perception (exact Bayesian filtering over the joint states) ----------
## q^{(t)}[j] ∝ (∏_m A^{(m)}[ô^{(t,m)}, j_{D_A^{(m)}}]) · Σ_i P_B(j | i, û^{(t-1,0)}) q^{(t-1)}[i]; at t = 0 the prediction is the joint prior
function perceive(_qₜ₋₁, Lôₜ, ûₜ₋₁)
prediction = _qₜ₋₁ === nothing ? _D : predict(_qₜ₋₁, ûₜ₋₁)
likelihood = reduce((a, b) -> a .* b,
(spread(selectdim(LA[m], 1, argmax(Lôₜ[m])), LdepsA[m]) for m in eachindex(LA))) ## the slices A^{(m)}[ô^{(t,m)}, ...], spread out
normalize(likelihood .* prediction) ## _qₜ = q^{(t)}
end
## ---------- step 2: expected free energy of policy p ----------
## G_{\Sigma}^{[p]} = Σ_τ Σ_m [ o^{(τ,m)}_{π^{[p]}}·(log o^{(τ,m)}_{π^{[p]}} - log C^{(m)}) + Σ_j H[A^{(m)}][j_{D_A^{(m)}}] q^{(τ)}_{π^{[p]}}[j] ]
## (risk: expected vs. preferred observations) (ambiguity: expected sensor noise)
function efe(_qₜ, p, LC)
_q = _qₜ
GΣ = 0.0 ## G_Σ^{[p]}
for τ in 1:H ## code τ = math τ + 1
_q = predict(_q, _V[p, τ]) ## q^{(τ)}_{π^{[p]}}: predicted joint belief
for m in eachindex(LA)
q = marginal(_q, LdepsA[m]) ## predicted belief over the factors sensor m depends on
o = reshape(LA[m], size(LA[m], 1), :) * vec(q) ## o^{(τ,m)}_{π^{[p]}}: expected observation
GΣ += o' * (log_stable(o) - LC[m]) + sum(LH[m] .* q)
end
end
GΣ
end
## ---------- step 3: from policy scores to an action ----------
function plan(_qₜ, LC)
_GΣ = [efe(_qₜ, p, LC) for p in 1:P] ## G_{\Sigma}^{[p]}, one per policy
_qπ = softmax(-_GΣ) ## (9.76) ## Q(π^{(t)}=p) = σ(-G_Σ)^{[p]}
_qu = [sum(_qπ[_V[:, 1] .== u]) for u in 1:3] ## (9.69) Q(u^{(t,0)}): add up policies by their first action
(_GΣ, _qπ, _qu)
end
choose(_qu) = argmax(_qu) ## û^{(t,0)}: the most probable action (1 bedroom, 2 living room, 3 bathroom)choose (generic function with 1 method)Changes in the agent code compared to Part 4.
marginal adds it up to the factors a tensor depends on, and spread broadcasts an array over a few factors across the full joint shape.transition, predict), without building the 96 × 96 transition matrix. A factor is updated before the factors it depends on, so that it uses their previous values: the dirt factors before the whereabouts. An @assert checks this order against LdepsB._D, the product of the per-factor priors in LD.LC with both preferences, and LC_motion for the Part 4 agent with only the preference for no motion. efe and plan take the preferences as an argument, so the same code serves both agents.Notes on the agent code.
The code mirrors the agent’s math.
LA, LB, LC and LD are tuples, upright bold \(\mathbf{A}\), \(\mathbf{B}\), \(\mathbf{C}\) and \(\mathbf{D}\), built with tuple0. LA[m] is \(\boldsymbol{A}^{(m)}\), LB[n] is \(\boldsymbol{B}^{(n)}\), LC[m] is \(\log\boldsymbol{C}^{(m)}\), and LD[n] is \(\boldsymbol{D}^{(n)}\). The agent is given its model, so LA and LB are LAˣ and LBˣ without the asterisk, and the dependencies are LdepsA and LdepsB. The named members _Aˣı0ı and _Bˣı0ı are the environment tensors. Indexing the tuple returns the same array.LC stores log-preferences, already logged. efe subtracts LC[m]. It does not call log on it again. LC_motion is the Part 4 agent: \(\log\boldsymbol{C}^{(2)}=\boldsymbol{0}\)._qₜ and _qₜ₋₁ are the joint beliefs \(\boldsymbol{q}^{(t)}\) and \(\boldsymbol{q}^{(t-1)}\), arrays of size \(3\times 4\times 2\times 2\times 2\). Inside efe, _q is the predicted joint \(\boldsymbol{q}^{(\tau)}_{\pi^{[p]}}\). _D is the uniform joint prior, the product of the factors in LD.marginals(_qₜ) returns Lsₜ, the marginals \(\big(\boldsymbol{s}^{(t,0)},\ldots,\boldsymbol{s}^{(t,4)}\big)\)._V is \(\boldsymbol{V}\). _GΣ is \(\boldsymbol{G}_{\Sigma}\), not the matrix \(\boldsymbol{G}\). efe adds the steps as it goes, so the per-step matrix is not stored. _qπ is \(\boldsymbol{\pi}\). _qu is \(Q(u^{(t,0)})\).LH holds \(\boldsymbol{H}^{(m)}\), the entropy of each column of \(\boldsymbol{A}^{(m)}\), an array over the factors in \(\mathcal{D}_A^{(m)}\). It is computed once.Helpers. A tensor depends on a few factors. The joint has an axis for every factor.
marginal(_q, deps) adds the joint up over factors that are not in deps, and orders the axes as deps orders them.spread(arr, deps) broadcasts an array over those factors out to the joint shape, with size 1 on the other axes.One function per step.
transition and predict compute \(\sum_{\mathbf{i}} P_{\mathbf{B}}(\mathbf{j}\mid\mathbf{i}, u)\,\boldsymbol{q}[\mathbf{i}]\), one factor at a time. A factor is updated before the factors it depends on, so it sees their previous values: the dirt factors before the whereabouts. That order is update_order. The @assert checks it against LdepsB. In the other order, dirt would be cleaned in the room the Roomba is moving to, rather than the room it was in.perceive is perception. It predicts with the previous heading, multiplies by the three likelihood slices for the received observations, and normalizes. At \(t=0\) it starts from _D. This is exact filtering over the 96 joint states.efe is the expected free energy of one policy, \(G_{\Sigma}^{[p]}\), risk plus ambiguity, summed over the steps and the three modalities. The formula is in 4.5.6.plan and choose turn the scores into \(\boldsymbol{\pi}\) and \(Q(u^{(t,0)})\), and choose picks the mode. efe and plan take the log-preferences as an argument, so the same code runs LC and LC_motion.Tuples are 0-based. Inside the joint array, Julia’s indices are 1-based: factor \(n\) is axis \(n+1\), which is why the code reads i[d+1]. The math step \(\tau=0,\ldots,H-1\) is column \(\tau+1\) of _V.
This part does not use RxInfer. With known tensors and 96 joint states, the three steps are a few dozen lines, and each quantity in the planning recursion stays visible.
Next, we define the agent and the time-stepping procedure.
## LC: the agent's preferences, e.g. LC (both goals) or LC_motion (the Part 4 agent)
function create_agent(; LC)
_qₜ₋₁ = nothing ## q^{(t-1)}: the previous joint belief (none before t = 0)
ûₜ₋₁ = nothing ## û^{(t-1,0)}: the previous action, as an index 1..3 (none before t = 0)
_qₜ = nothing ## q^{(t)}: the current joint belief, a 3×4×2×2×2 array
ûₜ = nothing ## û^{(t,0)}: the current action
_GΣ = _qπ = _qu = nothing ## the current plan: G, Q(π^{(t)}), Q(u^{(t,0)})
## step 1 and 2: update the joint belief with the new observations, then score all policies
compute = (Lôₜ) -> begin
_qₜ = perceive(_qₜ₋₁, Lôₜ, ûₜ₋₁)
_GΣ, _qπ, _qu = plan(_qₜ, LC)
nothing
end
## step 3: choose the next action, returned as a one-hot action tuple û^{(t)}
act = () -> begin
ûₜ = choose(_qu)
tuple0(Float64.(1:3 .== ûₜ))
end
## predicted beliefs under policy p, τ = 0, ..., H-1 (for inspection):
## each entry is the tuple of marginals of the predicted joint belief q^{(τ)}_{π^{[p]}}
future = (p) -> begin
_q = _qₜ
Lsπː = Any[]
for τ in 1:H
_q = predict(_q, _V[p, τ]) ## q^{(τ)}_{π^{[p]}}: predicted joint belief
push!(Lsπː, marginals(_q))
end
offset0(Lsπː)
end
## move one tick forward: the current belief and action become the previous ones
slide = () -> begin
_qₜ₋₁ = _qₜ
ûₜ₋₁ = ûₜ
nothing
end
belief = () -> marginals(_qₜ) ## (s^{(t,0)}, ..., s^{(t,4)}): the marginals, for recording
scores = () -> (_GΣ, _qπ, _qu) ## the current plan, for inspection
return (compute, act, future, slide, belief, scores)
endcreate_agent (generic function with 1 method)Changes in create_agent compared to Part 4.
create_agent(; LC), so the same code builds both the agent with two preferences and the Part 4 agent with only the preference for no motion._sₜ, rather than a tuple with a single belief. The joint belief is only used inside the agent.belief returns the marginals, one per factor, which is what is recorded and evaluated.future predicts joint beliefs with the factor-by-factor prediction, and returns their marginals at each step, so the predicted room, person location and dirt under a policy can be inspected.The agent scores each policy by the expected free energy of the observations it predicts. Let \(\boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}}\) be the observation of modality \(m\) expected at step \(\tau\) under policy \(p\), and let \(\boldsymbol{q}^{(\tau)}_{\pi^{[p]}}\) be the predicted joint belief from 4.5.4. Write \(\boldsymbol{q}^{(\tau,m)}_{\pi^{[p]}}\) for the marginal of that joint onto the factors in \(\mathcal{D}_A^{(m)}\). The per-step score, an entry of \(\boldsymbol{G}\in\mathbb{R}^{P\times H}\), is risk plus ambiguity,
\[ G^{[p,\tau]} = \sum_{m=0}^{2} \Big( \boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}} \cdot \big( \log\boldsymbol{o}^{(\tau,m)}_{\pi^{[p]}} - \log\boldsymbol{C}^{(m)} \big) + \boldsymbol{H}^{(m)} \cdot \boldsymbol{q}^{(\tau,m)}_{\pi^{[p]}} \Big). \]
The first term is the risk, how far the expected observation is from the log-preference. The second is the ambiguity, how noisy the sensor is expected to be. \(\boldsymbol{H}^{(m)}\) is the entropy of each column of \(\boldsymbol{A}^{(m)}\), so the ambiguity is large when the predicted state lands where the sensor is uninformative. Because \(\log\boldsymbol{C}^{(0)}=\boldsymbol{0}\), surface contributes only its entropy to the risk term, \(\boldsymbol{o}\cdot\log\boldsymbol{o}\), which does not favor one floor over another.
The code adds the steps inside efe and keeps the sum, not the matrix \(\boldsymbol{G}\):
\[ G_{\Sigma}^{[p]} = \sum_{\tau=0}^{H-1} G^{[p,\tau]}, \qquad \boldsymbol{G}_{\Sigma}\in\mathbb{R}^{P}. \]
There is no habit vector \(\boldsymbol{E}\) and no variational free energy \(\boldsymbol{F}_{\Sigma}\), because the belief is the exact filter, not a mean-field approximation. The policy posterior is
\[ Q\big(\pi^{(t)}\big) = \boldsymbol{\pi} = \sigma\big(-\boldsymbol{G}_{\Sigma}\big). \]
The posterior over the next heading adds that mass over the policies whose first action is \(u\),
\[ Q\big(u^{(t,0)}\big) = \sum_{p} \delta_{u = V^{[p,0]}}\, Q\big(\pi^{(t)}=p\big). \]
The agent carries out the mode, \(\hat{u}^{(t,0)}=\arg\max_{u} Q(u^{(t,0)})\). Only that first action is executed. At the next tick the agent observes the result and plans again.
Policies that share a prefix are scored separately. Scoring them on a tree would give the same \(G_{\Sigma}^{[p]}\), because the predicted belief after a shared prefix does not depend on which policy is being scored. With \(P=27\) the tree is not worth the extra code.
The two preferences make two of the predicted modalities matter most.
The Roomba is handed to the agent. The question is whether it cleans the apartment while staying out of the person’s way.
There is no separate inference pass on a finished log. The agent and the environment run together for \(T+1=600\) ticks. At each tick the environment reports \(\hat{\mathbf{o}}^{(t)}\), the agent updates \(\boldsymbol{q}^{(t)}\) and chooses \(\hat{\boldsymbol{u}}^{(t,0)}\) (compute, then act), the environment carries the heading out (execute), and the agent slides. The action chosen at \(t=T\) is not carried out. The loop in run_agent follows that order: for \(t>0\), execute the previous action, then state, observe, compute, and, for \(t<T\), act and slide.
The run records \(\hat{\mathbf{s}}^{\ast(0:T)}\), \(\hat{\mathbf{o}}^{(0:T)}\), \(\hat{\mathbf{u}}^{(0:T-1)}\) and the marginals \(\boldsymbol{s}^{(t,n)}\). From them the notebook computes \(J_{\text{disturb}}\), \(J_{\text{dirty}}^{(n)}\) and \(J_{\text{track}}^{(n)}\).
One run is plotted in detail, so the beliefs can be read tick by tick. A single run can be misleading, so the comparison with the baselines is then averaged over 30 seeds. The baselines are: always the heading “bedroom”, a fixed cycle through the three rooms, the heading chosen at random, and the Part 4 agent (LC_motion, only “no motion”). The agent of this part is LC, both preferences. The detailed plots use seed 42, the same environment seed as the random walk in 4.3, so the three detailed paths differ only in how the heading is chosen.
The beliefs are exact filtering over the 96 joint states. There is no free-energy iteration to plot. The expected behavior of the agent with both preferences is: leave a room soon after “motion”, stay while pickup is “dirt picked up”, visit the living room in short bursts when it expects the room to be quiet, and return to rooms it has not seen for a while.
Changes in the evaluation setup compared to Part 4.
The runs use simulated data from the generative process in 4.3.
Naive approach. This part has no separate naive estimator. The baselines in 4.2 are the comparison.
Active inference approach. The runs below use the agent of 4.5.
### Simulation parameters
## Time steps t = 0, ..., T (600 ticks)
T = 599
## Initial state: the Roomba in the bedroom, the person in the living room, all rooms clean
## factor 0 (site): 1 bedroom 2 living room 3 bathroom
## factor 1 (occupant): 1 bedroom 2 living room 3 bathroom 4 away
## factors 2–4 (Dirt in bedroom, living room, bathroom): 1 clean 2 dirty
Lŝˣ₀ = tuple0([1.0, 0.0, 0.0], ## ŝ^{*(0,0)}: Roomba in the bedroom
[0.0, 1.0, 0.0, 0.0], ## ŝ^{*(0,1)}: person in the living room
[1.0, 0.0], ## ŝ^{*(0,2)}: bedroom clean
[1.0, 0.0], ## ŝ^{*(0,3)}: living room clean
[1.0, 0.0]) ## ŝ^{*(0,4)}: bathroom clean → ŝ^{*(0)}5-element OffsetArray(::Vector{Vector{Float64}}, 0:4) with eltype Vector{Float64} with indices 0:4:
[1.0, 0.0, 0.0]
[0.0, 1.0, 0.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]Changes in the simulation parameters compared to Part 4.
Let’s take a look at our results. We compare three Roombas: the random Roomba from 4.3, the Part 4 agent with only the preference for “no motion”, and the agent with both preferences. If the preferences work, both agents should spend clearly less of their time in the person’s room than the random Roomba (a lower disturbance). The two agents should differ in cleanliness: the Part 4 agent is expected to neglect the living room, as in Part 4, so that room stays dirty much of the time, while the agent with both preferences should keep all rooms, and the living room in particular, cleaner. The price is likely some extra disturbance, since the living room is also where the person spends most time. Behind this behavior, the agents should keep good track of the state (high tracking accuracies), but not equally well for every factor: the whereabouts should be tracked well, the dirt in the Roomba’s own room too, but the occupant and the dirt in other rooms less so, since the agent only senses them when it is in the same room.
It is also worth looking at how the agent with both preferences behaves, not just how well. We expect it to:
Let’s see if it worked.
Changes in the expected results compared to Part 4.
## one run of an agent with preferences LC, in a fresh environment with the same seed as the random Roomba (4.3)
function run_agent(LC; seed=42)
(execute_ai, observe_ai, state_ai) = create_envir(; LAˣ, LBˣ, LdepsA, LdepsB, Lŝˣ₀, seed)
agent = create_agent(; LC)
(compute_ai, act_ai, future_ai, slide_ai, belief_ai, scores_ai) = agent
Lŝˣː = offset0(Vector{Any}(undef, T+1)) ## ŝ^{*(0:T)}: true states, each a tuple (Room, Person, Dirt ×3)
Lôː = offset0(Vector{Any}(undef, T+1)) ## ô^{(0:T)}: observations, each a tuple (Texture, Motion, Dirt)
Lûː = offset0(Vector{Any}(undef, T)) ## û^{(0:T-1)}: chosen actions, each a tuple (heading)
Lsː = offset0(Vector{Any}(undef, T+1)) ## s^{(0:T,•)}: the agent's marginal beliefs, each a tuple of five
Lquː = offset0(Vector{Any}(undef, T+1)) ## Q(u^{(t,0)}): the agent's action probabilities at each tick
for t = 0:T
t > 0 && execute_ai(Lûː[t-1]) ## 4. the environment carries out the previous action
Lŝˣː[t] = state_ai()
Lôː[t] = observe_ai() ## 1. the environment reports the observations
compute_ai(Lôː[t]) ## 2.–3. the agent updates its joint belief and scores the policies
Lsː[t] = belief_ai() ## the marginals of the joint belief
Lquː[t] = scores_ai()[3]
if t < T
Lûː[t] = act_ai() ## 3. the agent chooses the next destination
slide_ai() ## 5. the agent moves one tick forward
end
end
return (; Lŝˣː, Lôː, Lûː, Lsː, Lquː, agent)
end
@time run_motion = run_agent(LC_motion) ## the Part 4 agent: only the preference for no motion
@time run_both = run_agent(LC) ## the agent of this part: no motion and dirt picked up
## from here on, the recorded data and the agent functions are those of the agent with both preferences
(; Lŝˣː, Lôː, Lûː, Lsː, Lquː) = run_both
(compute_ai, act_ai, future_ai, slide_ai, belief_ai, scores_ai) = run_both.agent 62.080730 seconds (690.02 M allocations: 19.469 GiB, 7.14% gc time, 9.14% compilation time)
54.749338 seconds (682.88 M allocations: 19.118 GiB, 7.39% gc time)(var"#244#245"{OffsetVector{Vector{Float64}, Vector{Vector{Float64}}}}([[0.0, 0.0, 0.0], [-0.10536051565782628, -2.3025850929940455], [-1.6094379124341003, -0.2231435513142097]], Core.Box([4.712829172839044, 4.6755360411607025, 4.664757772245612, 3.9831592151363213, 6.099011184306572, 5.063027877527064, 4.255826478762758, 4.842182109200561, 4.736521673165976, 4.335601547138698 … 5.351867956391471, 4.359521318208632, 5.18402798854745, 4.50917436238906, 4.287226384422834, 6.123015116149876, 5.267611204568751, 4.49395290602073, 5.413905455279947, 5.260931723220188]), Core.Box([0.040149962553435115, 0.0416755505623596, 0.04212717032102044, 0.08328691595176727, 0.010038618410293798, 0.028287578981017737, 0.06341016639376894, 0.035278318603227186, 0.03920988986297163, 0.058548128969931626 … 0.02119113055677273, 0.057164287692077626, 0.025063752829259694, 0.04921883218138775, 0.061450028274987564, 0.009800521165493218, 0.023054004283829792, 0.04997374535127382, 0.019916434103922027, 0.02320850850504581]), Core.Box([0.5255988633239985, 0.1792226986197159, 0.29517843805628563]), Core.Box([5.646567725807728e-6 3.029390586891511e-5 2.0306532956214736e-5 2.928887991651126e-5; 0.0005038624036505912 0.0009289654030892082 0.00023519118989974214 0.0006774373618875837; 0.08167929081092434 0.7148994341336965 0.00802980633376126 0.10980340628635316;;; 5.076947228106625e-8 2.72065547462534e-7 1.824904885284146e-7 2.634652826310733e-7; 4.314679029903219e-5 7.956613755032243e-5 2.0143304667206638e-5 5.804767326679341e-5; 0.006994361972182242 0.06123131208979698 0.0006877226684764285 0.00940872890915015;;;; 6.408535927561296e-7 3.4578301582622525e-6 2.3115872227984014e-6 3.3292012009719804e-6; 6.304606586285456e-8 1.1675719938606652e-7 2.9523651522008007e-8 8.473787481261099e-8; 0.00010309209565969805 0.0008604276700927063 1.053848857439515e-5 0.00013706055370616183;;; 5.7269446348624975e-9 3.076373154246357e-8 2.0618354236861335e-8 2.9740621364692983e-8; 5.398550044215362e-9 9.999476530093149e-9 2.5284140719951548e-9 7.260367853155942e-9; 8.825224826584376e-6 7.36831839765967e-5 9.016795674808563e-7 1.1738777759483826e-5;;;;; 4.897394536352162e-10 1.5003981483626187e-9 5.154032399609853e-10 1.2342259245868565e-9; 1.6126876152167532e-5 2.9592398164666243e-5 7.5043707387497385e-6 2.1685793391538797e-5; 0.0002750057627469069 0.0023969843583863406 2.6940913635468404e-5 0.00036979994402424547;;; 7.242551429257796e-13 3.992889082988592e-12 2.6620835518092968e-12 3.818292584433334e-12; 1.3809820448702144e-6 2.5346086262545614e-6 6.427259118370605e-7 1.8582126806979485e-6; 2.3549384901233588e-5 0.00020530330670857412 2.30740494826592e-6 3.1687423872941544e-5;;;; 1.7005284590854252e-10 4.71081407334668e-10 1.231802569485561e-10 3.695637738827637e-10; 1.9928488494365314e-9 3.6904039462674004e-9 9.33181238077456e-10 2.6785342617427333e-9; 3.228529403585477e-7 2.839783045820939e-6 3.1828698246477505e-8 4.339372940306373e-7;;; 8.581112331059218e-14 5.04638541602337e-13 3.3193210313294246e-13 4.684217068051143e-13; 1.7064547370145628e-10 3.1606011000963285e-10 7.991828092642982e-11 2.2949945251697602e-10; 2.764537900332536e-8 2.432092624288759e-7 2.7257737048408356e-9 3.717984701007775e-8]), Core.Box(1), Core.Box([6.196288599453033e-7 1.8215553665067577e-7 1.859473145959444e-8 1.9125693364833357e-7; 0.07019685993301164 0.766067833398496 0.006854489807309227 0.08827321198444395; 0.0004306213045750505 0.0002960966112093086 0.0006728408045088188 0.0005428750967890881;;; 5.765353056341923e-11 1.672000813704816e-11 1.7146544200437883e-12 1.7451780709175015e-11; 0.004486426722516413 0.04898036672526618 0.00043815264998514023 0.0056481034735606; 2.7499930000784277e-5 1.882825215458267e-5 4.2840470628932015e-5 3.462685721611603e-5;;;; 1.998307693457894e-7 5.8857382219944603e-8 6.0040998693836045e-9 6.15822658007706e-8; 0.00041845039573640817 0.004596085977032206 4.076991512060059e-5 0.0005262838262850438; 2.4650906186445617e-5 1.7374846739275067e-5 3.911726847123953e-5 3.1416248669974124e-5;;; 1.8591689143952604e-11 5.401795330599345e-12 5.535815401054813e-13 5.618720783005184e-12; 2.6743207875687236e-5 0.0002938299685185997 2.605944943328121e-6 3.367040190794764e-5; 1.568357501274358e-6 1.0773790688494035e-6 2.4458046587300692e-6 1.9749591430464185e-6;;;;; 1.9289989271340347e-8 5.733189928570033e-9 5.845704337626592e-10 6.043994369196014e-9; 7.753356689741947e-5 0.0006860848137524661 8.889276773111629e-6 9.668658751346022e-5; 3.118681311764778e-8 2.5980976898941596e-8 5.565025381737675e-8 4.352227003899792e-8;;; 1.7846573684929208e-12 5.177981319830459e-13 5.3090730730228357e-14 5.40576281632828e-13; 4.955813783381859e-6 4.3875630880266216e-5 5.683353392347695e-7 6.1889889244682225e-6; 1.921443219945557e-9 1.3188660728949647e-9 2.9990655997599103e-9 2.424467795565098e-9;;;; 6.221165465262981e-9 1.8525534343436155e-9 1.8876145840065672e-10 1.9461162875074287e-9; 2.769721343569182e-7 3.0298470768583424e-6 2.6971549515151353e-8 3.483664163274129e-7; 2.0763486921218907e-9 2.923356467704036e-9 5.521234427748079e-9 3.9919937960249355e-9;;; 5.755042735893337e-13 1.672875276329198e-13 1.7140559380475737e-14 1.7404258637595014e-13; 1.7703526670403124e-8 1.9375931081809578e-7 1.7243329579990916e-9 2.2299389677570852e-8; 1.0949520306785859e-10 7.621899571964949e-11 1.725267755908005e-10 1.3908231045740405e-10])), var"#246#247"(Core.Box([0.5255988633239985, 0.1792226986197159, 0.29517843805628563]), Core.Box(1)), var"#248#249"(Core.Box([5.646567725807728e-6 3.029390586891511e-5 2.0306532956214736e-5 2.928887991651126e-5; 0.0005038624036505912 0.0009289654030892082 0.00023519118989974214 0.0006774373618875837; 0.08167929081092434 0.7148994341336965 0.00802980633376126 0.10980340628635316;;; 5.076947228106625e-8 2.72065547462534e-7 1.824904885284146e-7 2.634652826310733e-7; 4.314679029903219e-5 7.956613755032243e-5 2.0143304667206638e-5 5.804767326679341e-5; 0.006994361972182242 0.06123131208979698 0.0006877226684764285 0.00940872890915015;;;; 6.408535927561296e-7 3.4578301582622525e-6 2.3115872227984014e-6 3.3292012009719804e-6; 6.304606586285456e-8 1.1675719938606652e-7 2.9523651522008007e-8 8.473787481261099e-8; 0.00010309209565969805 0.0008604276700927063 1.053848857439515e-5 0.00013706055370616183;;; 5.7269446348624975e-9 3.076373154246357e-8 2.0618354236861335e-8 2.9740621364692983e-8; 5.398550044215362e-9 9.999476530093149e-9 2.5284140719951548e-9 7.260367853155942e-9; 8.825224826584376e-6 7.36831839765967e-5 9.016795674808563e-7 1.1738777759483826e-5;;;;; 4.897394536352162e-10 1.5003981483626187e-9 5.154032399609853e-10 1.2342259245868565e-9; 1.6126876152167532e-5 2.9592398164666243e-5 7.5043707387497385e-6 2.1685793391538797e-5; 0.0002750057627469069 0.0023969843583863406 2.6940913635468404e-5 0.00036979994402424547;;; 7.242551429257796e-13 3.992889082988592e-12 2.6620835518092968e-12 3.818292584433334e-12; 1.3809820448702144e-6 2.5346086262545614e-6 6.427259118370605e-7 1.8582126806979485e-6; 2.3549384901233588e-5 0.00020530330670857412 2.30740494826592e-6 3.1687423872941544e-5;;;; 1.7005284590854252e-10 4.71081407334668e-10 1.231802569485561e-10 3.695637738827637e-10; 1.9928488494365314e-9 3.6904039462674004e-9 9.33181238077456e-10 2.6785342617427333e-9; 3.228529403585477e-7 2.839783045820939e-6 3.1828698246477505e-8 4.339372940306373e-7;;; 8.581112331059218e-14 5.04638541602337e-13 3.3193210313294246e-13 4.684217068051143e-13; 1.7064547370145628e-10 3.1606011000963285e-10 7.991828092642982e-11 2.2949945251697602e-10; 2.764537900332536e-8 2.432092624288759e-7 2.7257737048408356e-9 3.717984701007775e-8])), var"#250#251"(Core.Box(1), Core.Box([5.646567725807728e-6 3.029390586891511e-5 2.0306532956214736e-5 2.928887991651126e-5; 0.0005038624036505912 0.0009289654030892082 0.00023519118989974214 0.0006774373618875837; 0.08167929081092434 0.7148994341336965 0.00802980633376126 0.10980340628635316;;; 5.076947228106625e-8 2.72065547462534e-7 1.824904885284146e-7 2.634652826310733e-7; 4.314679029903219e-5 7.956613755032243e-5 2.0143304667206638e-5 5.804767326679341e-5; 0.006994361972182242 0.06123131208979698 0.0006877226684764285 0.00940872890915015;;;; 6.408535927561296e-7 3.4578301582622525e-6 2.3115872227984014e-6 3.3292012009719804e-6; 6.304606586285456e-8 1.1675719938606652e-7 2.9523651522008007e-8 8.473787481261099e-8; 0.00010309209565969805 0.0008604276700927063 1.053848857439515e-5 0.00013706055370616183;;; 5.7269446348624975e-9 3.076373154246357e-8 2.0618354236861335e-8 2.9740621364692983e-8; 5.398550044215362e-9 9.999476530093149e-9 2.5284140719951548e-9 7.260367853155942e-9; 8.825224826584376e-6 7.36831839765967e-5 9.016795674808563e-7 1.1738777759483826e-5;;;;; 4.897394536352162e-10 1.5003981483626187e-9 5.154032399609853e-10 1.2342259245868565e-9; 1.6126876152167532e-5 2.9592398164666243e-5 7.5043707387497385e-6 2.1685793391538797e-5; 0.0002750057627469069 0.0023969843583863406 2.6940913635468404e-5 0.00036979994402424547;;; 7.242551429257796e-13 3.992889082988592e-12 2.6620835518092968e-12 3.818292584433334e-12; 1.3809820448702144e-6 2.5346086262545614e-6 6.427259118370605e-7 1.8582126806979485e-6; 2.3549384901233588e-5 0.00020530330670857412 2.30740494826592e-6 3.1687423872941544e-5;;;; 1.7005284590854252e-10 4.71081407334668e-10 1.231802569485561e-10 3.695637738827637e-10; 1.9928488494365314e-9 3.6904039462674004e-9 9.33181238077456e-10 2.6785342617427333e-9; 3.228529403585477e-7 2.839783045820939e-6 3.1828698246477505e-8 4.339372940306373e-7;;; 8.581112331059218e-14 5.04638541602337e-13 3.3193210313294246e-13 4.684217068051143e-13; 1.7064547370145628e-10 3.1606011000963285e-10 7.991828092642982e-11 2.2949945251697602e-10; 2.764537900332536e-8 2.432092624288759e-7 2.7257737048408356e-9 3.717984701007775e-8]), Core.Box(1), Core.Box([6.196288599453033e-7 1.8215553665067577e-7 1.859473145959444e-8 1.9125693364833357e-7; 0.07019685993301164 0.766067833398496 0.006854489807309227 0.08827321198444395; 0.0004306213045750505 0.0002960966112093086 0.0006728408045088188 0.0005428750967890881;;; 5.765353056341923e-11 1.672000813704816e-11 1.7146544200437883e-12 1.7451780709175015e-11; 0.004486426722516413 0.04898036672526618 0.00043815264998514023 0.0056481034735606; 2.7499930000784277e-5 1.882825215458267e-5 4.2840470628932015e-5 3.462685721611603e-5;;;; 1.998307693457894e-7 5.8857382219944603e-8 6.0040998693836045e-9 6.15822658007706e-8; 0.00041845039573640817 0.004596085977032206 4.076991512060059e-5 0.0005262838262850438; 2.4650906186445617e-5 1.7374846739275067e-5 3.911726847123953e-5 3.1416248669974124e-5;;; 1.8591689143952604e-11 5.401795330599345e-12 5.535815401054813e-13 5.618720783005184e-12; 2.6743207875687236e-5 0.0002938299685185997 2.605944943328121e-6 3.367040190794764e-5; 1.568357501274358e-6 1.0773790688494035e-6 2.4458046587300692e-6 1.9749591430464185e-6;;;;; 1.9289989271340347e-8 5.733189928570033e-9 5.845704337626592e-10 6.043994369196014e-9; 7.753356689741947e-5 0.0006860848137524661 8.889276773111629e-6 9.668658751346022e-5; 3.118681311764778e-8 2.5980976898941596e-8 5.565025381737675e-8 4.352227003899792e-8;;; 1.7846573684929208e-12 5.177981319830459e-13 5.3090730730228357e-14 5.40576281632828e-13; 4.955813783381859e-6 4.3875630880266216e-5 5.683353392347695e-7 6.1889889244682225e-6; 1.921443219945557e-9 1.3188660728949647e-9 2.9990655997599103e-9 2.424467795565098e-9;;;; 6.221165465262981e-9 1.8525534343436155e-9 1.8876145840065672e-10 1.9461162875074287e-9; 2.769721343569182e-7 3.0298470768583424e-6 2.6971549515151353e-8 3.483664163274129e-7; 2.0763486921218907e-9 2.923356467704036e-9 5.521234427748079e-9 3.9919937960249355e-9;;; 5.755042735893337e-13 1.672875276329198e-13 1.7140559380475737e-14 1.7404258637595014e-13; 1.7703526670403124e-8 1.9375931081809578e-7 1.7243329579990916e-9 2.2299389677570852e-8; 1.0949520306785859e-10 7.621899571964949e-11 1.725267755908005e-10 1.3908231045740405e-10])), var"#252#253"(Core.Box([5.646567725807728e-6 3.029390586891511e-5 2.0306532956214736e-5 2.928887991651126e-5; 0.0005038624036505912 0.0009289654030892082 0.00023519118989974214 0.0006774373618875837; 0.08167929081092434 0.7148994341336965 0.00802980633376126 0.10980340628635316;;; 5.076947228106625e-8 2.72065547462534e-7 1.824904885284146e-7 2.634652826310733e-7; 4.314679029903219e-5 7.956613755032243e-5 2.0143304667206638e-5 5.804767326679341e-5; 0.006994361972182242 0.06123131208979698 0.0006877226684764285 0.00940872890915015;;;; 6.408535927561296e-7 3.4578301582622525e-6 2.3115872227984014e-6 3.3292012009719804e-6; 6.304606586285456e-8 1.1675719938606652e-7 2.9523651522008007e-8 8.473787481261099e-8; 0.00010309209565969805 0.0008604276700927063 1.053848857439515e-5 0.00013706055370616183;;; 5.7269446348624975e-9 3.076373154246357e-8 2.0618354236861335e-8 2.9740621364692983e-8; 5.398550044215362e-9 9.999476530093149e-9 2.5284140719951548e-9 7.260367853155942e-9; 8.825224826584376e-6 7.36831839765967e-5 9.016795674808563e-7 1.1738777759483826e-5;;;;; 4.897394536352162e-10 1.5003981483626187e-9 5.154032399609853e-10 1.2342259245868565e-9; 1.6126876152167532e-5 2.9592398164666243e-5 7.5043707387497385e-6 2.1685793391538797e-5; 0.0002750057627469069 0.0023969843583863406 2.6940913635468404e-5 0.00036979994402424547;;; 7.242551429257796e-13 3.992889082988592e-12 2.6620835518092968e-12 3.818292584433334e-12; 1.3809820448702144e-6 2.5346086262545614e-6 6.427259118370605e-7 1.8582126806979485e-6; 2.3549384901233588e-5 0.00020530330670857412 2.30740494826592e-6 3.1687423872941544e-5;;;; 1.7005284590854252e-10 4.71081407334668e-10 1.231802569485561e-10 3.695637738827637e-10; 1.9928488494365314e-9 3.6904039462674004e-9 9.33181238077456e-10 2.6785342617427333e-9; 3.228529403585477e-7 2.839783045820939e-6 3.1828698246477505e-8 4.339372940306373e-7;;; 8.581112331059218e-14 5.04638541602337e-13 3.3193210313294246e-13 4.684217068051143e-13; 1.7064547370145628e-10 3.1606011000963285e-10 7.991828092642982e-11 2.2949945251697602e-10; 2.764537900332536e-8 2.432092624288759e-7 2.7257737048408356e-9 3.717984701007775e-8])), var"#254#255"(Core.Box([4.712829172839044, 4.6755360411607025, 4.664757772245612, 3.9831592151363213, 6.099011184306572, 5.063027877527064, 4.255826478762758, 4.842182109200561, 4.736521673165976, 4.335601547138698 … 5.351867956391471, 4.359521318208632, 5.18402798854745, 4.50917436238906, 4.287226384422834, 6.123015116149876, 5.267611204568751, 4.49395290602073, 5.413905455279947, 5.260931723220188]), Core.Box([0.040149962553435115, 0.0416755505623596, 0.04212717032102044, 0.08328691595176727, 0.010038618410293798, 0.028287578981017737, 0.06341016639376894, 0.035278318603227186, 0.03920988986297163, 0.058548128969931626 … 0.02119113055677273, 0.057164287692077626, 0.025063752829259694, 0.04921883218138775, 0.061450028274987564, 0.009800521165493218, 0.023054004283829792, 0.04997374535127382, 0.019916434103922027, 0.02320850850504581]), Core.Box([0.5255988633239985, 0.1792226986197159, 0.29517843805628563])))Changes in the run compared to Part 4.
run_agent, which runs one agent with given preferences in a fresh environment. It is called once for the Part 4 agent (LC_motion) and once for the agent with both preferences (LC), both with the same seed as the random Roomba.LdepsA and LdepsB, and the agent its preferences.run_motion for the comparison.## t, site, occupant, bed dust, lounge grime, bath soil: the true states of the agent's run
[(t,
room_labels[argmax(Lŝˣː[t][0])],
person_labels[argmax(Lŝˣː[t][1])],
(dirt_labels[argmax(Lŝˣː[t][n])] for n in 2:4)...)
for t in 0:T]600-element Vector{Tuple{Int64, Vararg{String, 5}}}:
(0, "bedroom", "living room", "clean", "clean", "clean")
(1, "bathroom", "living room", "clean", "clean", "clean")
(2, "living room", "living room", "clean", "clean", "clean")
(3, "bathroom", "living room", "clean", "clean", "clean")
(4, "bedroom", "living room", "clean", "clean", "clean")
(5, "bathroom", "bedroom", "clean", "clean", "clean")
(6, "bedroom", "bedroom", "clean", "clean", "clean")
(7, "bathroom", "living room", "clean", "clean", "clean")
(8, "bedroom", "living room", "clean", "clean", "clean")
(9, "bathroom", "living room", "clean", "dirty", "dirty")
⋮
(591, "bathroom", "away", "clean", "clean", "clean")
(592, "living room", "away", "clean", "clean", "clean")
(593, "bathroom", "living room", "clean", "clean", "clean")
(594, "bedroom", "bedroom", "clean", "clean", "clean")
(595, "bathroom", "bathroom", "clean", "clean", "clean")
(596, "living room", "bathroom", "clean", "clean", "clean")
(597, "bathroom", "living room", "clean", "clean", "clean")
(598, "living room", "living room", "clean", "clean", "clean")
(599, "bathroom", "living room", "clean", "clean", "clean")[action_labels[argmax(Lû[0])] for Lû in Lûː] ## destination chosen by the agent with both preferences at t = 0, ..., T-1, by name599-element OffsetArray(::Vector{String}, 0:598) with eltype String with indices 0:598:
"living room"
"living room"
"bedroom"
"bedroom"
"living room"
"bedroom"
"living room"
"bedroom"
"living room"
"bedroom"
⋮
"living room"
"living room"
"bedroom"
"bedroom"
"living room"
"living room"
"bedroom"
"living room"
"bedroom"[round.(Ls[0], digits=2) for Ls in Lsː] ## s^{(t,0)} for t = 0, ..., T: the agent's belief about the site600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[0.9, 0.05, 0.05]
[0.01, 0.01, 0.98]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[0.99, 0.0, 0.01]
[0.01, 0.0, 0.99]
[0.99, 0.0, 0.01]
[0.1, 0.04, 0.86]
[0.99, 0.0, 0.01]
[0.0, 0.0, 1.0]
⋮
[0.01, 0.0, 0.99]
[0.0, 0.98, 0.02]
[0.0, 0.01, 0.99]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 0.99, 0.01]
[0.0, 0.01, 0.99]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0][round.(Ls[1], digits=2) for Ls in Lsː] ## s^{(t,1)} for t = 0, ..., T: the agent's belief about the occupant600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[0.09, 0.3, 0.3, 0.31]
[0.15, 0.46, 0.04, 0.35]
[0.02, 0.93, 0.01, 0.05]
[0.05, 0.87, 0.01, 0.07]
[0.01, 0.84, 0.04, 0.1]
[0.05, 0.81, 0.01, 0.13]
[0.01, 0.79, 0.04, 0.15]
[0.05, 0.75, 0.02, 0.19]
[0.01, 0.74, 0.04, 0.2]
[0.04, 0.72, 0.01, 0.22]
⋮
[0.03, 0.55, 0.01, 0.41]
[0.07, 0.19, 0.05, 0.69]
[0.08, 0.23, 0.01, 0.68]
[0.54, 0.13, 0.01, 0.32]
[0.45, 0.21, 0.01, 0.33]
[0.47, 0.07, 0.04, 0.41]
[0.4, 0.16, 0.01, 0.42]
[0.08, 0.82, 0.01, 0.1]
[0.09, 0.78, 0.01, 0.12][round.(Ls[3], digits=2) for Ls in Lsː] ## s^{(t,3)} for t = 0, ..., T: the agent's belief about the dirt in the living room600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[0.52, 0.48]
[0.48, 0.52]
[0.89, 0.11]
[0.98, 0.02]
[0.93, 0.07]
[0.88, 0.12]
[0.84, 0.16]
[0.8, 0.2]
[0.76, 0.24]
[0.72, 0.28]
⋮
[0.5, 0.5]
[0.89, 0.11]
[0.98, 0.02]
[0.93, 0.07]
[0.88, 0.12]
[0.98, 0.02]
[1.0, 0.0]
[0.99, 0.01]
[1.0, 0.0]Changes in the belief listings compared to Part 4.
## t, and the agent's most probable value of each factor: room, person, bed dust, lounge grime, bath soil
[(t,
room_labels[argmax(Lsː[t][0])],
person_labels[argmax(Lsː[t][1])],
(dirt_labels[argmax(Lsː[t][n])] for n in 2:4)...)
for t in 0:T]600-element Vector{Tuple{Int64, Vararg{String, 5}}}:
(0, "bedroom", "away", "clean", "clean", "clean")
(1, "bathroom", "living room", "clean", "dirty", "clean")
(2, "living room", "living room", "clean", "clean", "clean")
(3, "bathroom", "living room", "clean", "clean", "clean")
(4, "bedroom", "living room", "clean", "clean", "clean")
(5, "bathroom", "living room", "clean", "clean", "clean")
(6, "bedroom", "living room", "clean", "clean", "clean")
(7, "bathroom", "living room", "clean", "clean", "clean")
(8, "bedroom", "living room", "clean", "clean", "clean")
(9, "bathroom", "living room", "clean", "clean", "clean")
⋮
(591, "bathroom", "living room", "clean", "dirty", "clean")
(592, "living room", "away", "clean", "clean", "clean")
(593, "bathroom", "away", "clean", "clean", "clean")
(594, "bedroom", "bedroom", "clean", "clean", "clean")
(595, "bathroom", "bedroom", "clean", "clean", "clean")
(596, "living room", "bedroom", "clean", "clean", "clean")
(597, "bathroom", "away", "clean", "clean", "clean")
(598, "living room", "living room", "clean", "clean", "clean")
(599, "bathroom", "living room", "clean", "clean", "clean")Changes in the agent’s named state listing compared to Part 4.
## t, true (room, person, dirt here), agent's most probable (room, person, dirt here), destination chosen (none at t = T)
## "dirt here" is the dirt factor of the room the Roomba is actually in: factor r + 1 for room r
here(t) = argmax(Lŝˣː[t][0]) + 1
[(t,
(room_labels[argmax(Lŝˣː[t][0])], person_labels[argmax(Lŝˣː[t][1])], dirt_labels[argmax(Lŝˣː[t][here(t)])]),
(room_labels[argmax(Lsː[t][0])], person_labels[argmax(Lsː[t][1])], dirt_labels[argmax(Lsː[t][here(t)])]),
t < T ? action_labels[argmax(Lûː[t][0])] : "-")
for t in 0:T]600-element Vector{Tuple{Int64, Tuple{String, String, String}, Tuple{String, String, String}, String}}:
(0, ("bedroom", "living room", "clean"), ("bedroom", "away", "clean"), "living room")
(1, ("bathroom", "living room", "clean"), ("bathroom", "living room", "clean"), "living room")
(2, ("living room", "living room", "clean"), ("living room", "living room", "clean"), "bedroom")
(3, ("bathroom", "living room", "clean"), ("bathroom", "living room", "clean"), "bedroom")
(4, ("bedroom", "living room", "clean"), ("bedroom", "living room", "clean"), "living room")
(5, ("bathroom", "bedroom", "clean"), ("bathroom", "living room", "clean"), "bedroom")
(6, ("bedroom", "bedroom", "clean"), ("bedroom", "living room", "clean"), "living room")
(7, ("bathroom", "living room", "clean"), ("bathroom", "living room", "clean"), "bedroom")
(8, ("bedroom", "living room", "clean"), ("bedroom", "living room", "clean"), "living room")
(9, ("bathroom", "living room", "dirty"), ("bathroom", "living room", "clean"), "bedroom")
⋮
(591, ("bathroom", "away", "clean"), ("bathroom", "living room", "clean"), "living room")
(592, ("living room", "away", "clean"), ("living room", "away", "clean"), "bedroom")
(593, ("bathroom", "living room", "clean"), ("bathroom", "away", "clean"), "bedroom")
(594, ("bedroom", "bedroom", "clean"), ("bedroom", "bedroom", "clean"), "living room")
(595, ("bathroom", "bathroom", "clean"), ("bathroom", "bedroom", "clean"), "living room")
(596, ("living room", "bathroom", "clean"), ("living room", "bedroom", "clean"), "bedroom")
(597, ("bathroom", "living room", "clean"), ("bathroom", "away", "clean"), "living room")
(598, ("living room", "living room", "clean"), ("living room", "living room", "clean"), "bedroom")
(599, ("bathroom", "living room", "clean"), ("bathroom", "living room", "clean"), "-")Changes in the combined listing compared to Part 4.
[Lô[0] for Lô in Lôː] ## ô^{(t,0)} for t = 0, ..., T: the received Texture observations (agent with both preferences)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 1.0, 0.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
⋮
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[1.0, 0.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0]
[0.0, 1.0, 0.0]
[0.0, 0.0, 1.0][Lô[1] for Lô in Lôː] ## ô^{(t,1)} for t = 0, ..., T: the received Motion observations (no motion, motion)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
⋮
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
[1.0, 0.0][Lô[2] for Lô in Lôː] ## ô^{(t,2)} for t = 0, ..., T: the received Dirt observations (nothing picked up, dirt picked up)600-element OffsetArray(::Vector{Vector{Float64}}, 0:599) with eltype Vector{Float64} with indices 0:599:
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[0.0, 1.0]
⋮
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]
[1.0, 0.0]Lsː ## s^{(0:T,•)}: the agent's marginal beliefs at each tick, a tuple of five (replaces result.posteriors[:sː])600-element OffsetArray(::Vector{Any}, 0:599) with eltype Any with indices 0:599:
[[0.8999999999999992, 0.04999999999999996, 0.04999999999999996], [0.0901639344262295, 0.2991803278688523, 0.2991803278688523, 0.3114754098360654], [0.8642857142857134, 0.13571428571428573], [0.5202380952380948, 0.4797619047619045], [0.5202380952380948, 0.4797619047619044]]
[[0.011367952408460372, 0.008292604448200474, 0.9803394431433393], [0.1458553298921994, 0.45565951306158575, 0.044177477359159556, 0.35430767968705523], [0.971184229595561, 0.02881577040443884], [0.4790870043788897, 0.5209129956211105], [0.8937082778257746, 0.10629172217422557]]
[[3.707835519882867e-5, 0.9972216053456141, 0.002741316299187123], [0.01823635881778734, 0.9290554973859432, 0.007274549562429381, 0.04543359423384016], [0.9521744505164768, 0.047825549483523466], [0.8862670271702324, 0.11373297282976777], [0.977016695609572, 0.022983304390427964]]
[[0.00013806310721389288, 0.0021817392339615743, 0.997680197658824], [0.04509132503917317, 0.8719989929611572, 0.009249638547296947, 0.0736600434523723], [0.9343471153767429, 0.06565288462325668], [0.9772534383996808, 0.022746561600318942], [0.9941243430129655, 0.005875656987034319]]
[[0.9931283756382567, 8.70294028607196e-6, 0.006862921421457621], [0.014500137888461325, 0.840103984843838, 0.04363104468599154, 0.10176483258170947], [0.9899022108614827, 0.01009778913851743], [0.9283044139909791, 0.07169558600902122], [0.9988746132722217, 0.001125386727778822]]
[[0.006349392638214023, 0.00014880178157041708, 0.9935018055802154], [0.04720274960005633, 0.8122493952106649, 0.0126945398183668, 0.12785331537091196], [0.9979995164568797, 0.002000483543120124], [0.8819021353892197, 0.11809786461078016], [0.9964600622904655, 0.0035399377095341068]]
[[0.9936616608978259, 6.49771117262832e-7, 0.006337689331057003], [0.014672542024492101, 0.787503934870751, 0.04320710995124809, 0.15461641315350913], [0.9975346802360661, 0.0024653197639340116], [0.8377945433741149, 0.16220545662588534], [0.998898398945551, 0.0011016010544491986]]
[[0.0993109849513692, 0.037953802762327134, 0.8627352122863038], [0.047266374771232744, 0.7474164451688086, 0.01914275726708962, 0.186174422792869], [0.9979797442161729, 0.0020202557838270075], [0.8026530604511034, 0.19734693954889648], [0.9940124254577647, 0.00598757454223533]]
[[0.9920981935984556, 0.00017767771089782338, 0.007724128690646453], [0.013837487368103345, 0.740259471761602, 0.04285499862976365, 0.20304804224053105], [0.9978207638180785, 0.0021792361819217114], [0.7566540866509732, 0.2433459133490267], [0.9924179158954184, 0.007582084104581793]]
[[0.003813899792534884, 0.000642373447463985, 0.9955437267600009], [0.04314890672142287, 0.721102861147421, 0.011714020938855046, 0.22403421119230138], [0.9995568422128128, 0.00044315778718698163], [0.7179928931182721, 0.28200710688172775], [0.590669892847391, 0.4093301071526091]]
⋮
[[0.006325124007613514, 0.00041419777156261723, 0.9932606782208236], [0.032431646952157236, 0.5482716085927167, 0.008391793321523134, 0.4109049511336023], [0.999227115651215, 0.0007728843487848543], [0.4954894285196426, 0.504510571480357], [0.9964763485012519, 0.0035236514987476214]]
[[0.00012109020330898081, 0.9798241232476036, 0.020054786549087836], [0.07418549464293862, 0.19250005005077622, 0.04599104200202928, 0.687323413304256], [0.9791413750379744, 0.0208586249620259], [0.8859499507809793, 0.11405004921902095], [0.999283923010142, 0.0007160769898581779]]
[[0.0011096723304279614, 0.0052175855052245985, 0.9936727421643476], [0.07681347623482956, 0.23177284664310935, 0.007679614425506883, 0.6837340626965547], [0.9595752303539369, 0.04042476964606353], [0.9771712847149162, 0.02282871528508417], [0.9965455891268288, 0.0034544108731713566]]
[[0.9957877521401536, 4.574653130166486e-5, 0.004166501328544328], [0.5437556465813304, 0.12526718321086408, 0.009134838967268064, 0.3218423312405376], [0.9931481537949108, 0.006851846205088994], [0.92834215431123, 0.07165784568876982], [0.9993076003816705, 0.0006923996183292989]]
[[0.00422863511200362, 0.00017314567767933543, 0.9955982192103171], [0.45360988338236674, 0.20835127306462492, 0.008130466065796527, 0.3299083774872117], [0.9986317318749505, 0.0013682681250495221], [0.8819469710562763, 0.11805302894372365], [0.9965666926471821, 0.0034333073528178613]]
[[3.432483092347403e-5, 0.9908613248869139, 0.009104350282163007], [0.46924338661980775, 0.07240115442346172, 0.04395311796410478, 0.41440234099262585], [0.978656971223263, 0.02134302877673745], [0.9787614634807491, 0.021238536519251453], [0.9993305906039898, 0.0006694093960103424]]
[[0.00039715778389172565, 0.005711170778866605, 0.9938916714372417], [0.40249597432316514, 0.16448530489463292, 0.010876099140220643, 0.42214262164198135], [0.9590918625188339, 0.04090813748116623], [0.9957423136196263, 0.004257686380373938], [0.9965311297870972, 0.003468870212902815]]
[[1.3798984554462658e-6, 0.9978125849896099, 0.0021860351119347434], [0.07569648515723516, 0.821004956130393, 0.008102838691203146, 0.09519572002116888], [0.9399034053508648, 0.06009659464913552], [0.993937665423934, 0.006062334576066148], [0.9990710774638152, 0.000928922536184966]]
[[9.613588531829398e-5, 0.002628015574712958, 0.9972758485399684], [0.08965540798815509, 0.7807450735868195, 0.009044588570417192, 0.12055492985460804], [0.9211113651425612, 0.07888863485743891], [0.9987796363638087, 0.0012203636361910622], [0.9965831413934997, 0.003416858606500349]]Changes in the recorded beliefs compared to Part 4.
_GΣ, _qπ, _qu = scores_ai() ## the plan of the agent with both preferences at the last tick, t = T
## the 27 policies, from lowest expected free energy (most probable) to highest
[(join(action_labels[_V[p, :]], " → "), round(_GΣ[p], digits=2), round(_qπ[p], digits=3)) for p in sortperm(_GΣ)]27-element Vector{Tuple{String, Float64, Float64}}:
("bedroom → living room → bedroom", 3.98, 0.083)
("bedroom → bathroom → bedroom", 4.26, 0.063)
("bedroom → living room → bathroom", 4.29, 0.061)
("bedroom → living room → living room", 4.32, 0.059)
("bathroom → bedroom → living room", 4.32, 0.059)
("bedroom → bedroom → living room", 4.34, 0.059)
("bedroom → bedroom → bathroom", 4.36, 0.057)
("bedroom → bathroom → living room", 4.45, 0.052)
("bedroom → bathroom → bathroom", 4.49, 0.05)
("bathroom → bedroom → bathroom", 4.51, 0.049)
⋮
("bathroom → living room → bathroom", 5.27, 0.023)
("bathroom → bathroom → living room", 5.35, 0.021)
("living room → bedroom → living room", 5.39, 0.02)
("living room → bathroom → bathroom", 5.41, 0.02)
("living room → bathroom → living room", 5.78, 0.014)
("living room → living room → bedroom", 6.1, 0.01)
("living room → living room → bathroom", 6.12, 0.01)
("bathroom → living room → living room", 6.15, 0.01)
("living room → living room → living room", 7.21, 0.003)Changes in the plan listing compared to Part 4.
Finally, we can check whether the agent achieved what it was built for: keeping the Roomba out of people’s way while still cleaning the apartment. From the true states \(\hat{\mathbf{s}}^{\ast(t)}\), we compute, for each of the three Roombas, the disturbance, the fraction of ticks the Roomba spent in the same room as the person, and the cleanliness, the fraction of ticks each room was dirty. The coverage, the fraction of ticks spent in each room, is shown as background, since it helps explain the cleanliness. The comparison between the Part 4 agent and the agent with both preferences shows what the cleaning goal achieves, and what it costs in disturbance.
Behind this behavior lies the agent’s tracking. From its marginal beliefs \(\boldsymbol{s}^{(t,n)}\), we derive its most probable value of each factor at each tick, and compare it with the true value. We expect the tracking to differ per factor:
Tracking matters for both goals: when the agent thinks the person is elsewhere while they are not, it stays where it causes disturbance, and when it thinks a room is clean while it is not, it stays away from where it is needed.
There is no free energy curve to check this time: the agent’s beliefs come from exact Bayesian filtering at each tick, not from an iterative approximation.
Changes in the evaluation measures compared to Part 4.
Lqsː = Lsː ## the agent's marginal beliefs \boldsymbol{s}^{(t,n)}, t = 0, ..., T (0-based; recorded during the run)
## ---------- one run in detail: beliefs, heading, disturbance ----------
## Tick t is drawn from t - 0.5 to t + 0.5 so the panels share the heatmap tiles.
function belief_panel(n, labels, true_idx; title, true_label)
cols = hcat((Ls[n] for Ls in parent(Lqsː))...) ## (values, time): rows are values
nvals, ntime = size(cols)
xs = collect(Float64.(step_x(ntime)))
ys = collect(range(0.5, step=1.0, length=nvals + 1))
plt = heatmap(xs, ys, cols;
seriescolor=cgrad(:greens), clims=(0, 1), colorbar=:none,
yticks=(1:nvals, labels),
title=title, legend=:topright,
xlims=(-0.5, ntime - 0.5), ylabel="")
# steppost: the value holds from the left edge of tile t to the next edge
plot!(plt, xs, step_y(Float64.(true_idx)); linetype=:steppost, color=:black, lw=1.5, label=true_label)
plt
end
true_site = [argmax(Lŝˣ[whereabouts_n]) for Lŝˣ in parent(Lŝˣː)] ## whereabouts ŝ^{*(t,0)}
true_occupant = [argmax(Lŝˣ[occupant_n]) for Lŝˣ in parent(Lŝˣː)] ## occupant ŝ^{*(t,1)}
true_dust = [[argmax(Lŝˣ[n]) for Lŝˣ in parent(Lŝˣː)] for n in (bed_dust_n, lounge_grime_n, bath_soil_n)] ## bed dust ŝ^{*(t,2)}, lounge grime ŝ^{*(t,3)}, bath soil ŝ^{*(t,4)}
p_site = belief_panel(whereabouts_n, whereabouts_labels, true_site;
title="Belief about \$\\mathit{whereabouts}\$ \$\\textit{\\textbf{s}}^{(t,0)}\$ (shading), true \$\\hat{s}^{\\ast(t,0)}\$ (line)",
true_label="true ŝ*(t,0)")
p_occ = belief_panel(occupant_n, occupant_labels, true_occupant;
title="Belief about \$\\mathit{occupant}\$ \$\\textit{\\textbf{s}}^{(t,1)}\$ (shading), true \$\\hat{s}^{\\ast(t,1)}\$ (line)",
true_label="true ŝ*(t,1)")
p_bed = belief_panel(bed_dust_n, dust_labels, true_dust[1];
title="Belief about \$\\mathit{bed\\ dust}\$ \$\\textit{\\textbf{s}}^{(t,2)}\$ (shading), true \$\\hat{s}^{\\ast(t,2)}\$ (line)",
true_label="true ŝ*(t,2)")
p_lounge = belief_panel(lounge_grime_n, dust_labels, true_dust[2];
title="Belief about \$\\mathit{lounge\\ grime}\$ \$\\textit{\\textbf{s}}^{(t,3)}\$ (shading), true \$\\hat{s}^{\\ast(t,3)}\$ (line)",
true_label="true ŝ*(t,3)")
p_bath = belief_panel(bath_soil_n, dust_labels, true_dust[3];
title="Belief about \$\\mathit{bath\\ soil}\$ \$\\textit{\\textbf{s}}^{(t,4)}\$ (shading), true \$\\hat{s}^{\\ast(t,4)}\$ (line)",
true_label="true ŝ*(t,4)")
## GR draws its own colorbar as a vertical strip. A one-row heatmap is horizontal and centered.
function shading_bar()
n = 200
xs = collect(range(0.0, 1.0; length=n + 1))
cols = reshape(collect(range(0.0, 1.0; length=n)), 1, n)
heatmap(xs, [0.0, 1.0], cols;
seriescolor=cgrad(:greens), clims=(0, 1), colorbar=:none,
yticks=false, xticks=(0:0.2:1),
xlabel="shading: probability \$s^{(t,n)[i]}\$ of value \$i\$",
legend=:none, title="")
end
heads = [argmax(Lû[heading_n]) for Lû in parent(Lûː)] ## heading û^{(t,0)}, t = 0, ..., T-1
failed = [next_room(true_site[t + 1], heads[t + 1]) != true_site[t + 2] for t in 0:(T - 1)]
p_act = plot(collect(Float64.(step_x(T))), step_y(Float64.(heads));
linetype=:steppost, color=:gray, label="heading û(t,0)", legend=:topright,
yticks=(1:3, heading_labels), ylims=(0.5, 3.5), xlims=(-0.5, T + 0.5),
title="Chosen \$\\mathit{heading}\$ \$\\hat{u}^{(t,0)}\$, failed moves in red")
scatter!(p_act, (0:(T - 1))[failed], heads[failed];
color=:red, markersize=3, markerstrokewidth=0, label="failed move")
same = true_site .== true_occupant ## whereabouts ŝ^{*(t,0)} = occupant ŝ^{*(t,1)}
p_evt = scatter((0:T)[same], fill(1, count(same));
color=:black, markersize=2, markerstrokewidth=0,
label="same room", legend=:topright,
yticks=false, xlims=(-0.5, T + 0.5), ylims=(0.5, 1.5),
xlabel="Step \$t\$",
title="Disturbance, ticks where \$\\mathit{whereabouts}\$ \$\\hat{s}^{\\ast(t,0)}\$ and \$\\mathit{occupant}\$ \$\\hat{s}^{\\ast(t,1)}\$ match")
plot(p_site, p_occ, p_bed, p_lounge, p_bath, shading_bar(), p_act, p_evt;
layout=@layout([a; b; c; d; e; f{0.16h}; g; h]),
size=(980, 1760),
left_margin=22Plots.mm, bottom_margin=4Plots.mm)
In the plot above, look for three things.
## ---------- behavior: disturbance, cleanliness and coverage (4.3.5), for one run ----------
function measures(Lŝˣː)
rooms = [argmax(Lŝˣ[0]) for Lŝˣ in parent(Lŝˣː)]
persons = [argmax(Lŝˣ[1]) for Lŝˣ in parent(Lŝˣː)]
dirty = [[argmax(Lŝˣ[n]) == 2 for Lŝˣ in parent(Lŝˣː)] for n in 2:4] ## bed dust, lounge grime, bath soil
(; disturbance = mean(rooms .== persons), ## J_disturb
dirtiness = [mean(dirty[r]) for r in 1:3], ## J_dirty(r)
coverage = [mean(rooms .== r) for r in 1:3]) ## background
end
m_motion = measures(run_motion.Lŝˣː) ## the Part 4 agent
m_both = measures(Lŝˣː) ## the agent with both preferences
pct(x) = round.(100 .* x, digits=1)
println(" disturbance dirty (bedroom, living room, bathroom) average dirty coverage")
println("random Roomba: $(pct(disturbance_random))% $(pct(dirtiness_random))% $(pct(mean(dirtiness_random)))% $(pct(coverage_random))%")
println("Part 4 agent: $(pct(m_motion.disturbance))% $(pct(m_motion.dirtiness))% $(pct(mean(m_motion.dirtiness)))% $(pct(m_motion.coverage))%")
println("both preferences: $(pct(m_both.disturbance))% $(pct(m_both.dirtiness))% $(pct(mean(m_both.dirtiness)))% $(pct(m_both.coverage))%")
## ---------- tracking (agent with both preferences) ----------
factor_names = ["site", "occupant", "bed dust", "lounge grime", "bath soil"]
accuracy = [mean(argmax(Lqsː[t][n]) == argmax(Lŝˣː[t][n]) for t in 0:T) for n in 0:4] ## J_track(n)
for n in 0:4
println("tracking, $(factor_names[n+1]): $(pct(accuracy[n+1]))%")
end
rooms = [argmax(Lŝˣ[0]) for Lŝˣ in parent(Lŝˣː)] ## true room: 1 bedroom, 2 living room, 3 bathroom
persons = [argmax(Lŝˣ[1]) for Lŝˣ in parent(Lŝˣː)] ## true person location
inferred_rooms = [argmax(Ls[0]) for Ls in parent(Lqsː)] ## most probable room
## the person in the Roomba (true) room: the person marginal's entry for that room
person_here_true = persons .== rooms
p_person_here = [Lqsː[t][1][rooms[t+1]] for t in 0:T]
inferred_person_here = p_person_here .> 0.5
accuracy_person_here = mean(inferred_person_here .== person_here_true)
## the dirt in the Roomba (true) room: the dirt factor r + 1 for room r
dirty_here_true = [argmax(Lŝˣː[t][rooms[t+1]+1]) == 2 for t in 0:T]
p_dirty_here = [Lqsː[t][rooms[t+1]+1][2] for t in 0:T]
inferred_dirty_here = p_dirty_here .> 0.5
accuracy_dirty_here = mean(inferred_dirty_here .== dirty_here_true)
## can the agent overrule its sensors?
textures = [argmax(Lô[0]) for Lô in parent(Lôː)]
camera_wrong = textures .!= rooms
accuracy_cam_wrong = mean(inferred_rooms[camera_wrong] .== rooms[camera_wrong])
println("camera misread the floor at $(count(camera_wrong)) of $(T+1) ticks; ",
"agent still inferred the right room at $(pct(accuracy_cam_wrong))% of those")
motion_detected = [argmax(Lô[1]) for Lô in parent(Lôː)] .== 2 ## modality 1: stir
motion_wrong = motion_detected .!= person_here_true
accuracy_mot_wrong = mean(inferred_person_here[motion_wrong] .== person_here_true[motion_wrong])
println("motion sensor was wrong at $(count(motion_wrong)) of $(T+1) ticks; ",
"agent still inferred correctly whether the person was in its room at $(pct(accuracy_mot_wrong))% of those")
dirt_picked_up = [argmax(Lô[2]) for Lô in parent(Lôː)] .== 2 ## modality 2: pickup
dirt_wrong = dirt_picked_up .!= dirty_here_true
accuracy_dirt_wrong = mean(inferred_dirty_here[dirt_wrong] .== dirty_here_true[dirt_wrong])
println("dirt sensor was wrong at $(count(dirt_wrong)) of $(T+1) ticks; ",
"agent still inferred correctly whether its room was dirty at $(pct(accuracy_dirt_wrong))% of those")
disturbance dirty (bedroom, living room, bathroom) average dirty coverage
random Roomba: 15.8% [8.7, 20.7, 3.2]% 10.8% [22.5, 25.8, 51.7]%
Part 4 agent: 8.8% [6.5, 26.2, 2.8]% 11.8% [34.3, 15.3, 50.3]%
both preferences: 11.0% [4.3, 17.8, 2.7]% 8.3% [32.2, 17.8, 50.0]%
tracking, site: 94.7%
tracking, occupant: 70.3%
tracking, bed dust: 97.3%
tracking, lounge grime: 84.8%
tracking, bath soil: 98.0%
camera misread the floor at 53 of 600 ticks; agent still inferred the right room at 50.9% of those
motion sensor was wrong at 31 of 600 ticks; agent still inferred correctly whether the person was in its room at 51.6% of those
dirt sensor was wrong at 27 of 600 ticks; agent still inferred correctly whether its room was dirty at 77.8% of thoseChanges in the evaluation code compared to Part 4.
measures: disturbance (the Roomba in the person’s room), the fraction of ticks each room is dirty, and coverage as background. They are printed side by side.## ---------- comparison over 30 seeds (section 9) ----------
## Environment: MersenneTwister(seed). Random heading: MersenneTwister(seed + 1000).
## The detailed random walk in 4.3 keeps its own action stream, MersenneTwister(7).
function rollout_actions(action_at; seed)
(execute_i, _, state_i) = create_envir(; LAˣ, LBˣ, LdepsA, LdepsB, Lŝˣ₀, seed)
Lŝ = offset0(Vector{Any}(undef, T + 1))
for t = 0:T
t > 0 && execute_i(action_at(t - 1))
Lŝ[t] = state_i()
end
Lŝ
end
stay_at(_) = tuple0([1.0, 0.0, 0.0]) ## heading "bedroom"
function cycle_at(t)
seq = (1, 3, 2, 3) ## bedroom, bathroom, living room, bathroom
u = seq[mod(t, 4) + 1]
tuple0(Float64.(1:3 .== u))
end
function random_at(seed)
rng = MersenneTwister(seed + 1000)
_ -> tuple0(Float64.(1:3 .== rand(rng, 1:3)))
end
function mean_measures(runs)
ms = measures.(runs)
n = length(ms)
(; disturbance = sum(m.disturbance for m in ms) / n,
dirtiness = reduce(+, (m.dirtiness for m in ms)) / n,
coverage = reduce(+, (m.coverage for m in ms)) / n)
end
seeds = 1:30
println("averaging ", length(seeds), " seeds")
@time m_stay_30 = mean_measures([rollout_actions(stay_at; seed) for seed in seeds])
@time m_cycle_30 = mean_measures([rollout_actions(cycle_at; seed) for seed in seeds])
@time m_random_30 = mean_measures([rollout_actions(random_at(seed); seed) for seed in seeds])
@time m_motion_30 = mean_measures([run_agent(LC_motion; seed).Lŝˣː for seed in seeds])
@time m_both_30 = mean_measures([run_agent(LC; seed).Lŝˣː for seed in seeds])
rows_30 = [
("stay in the bedroom", m_stay_30),
("fixed cycle", m_cycle_30),
("random heading", m_random_30),
("Part 4 agent", m_motion_30),
("both preferences", m_both_30),
]
println("strategy, over $(length(seeds)) seeds | disturbance | bed dust | lounge grime | bath soil | average dirty")
for (name, m) in rows_30
d = m.dirtiness
println(name, " | ",
round(100 * m.disturbance, digits=1), "% | ",
round(100 * d[1], digits=1), "% | ",
round(100 * d[2], digits=1), "% | ",
round(100 * d[3], digits=1), "% | ",
round(100 * mean(d), digits=1), "%")
endaveraging 30 seeds
1.263137 seconds (3.81 M allocations: 145.383 MiB, 7.73% gc time, 53.23% compilation time)
4.311807 seconds (3.00 M allocations: 106.576 MiB, 84.38% gc time, 2.27% compilation time)
3.952080 seconds (3.10 M allocations: 111.704 MiB, 82.00% gc time, 2.97% compilation time)
1642.327908 seconds (20.49 G allocations: 573.552 GiB, 7.93% gc time, 0.01% compilation time)
1588.077947 seconds (20.49 G allocations: 573.548 GiB, 7.74% gc time, 0.00% compilation time)
strategy, over 30 seeds | disturbance | bed dust | lounge grime | bath soil | average dirty
stay in the bedroom | 12.1% | 0.0% | 97.4% | 94.3% | 63.9%
fixed cycle | 17.3% | 6.2% | 13.7% | 2.5% | 7.5%
random heading | 17.6% | 9.1% | 20.0% | 3.4% | 10.9%
Part 4 agent | 11.4% | 4.1% | 27.3% | 2.8% | 11.4%
both preferences | 13.0% | 4.6% | 18.4% | 2.8% | 8.6%┌ Warning: JSON-RPC endpoint has been blocked writing to its peer; outgoing messages are queueing up and will not be delivered until the peer resumes reading
│ blocked_seconds = 1646.7
│ queued_messages = 1
└ @ VSCodeServer.JSONRPC /home/vscode/.cursor-server/extensions/julialang.language-julia-1.250.2-universal/scripts/packages/JSONRPC/src/core.jl:414The agent had two goals. It had to keep the Roomba out of the person’s way, as in Part 4, and it had to clean. It observed surface, commotion and pickup, and it chose a heading. It had to infer the whereabouts, the occupant, and the three dirt factors, from sensors that each see only some of them. Four ingredients made this possible:
The question of this part: can those two preferences share one policy when the living room is both the dirtiest room and the room the person uses most?
On the detailed run, seed 42, cleaning helps and it is not free. The three Roombas in that one environment compare as follows. These are one run each, the run plotted above.
| \(J_{\text{disturb}}\) | bed dust | lounge grime | bath soil | \(J_{\text{dirty}}\) | |
|---|---|---|---|---|---|
| random heading | 15.8% | 8.7% | 20.7% | 3.2% | 10.8% |
| Part 4 agent, “no motion” only | 8.8% | 6.5% | 26.2% | 2.8% | 11.8% |
| both preferences | 11.0% | 4.3% | 17.8% | 2.7% | 8.3% |
The Part 4 agent disturbs the person least, and it leaves lounge grime “dirty” 26.2% of the time, dirtier than the random heading at 20.7%. Both preferences bring that down to 17.8%, and \(J_{\text{dirty}}\) from 11.8% to 8.3%. The price on this run is 2.2 percentage points of disturbance, 11.0% against 8.8%, still about 30% below the random heading. The agent does it without simply living in the living room: its time there is 17.8% against 15.3% for the Part 4 agent. The dirt under the Roomba comes in short stretches, because the agent stays while pickup reports dirt and leaves once the room is clean.
On that same run, the occupant now carries over. \(J_{\text{track}}^{(0)}\) for the whereabouts is 94.7%. Whether the person is in the Roomba’s room is right 96.0% of ticks. In Part 4 the belief followed the latest motion reading, and it was wrong at every tick the sensor was wrong. Here the motion sensor is wrong at 31 ticks, and the agent is still right at 51.6% of those. The occupant persists, so an earlier observation is still evidence. The dirt sensor is wrong at 27 ticks, and the agent overrules it at 77.8% of those, because dirt stays until it is cleaned and the agent knows where it has just cleaned. The camera misreads the floor at 53 ticks, and the agent still has the right whereabouts at 50.9% of those.
\(J_{\text{track}}^{(1)}\) for the occupant is only 70.3%. The agent senses the person only when they share the whereabouts. bed dust is tracked at 97.3% and bath soil at 98.0%, but those rooms are rarely dirty, so always guessing “clean” would already score 95.7% and 97.3%. lounge grime is the test that matters, 84.8% against 82.2% for always guessing “clean”. The agent knows the dirt where it is standing. Elsewhere it knows little more than the base rate.
Across 30 seeds, cleaning still costs a little disturbance. Seeds 1 to 30. The environment uses MersenneTwister(seed). The random heading uses MersenneTwister(seed+1000). Every percentage below is a mean over those 30 seeds.
| \(J_{\text{disturb}}\) | bed dust | lounge grime | bath soil | \(J_{\text{dirty}}\) | |
|---|---|---|---|---|---|
| stay, heading “bedroom” | 12.1% | 0.0% | 97.4% | 94.3% | 63.9% |
| fixed cycle | 17.3% | 6.2% | 13.7% | 2.5% | 7.5% |
| random heading | 17.6% | 9.1% | 20.0% | 3.4% | 10.9% |
| Part 4 agent, “no motion” only | 11.4% | 4.1% | 27.3% | 2.8% | 11.4% |
| both preferences | 13.0% | 4.6% | 18.4% | 2.8% | 8.6% |
Over these 30 seeds the Part 4 agent disturbs least, 11.4%. Both preferences raise that to 13.0%, and bring lounge grime from 27.3% down to 18.4% and \(J_{\text{dirty}}\) from 11.4% to 8.6%. The random heading sits at 17.6% disturbance and 10.9% dirty. Staying in the bedroom keeps bed dust at 0.0% and leaves the other two rooms dirty, 97.4% and 94.3%, so \(J_{\text{dirty}}\) is 63.9% at 12.1% disturbance. The fixed cycle is the cleanest of the five, \(J_{\text{dirty}}\) 7.5%, and it disturbs about as much as the random heading, 17.3%. The preference for “dirt picked up” is cleaner than the Part 4 agent. A cycle that visits every room is cleaner still.
Caveats. The plotted run is one path, seed 42. On that path the disturbance gap between the two agents is 2.2 points, 11.0% against 8.8%. Over the 30 seeds the gap is 1.6 points, 13.0% against 11.4%, in the same direction. The probabilities in 4.3.4 are assumed. A slower living room, or a person who avoids the Roomba, would change which preference wins. The agent was given \(\mathbf{A}\) and \(\mathbf{B}\), so the results say nothing about learning those tensors. The belief is the exact joint. A factorized approximation would drop the dependence commotion is informative about. Tracking was measured on the plotted run only.
Possible next steps:
The Roomba in this part both cleans and stays out of the way, and the balance between the two is now a preference the next part can move.
Three styles, each with one role. Italic lower case is a model-variable name. Double quotes are a value, spelled as in the label vector. Bold is a low-level heading or a label. Backticks are code. Italics, bold and capitals are not emphasis. Underlining is not used. A new domain term is defined in Terms added in this part, or in the sentence that first uses it, with no special type.
Names in this notebook, and the Part 4 words they replace:
| name | was, in Parts 1–4 | why the new name |
|---|---|---|
| whereabouts | Room | “room” is also the ordinary word for a room of the apartment |
| occupant | Person | “person” is the ordinary word for the inhabitant |
| bed dust | Dirt in the bedroom | the three dirt factors need names that share no word |
| lounge grime | Dirt in the living room | “lounge” is the living room, so the name does not repeat “living” or “room” |
| bath soil | Dirt in the bathroom | same reason |
| surface | Texture | the modality was already this word; it is not needed as an ordinary word here |
| commotion | Motion | “motion” is the ordinary word for what the sensor detects, and it was also the value |
| pickup | Dirt (the sensor) | “dirt” was already a state factor. The sensor’s modality had to take another name |
| heading | Destination | the values of the control factor are not the same strings as the values of whereabouts |
A value is quoted only when the sentence uses it as a value. “The Roomba starts in the living room” is ordinary prose. The whereabouts is “bedroom” quotes the value. The same strings are in whereabouts_labels, occupant_labels, dust_labels, heading_labels, surface_labels, commotion_labels and pickup_labels, so a search for "living room" finds the text and the code.
The Part 4 factor station is kept under that name when the text refers to the old factor. It is not a variable of this notebook.
Lower case, including after a full stop, is deliberate: a name is an identifier, so the sentence starts with an article rather than with the name. The words of a name are separated by spaces, not hyphens or underscores.
The code brackets follow the same split as the math. ı…ı is a member, _Bˣı0ı for \(\boldsymbol{B}^{\ast(0)}\). I…I is a position along an array. A leading _ means the math symbol is italic bold. L means a tuple. ˣ means the environment. An earlier draft of this part used a “tear-off” reading, in which LAˣ[0] was the way to spell the member. The member now has its own name. LAˣ[whereabouts_n] remains ordinary Julia indexing and returns that same array.
Each choice below is recorded so it is not undone by accident. The probabilities are assumed. When a measurement replaces one, this is the place to change the reason as well as the number in 4.3.4.
Five factors instead of one station factor. Parts 3 and 4 combined the room and the occupancy into six joint values. That cannot say “the person is in the living room” while the Roomba is in the bedroom, and it cannot give each room its own dirt. The split is the point of this part. Occupancy is no longer a property of the Roomba’s room. It is the occupant sharing the whereabouts.
The person is a factor, not an exogenous input. The alternative was an occupancy bit drawn afresh whenever the Roomba entered a room, which is what Part 4 did. A factor that persists lets earlier observations carry over, which is why tracking of “is the person here” can beat the motion sensor. The cost is that the occupant is hidden whenever the Roomba is in another room.
Three dirt factors, cleaned only by presence. Dirt could have been one number for the whole apartment. Per-room factors are what make “clean the living room” a different action from “clean the bedroom”. Cleaning requires the whereabouts to match that room, with \(p_{\text{clean}}=0.8\), so one tick is usually enough and sometimes is not. The accumulation rates \(p_{\text{dirty}}^{[r]}\) are \(0.02\), \(0.05\) and \(0.03\) for the bedroom, the living room and the bathroom. They are assumed. They put the fastest dirt in the room the person uses most, which is the conflict this part is about. The mean time for a clean room to get dirty, if the Roomba stays away, is about \(1/0.02=50\), \(1/0.05=20\) and \(1/0.03\approx 33\) ticks.
The map, and \(p_{\text{move}}=0.9\). There is no door between the bedroom and the living room. A heading from one to the other first goes to the bathroom. This is the Part 4 map, kept so the two parts can be compared. \(p_{\text{move}}=0.9\) is assumed. The alternative, a certain move, would hide the failed-move dots and would make the whereabouts trivial to predict.
Sensor rates, assumed. The camera is correct with probability \(0.9\), and splits the remaining \(0.1\) equally. The motion sensor reports “motion” with probability \(0.8\) when the person is in the room, and \(0.05\) otherwise. pickup reports “dirt picked up” with probability \(0.9\) in a dirty room and \(0.05\) in a clean one. These are round numbers, chosen so each sensor is useful and none is perfect. The light sensor of the earlier parts is left out on purpose.
Preference for “dirt picked up”, not for a clean floor. A preference for “nothing picked up” was the obvious alternative and it points the agent at rooms that are already clean. The weights \(\log 0.9\) against \(\log 0.1\), and \(\log 0.2\) against \(\log 0.8\), make “motion” a stronger pull than “dirt picked up”. That was chosen so the agent would not camp in the living room. A sweep of the two weights is a next step, not a result of this part. \(\log\boldsymbol{C}\) is stored already logged. surface and, for the Part 4 agent, pickup are \(\boldsymbol{0}\) rather than the log of a flat probability vector. The two differ by a constant and select the same actions. The numeric value of \(G_{\Sigma}^{[p]}\) shifts. The heading does not.
Exact joint belief, not a mean field. commotion and pickup each depend on more than one factor, so a product of marginals would drop “the person is in this room” as soon as it was learned. Ninety-six states are small enough to filter exactly. The marginals are computed only for the record and the metrics. A factorized belief is left for a later part.
Horizon \(H=3\), all \(27\) policies. \(H=2\) cannot see a reward that sits two moves away and then one cleaning tick. \(H=3\) can. Policies are enumerated, not rolled up into a tree. A tree would give the same scores. There is no \(\boldsymbol{E}\). Movement is not priced, so the agent does not prefer a short path except through what the path does to the observations.
The agent is given \(\mathbf{A}\) and \(\mathbf{B}\). Learning the dirt rates, or where the person tends to be, would mix exploration into the same objective. This part keeps that out so the two preferences can be read on their own.
The detailed random path uses its own stream, MersenneTwister(7). The environment in that walk uses seed \(42\). The action stream is not 42+1000. It was fixed at \(7\) when the walk-through in 4.3 was written, and changing it would replace every listing in that section. The 30-seed comparison does follow the series protocol: the environment uses MersenneTwister(seed), and the random heading uses MersenneTwister(seed+1000).
What was left out. Doors that stick, a schedule for the person, more than one person, learning, and a factorized belief. Coverage is no longer a primary metric. It is still printed, because a room that is never visited explains a dirt factor that stays “dirty”.