While practising Active Inference making use of RxInfer it would be beneficial to sometimes make use of existing code in Python. For example, using existing data generation code, or, code that simulates the dynamics of the environment can save a lot of time and effort. The PythonCall package in Julia bridges the gap between Julia and Python, enabling seamless integration of Python code within Julia projects. This package is particularly valuable for Julia developers who want to leverage the extensive libraries and tools available in Python without leaving the Julia environment.
With PythonCall, you can directly call Python functions, use Python objects, and manage Python packages as if they were native to Julia. This opens up a world of possibilities, allowing you to combine Julia’s high-performance computing capabilities with Python’s rich ecosystem.
In this project, we will explore how to set up PythonCall, import Python libraries, and execute Python code within Julia. You will see some examples to integrate Python’s functionality into Julia projects, maximizing your productivity and the capabilities of your codebase.
versioninfo() ## Julia versionJulia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 12 × Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
JULIA_NUM_THREADS = To effectively use the PythonCall package in Julia, you need to ensure that your environment is properly set up. This involves installing PythonCall, configuring Python environments, and ensuring that Python packages are accessible. Below are the key steps to set up PythonCall:
First, you need to install the PythonCall package in Julia. Open your Julia REPL and run:
import Pkg; Pkg.instantiate(); Pkg.activate()
Pkg.add(Pkg.PackageSpec(;name="PythonCall"))
using PythonCall Activating project at `~/.julia/environments/v1.10`
Resolving package versions...
No Changes to `~/.julia/environments/v1.10/Project.toml`
No Changes to `~/.julia/environments/v1.10/Manifest.toml`This command installs PythonCall and its dependencies. We also use some other packages.
Pkg.add(Pkg.PackageSpec(;name="Plots"))
Pkg.add(Pkg.PackageSpec(;name="LaTeXStrings"))
using Plots
using LaTeXStrings Resolving package versions...
No Changes to `~/.julia/environments/v1.10/Project.toml`
No Changes to `~/.julia/environments/v1.10/Manifest.toml`
Resolving package versions...
No Changes to `~/.julia/environments/v1.10/Project.toml`
No Changes to `~/.julia/environments/v1.10/Manifest.toml`Pkg.status()Status `~/.julia/environments/v1.10/Project.toml`
[992eb4ea] CondaPkg v0.2.23
[b964fa9f] LaTeXStrings v1.3.1
[91a5bcdd] Plots v1.40.5
[d330b81b] PyPlot v2.11.5
[6099a3de] PythonCall v0.9.22PythonCall requires a working Python installation. By default, PythonCall will use the Python installation that it finds in your system’s PATH. However, you can also specify a different Python installation or create a new one using CondaPkg or other package managers.
You can check which Python is being used by PythonCall with:
# ENV["PYTHON"] = "/home/vscode/.julia/environments/v1.10/.CondaPkg/env/bin/python" # Replace with the path to your Python executable
# Check the Python executable again
python_executable = pyimport("sys").executable
println("Python executable: ", python_executable)Python executable: /home/vscode/.julia/environments/v1.10/.CondaPkg/env/bin/pythonThis will display the path to the Python executable being used by PythonCall.
There are multiple ways to manage Python packages for use with PythonCall:
Using an Existing Python Environment: If you already have a Python environment set up with the required packages, PythonCall can use it directly. Ensure that the environment is active in your shell or specify its path explicitly.
Creating and Managing Python Environments with CondaPkg: You can create isolated Python environments within Julia using CondaPkg. This method is particularly useful if you want to manage dependencies within your Julia project:
using CondaPkg
env = CondaPkg.create()
CondaPkg.add(env, "numpy")Using Virtualenv or Venv: If you prefer to use virtualenv or venv, you can activate the environment and then launch Julia, ensuring PythonCall uses this environment.
Pkg.add(Pkg.PackageSpec(;name="CondaPkg"))
using CondaPkg Resolving package versions...
No Changes to `~/.julia/environments/v1.10/Project.toml`
No Changes to `~/.julia/environments/v1.10/Manifest.toml`CondaPkg.add("numpy")
CondaPkg.add("pandas")
CondaPkg.add("matplotlib"; version="3.9.1")
CondaPkg.status() CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
CondaPkg Dependencies already up to date
CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
CondaPkg Dependencies already up to date
CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
CondaPkg Resolving changes
~ matplotlib
CondaPkg Installing packages
│ /home/vscode/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
│ -r /home/vscode/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
│ install
│ -y
│ -p /home/vscode/.julia/environments/v1.10/.CondaPkg/env
│ --override-channels
│ --no-channel-priority
│ libstdcxx-ng[version='>=3.4,<13.0']
│ matplotlib[version='3.9.1']
│ numpy[version='*']
│ openssl[version='>=3, <3.1']
│ pandas[version='*']
│ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
└ -c conda-forge
CondaPkg Status /home/vscode/.julia/environments/v1.10/CondaPkg.toml
Environment
/home/vscode/.julia/environments/v1.10/.CondaPkg/env
Packages
matplotlib v3.9.1 (3.9.1)
numpy v2.1.0
pandas v2.2.2
Transaction
Prefix: /home/vscode/.julia/environments/v1.10/.CondaPkg/env
Updating specs:
- libstdcxx-ng[version='>=3.4,<13.0']
- matplotlib==3.9.1
- numpy=*
- openssl[version='>=3, <3.1']
- pandas=*
- conda-forge::python[version='>=3.8,<4',build=*cpython*]
Package Version Build Channel Size
─────────────────────────────────────────────────────────────────
Reinstall:
─────────────────────────────────────────────────────────────────
o python 3.11.0 he550d4f_1_cpython conda-forge Cached
Summary:
Reinstall: 1 packages
Total download: 0 B
─────────────────────────────────────────────────────────────────
Transaction starting
Reinstalling python-3.11.0-he550d4f_1_cpython
Transaction finished
To activate this environment, use:
micromamba activate /home/vscode/.julia/environments/v1.10/.CondaPkg/env
Or to execute a single command in this environment, use:
micromamba run -p /home/vscode/.julia/environments/v1.10/.CondaPkg/env mycommand
PythonCall provides a flexible pyconvert function to convert Python objects to Julia types. For instance, converting a Python list to a Julia array:
py_list = Py([1, 2, 3])
julia_array = pyconvert(Vector{Int}, py_list)3-element Vector{Int64}:
1
2
3This example shows how to convert a Python list to a Julia vector using pyconvert.
PythonCall preserves the mutability of objects. For example, appending to a Python list from Julia:
x = ["foo", "bar"]
Py(x).append("baz")
@show xx = ["foo", "bar", "baz"]3-element Vector{String}:
"foo"
"bar"
"baz"In this example, the Python list is directly modified, and the changes are reflected in the original Julia object. These examples demonstrate the basic functionality of PythonCall, including module importation, type conversion, and mutability preservation.
re packagere = pyimport("re")Python: <module 're' from '/home/vscode/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/re/__init__.py'>words = re.findall("[a-zA-Z]+", "PythonCall.jl is very useful!")Python: ['PythonCall', 'jl', 'is', 'very', 'useful']sentence = Py(" ").join(words)Python: 'PythonCall jl is very useful'jlstr = pyconvert(String, sentence)"PythonCall jl is very useful"x = pylist([3, 4, 5])Python: [3, 4, 5]x[2] = 66pyslice(0, 2)Python: slice(0, 2, None)pylist([1, 2])Python: [1, 2]x[pyslice(0,2)] + pylist([1,2])Python: [3, 4, 1, 2]math packagemath = pyimport("math")
math.sqrt(16)Python: 4.0numpy packagenp = pyimport("numpy")
np.array([1, 2, 3])Python: array([1, 2, 3])The code for the local module called my_module.py is:
class MyClass:
def __init__(self, name):
self.name = name
def greet(self):
return f"Hello, {self.name}!"
def greet_fn():
return "I greet you!"## Import the my_module module
sys = pyimport("sys")
sys.path.append(".")
my_module = pyimport("my_module") ## import module
MyClass = pyimport("my_module").MyClass ## import MyClass
greet_fn = pyimport("my_module").greet_fn ## import greet_fnPython: <function greet_fn at 0x7f698a1f49a0>my_module.MyClass("Bob").greet()Python: 'Hello, Bob!'instance = MyClass("Alice")
instance.greet()Python: 'Hello, Alice!'greet_fn()Python: 'I greet you!'PriceSimulatorThe code for the local module called price_simulator.py is:
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
import matplotlib as mpl
eNAMES = ['AAA', 'BBB']
SEED_TRAIN = 77777777
x = ['Up', 'Neutral', 'Down']
W_BIAS_CDFS = pd.DataFrame(
[[.9, 1., 1.], #'Up' cdf
[.2, .8, 1.], #'Neutral' cdf
[0., .1, 1.]], #'Down' cdf
index=x,
columns=x,
)
W_UP_STEP = 1
W_DOWN_STEP = -1
W_VARIANCE = 2
class PriceSimulator():
def __init__(self,
biasCdfs=W_BIAS_CDFS,
upStep=W_UP_STEP,
downStep=W_DOWN_STEP,
variance=W_VARIANCE,
seed=None):
self.biasCdfs = biasCdfs
self.upStep = upStep
self.downStep = downStep
self.variance = variance
self.prng = np.random.RandomState(seed)
self.bias = 'Neutral'
def simulate(self): #assume the change in price is normal with mean bias and variance 2
phat_tt1_dict = {}
b_tt1_dict = {}
b_tt1_val_dict = {}
for e in eNAMES:
b_t = self.prng.choice(['Down', 'Neutral', 'Up'])
biasCdf = self.biasCdfs.loc[[b_t]]
coin = self.prng.random_sample()
if (coin < float(biasCdf['Up'].iloc[0])):
b_tt1 = 'Up' #new bias
b_tt1_val = self.upStep #bias
elif (coin >= float(biasCdf['Up'].iloc[0]) and coin < float(biasCdf['Neutral'].iloc[0])): #.
b_tt1 = 'Neutral' #new bias
b_tt1_val = 0 #bias
else:
b_tt1 = 'Down' #new bias
b_tt1_val = self.downStep #bias
self.bias = b_tt1
# phat_tt1 = self.prng.normal(b_tt1_val, self.variance) #change in price
phat_tt1_dict[e] = self.prng.normal(b_tt1_val, self.variance) #change in price
b_tt1_dict[e] = b_tt1
b_tt1_val_dict[e] = b_tt1_val
W_tt1 = {
"p_t": {e: phat_tt1_dict[e] for e in eNAMES},
"b_t": {e: b_tt1_dict[e] for e in eNAMES}, #just for display
"b_t_val": {e: b_tt1_val_dict[e] for e in eNAMES} #just for display
}
return W_tt1
def plot_output(self, df1):
n_charts = 3
ylabelsize = 16
mpl.rcParams['lines.linewidth'] = 1.2
default_colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
fig, axs = plt.subplots(n_charts, sharex=True)
fig.set_figwidth(13); fig.set_figheight(9)
fig.suptitle('Price Simulation', fontsize=20)
xi = 0
axs[xi].set_title(f'')
axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
for e in eNAMES:
axs[xi].step(df1[f'p_t_{e}'], random.choice(default_colors), where='post')
axs[xi].axhline(y=0, color='k', linestyle=':')
y1ab = '$p_{t}$'
axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)
xi = 1
axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
for e in eNAMES:
axs[xi].step(df1[f'b_t_val_{e}'], random.choice(default_colors), where='post')
axs[xi].axhline(y=0, color='k', linestyle=':')
y1ab = '$b_{t,val}$'
axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)
xi = 2
axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
for e in eNAMES:
axs[xi].step(df1[f'b_t_{e}'], random.choice(default_colors), where='post')
y1ab = '$b_{t}$'
axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)
axs[xi].set_xlabel('$t$', rotation=0, ha='center', va='center', fontweight='bold', size=ylabelsize)
def create_data(T__sim):
price_sim = PriceSimulator(seed=SEED_TRAIN)
PriceData = []
for i in range(T__sim):
res = price_sim.simulate()
entry = [itm[1][e] for itm in list(res.items()) for e in eNAMES]
PriceData.append(entry)
labels = [f'{itm[0]}_{e}' for itm in list(res.items()) for e in eNAMES]
df = pd.DataFrame.from_records(data=PriceData, columns=labels); df[:10]
return df## Import the price_simulator module
sys = pyimport("sys")
sys.path.append(".")
price_simulator = pyimport("price_simulator") ## import module
PriceSimulator = pyimport("price_simulator").PriceSimulator ## import PriceSimulator
create_data = pyimport("price_simulator").create_dataPython: <function create_data at 0x7f698238b920>pydf = create_data(20)| p_t_AAA | p_t_BBB | b_t_AAA | b_t_BBB | b_t_val_AAA | b_t_val_BBB | |
|---|---|---|---|---|---|---|
| 0 | 0.336515 | 3.108396 | Up | Neutral | 1 | 0 |
| 1 | 2.095779 | 1.136877 | Neutral | Neutral | 0 | 0 |
| 2 | 0.722554 | -1.143349 | Up | Up | 1 | 1 |
| 3 | 3.939653 | -0.485803 | Neutral | Down | 0 | -1 |
| 4 | 2.181077 | 0.421848 | Down | Down | -1 | -1 |
| 5 | -2.378513 | -0.016214 | Up | Up | 1 | 1 |
| 6 | -3.990976 | -0.327373 | Neutral | Down | 0 | -1 |
| 7 | -3.532132 | 3.986402 | Down | Up | -1 | 1 |
| 8 | -3.187230 | 1.223801 | Down | Up | -1 | 1 |
| 9 | -0.191991 | 0.716637 | Up | Up | 1 | 1 |
| 10 | 5.916625 | 0.756212 | Down | Down | -1 | -1 |
| 11 | 3.954901 | 2.564203 | Down | Up | -1 | 1 |
| 12 | 0.656361 | 0.518275 | Up | Neutral | 1 | 0 |
| 13 | 0.026968 | 3.311696 | Up | Up | 1 | 1 |
| 14 | -0.109170 | 2.463019 | Neutral | Up | 0 | 1 |
| 15 | 2.646287 | 1.938792 | Up | Up | 1 | 1 |
| 16 | 5.214935 | 5.014522 | Up | Up | 1 | 1 |
| 17 | -0.092501 | -3.121095 | Neutral | Down | 0 | -1 |
| 18 | 4.212627 | -2.043299 | Up | Up | 1 | 1 |
| 19 | 1.939671 | 0.207026 | Up | Down | 1 | -1 |
typeof(pydf)Py_p_t_AAA = pyconvert(Vector, pydf["p_t_AAA"])
_p_t_BBB = pyconvert(Vector, pydf["p_t_BBB"])
_b_t_val_AAA = pyconvert(Vector, pydf["b_t_val_AAA"])
_b_t_val_BBB = pyconvert(Vector, pydf["b_t_val_BBB"])
_b_t_AAA = pyconvert(Vector{String}, pydf["b_t_AAA"])
_b_t_BBB = pyconvert(Vector{String}, pydf["b_t_BBB"])20-element Vector{String}:
"Neutral"
"Neutral"
"Up"
"Down"
"Down"
"Up"
"Down"
"Up"
"Up"
"Up"
"Down"
"Up"
"Neutral"
"Up"
"Up"
"Up"
"Up"
"Down"
"Up"
"Down"_p1 = plot(
_p_t_AAA,
title= "Simulated data from Python code",
size=(700, 600),
seriestype=:step,
ylabel= "price\n"*L"p_t",
yguidefontrotation=-90,
left_margin=15Plots.mm,
color= "red",
label= "AAA",
legend=:topright,
)
_p1 = plot!(
_p_t_BBB,
seriestype=:step,
color= "green",
label= "BBB",
)
_p2 = plot(
_b_t_val_AAA,
seriestype=:step,
color= "red",
ylabel= "bias value\n"*L"b_{t,val}",
yguidefontrotation=-90,
legend= :none,
)
_p2 = plot!(
_b_t_val_BBB,
seriestype=:step,
color= "green",
legend= :none,
)
_p3 = scatter(
_b_t_AAA,
color= "red",
ylabel= "bias\n"*L"b_t",
yguidefontrotation=-90,
legend= :none,
)
_p3 = scatter!(
_b_t_BBB,
xlabel= L"time $t$",
color= "green",
legend= :none,
)
plot(_p1, _p2, _p3, layout= @layout([a; b; c]))