PORTFOLIO OF PROJECTS - Julia to Python

While practising Active Inference making use of RxInfer it would be beneficial to sometimes make use of existing code in Python. For example, using existing data generation code, or, code that simulates the dynamics of the environment can save a lot of time and effort. The PythonCall package in Julia bridges the gap between Julia and Python, enabling seamless integration of Python code within Julia projects. This package is particularly valuable for Julia developers who want to leverage the extensive libraries and tools available in Python without leaving the Julia environment.

With PythonCall, you can directly call Python functions, use Python objects, and manage Python packages as if they were native to Julia. This opens up a world of possibilities, allowing you to combine Julia’s high-performance computing capabilities with Python’s rich ecosystem.

In this project, we will explore how to set up PythonCall, import Python libraries, and execute Python code within Julia. You will see some examples to integrate Python’s functionality into Julia projects, maximizing your productivity and the capabilities of your codebase.

versioninfo() ## Julia version

Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
  JULIA_NUM_THREADS =

Setting Up PythonCall in Julia

To effectively use the PythonCall package in Julia, you need to ensure that your environment is properly set up. This involves installing PythonCall, configuring Python environments, and ensuring that Python packages are accessible. Below are the key steps to set up PythonCall:

1. Install PythonCall

First, you need to install the PythonCall package in Julia. Open your Julia REPL and run:

import Pkg; Pkg.instantiate(); Pkg.activate()
Pkg.add(Pkg.PackageSpec(;name="PythonCall"))

using PythonCall

  Activating project at `~/.julia/environments/v1.10`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

This command installs PythonCall and its dependencies. We also use some other packages.

Pkg.add(Pkg.PackageSpec(;name="Plots"))
Pkg.add(Pkg.PackageSpec(;name="LaTeXStrings"))

using Plots
using LaTeXStrings

   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

Pkg.status()

Status `~/.julia/environments/v1.10/Project.toml`
  [992eb4ea] CondaPkg v0.2.23
  [b964fa9f] LaTeXStrings v1.3.1
  [91a5bcdd] Plots v1.40.5
  [d330b81b] PyPlot v2.11.5
  [6099a3de] PythonCall v0.9.22

2. Verify Python Installation

PythonCall requires a working Python installation. By default, PythonCall will use the Python installation that it finds in your system’s PATH. However, you can also specify a different Python installation or create a new one using CondaPkg or other package managers.

You can check which Python is being used by PythonCall with:

# ENV["PYTHON"] = "/home/vscode/.julia/environments/v1.10/.CondaPkg/env/bin/python"  # Replace with the path to your Python executable

# Check the Python executable again
python_executable = pyimport("sys").executable
println("Python executable: ", python_executable)

Python executable: /home/vscode/.julia/environments/v1.10/.CondaPkg/env/bin/python

This will display the path to the Python executable being used by PythonCall.

3. Managing Python Packages

There are multiple ways to manage Python packages for use with PythonCall:

Using an Existing Python Environment: If you already have a Python environment set up with the required packages, PythonCall can use it directly. Ensure that the environment is active in your shell or specify its path explicitly.
Creating and Managing Python Environments with CondaPkg: You can create isolated Python environments within Julia using CondaPkg. This method is particularly useful if you want to manage dependencies within your Julia project:
```
using CondaPkg
env = CondaPkg.create()
CondaPkg.add(env, "numpy")
```
Using Virtualenv or Venv: If you prefer to use virtualenv or venv, you can activate the environment and then launch Julia, ensuring PythonCall uses this environment.

Pkg.add(Pkg.PackageSpec(;name="CondaPkg"))
using CondaPkg

   Resolving package versions...
  No Changes to `~/.julia/environments/v1.10/Project.toml`
  No Changes to `~/.julia/environments/v1.10/Manifest.toml`

CondaPkg.add("numpy")
CondaPkg.add("pandas")
CondaPkg.add("matplotlib"; version="3.9.1")
CondaPkg.status()

    CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
    CondaPkg Dependencies already up to date
    CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
    CondaPkg Dependencies already up to date
    CondaPkg Found dependencies: /home/vscode/.julia/environments/v1.10/CondaPkg.toml
    CondaPkg Found dependencies: /home/vscode/.julia/packages/PythonCall/flx5V/CondaPkg.toml
    CondaPkg Resolving changes
             ~ matplotlib
    CondaPkg Installing packages
             │ /home/vscode/.julia/artifacts/7973f2c7725e2d0eef7a95159454c4145f0945a2/bin/micromamba
             │ -r /home/vscode/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
             │ install
             │ -y
             │ -p /home/vscode/.julia/environments/v1.10/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0']
             │ matplotlib[version='3.9.1']
             │ numpy[version='*']
             │ openssl[version='>=3, <3.1']
             │ pandas[version='*']
             │ python[version='>=3.8,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge
CondaPkg Status /home/vscode/.julia/environments/v1.10/CondaPkg.toml
Environment
  /home/vscode/.julia/environments/v1.10/.CondaPkg/env
Packages
  matplotlib v3.9.1 (3.9.1)
  numpy v2.1.0
  pandas v2.2.2


Transaction

  Prefix: /home/vscode/.julia/environments/v1.10/.CondaPkg/env

  Updating specs:

   - libstdcxx-ng[version='>=3.4,<13.0']
   - matplotlib==3.9.1
   - numpy=*
   - openssl[version='>=3, <3.1']
   - pandas=*
   - conda-forge::python[version='>=3.8,<4',build=*cpython*]


  Package   Version  Build               Channel           Size
─────────────────────────────────────────────────────────────────
  Reinstall:
─────────────────────────────────────────────────────────────────

  o python   3.11.0  he550d4f_1_cpython  conda-forge     Cached

  Summary:

  Reinstall: 1 packages

  Total download: 0 B

─────────────────────────────────────────────────────────────────



Transaction starting
Reinstalling python-3.11.0-he550d4f_1_cpython

Transaction finished

To activate this environment, use:

    micromamba activate /home/vscode/.julia/environments/v1.10/.CondaPkg/env

Or to execute a single command in this environment, use:

    micromamba run -p /home/vscode/.julia/environments/v1.10/.CondaPkg/env mycommand

Examples

1. Converting Python Objects

PythonCall provides a flexible pyconvert function to convert Python objects to Julia types. For instance, converting a Python list to a Julia array:

py_list = Py([1, 2, 3])
julia_array = pyconvert(Vector{Int}, py_list)

3-element Vector{Int64}:
 1
 2
 3

This example shows how to convert a Python list to a Julia vector using pyconvert.

2. Mutability Preservation

PythonCall preserves the mutability of objects. For example, appending to a Python list from Julia:

x = ["foo", "bar"]
Py(x).append("baz")
@show x

x = ["foo", "bar", "baz"]

3-element Vector{String}:
 "foo"
 "bar"
 "baz"

In this example, the Python list is directly modified, and the changes are reflected in the original Julia object. These examples demonstrate the basic functionality of PythonCall, including module importation, type conversion, and mutability preservation.

3. Using the `re` package

re = pyimport("re")

Python: <module 're' from '/home/vscode/.julia/environments/v1.10/.CondaPkg/env/lib/python3.11/re/__init__.py'>

words = re.findall("[a-zA-Z]+", "PythonCall.jl is very useful!")

Python: ['PythonCall', 'jl', 'is', 'very', 'useful']

sentence = Py(" ").join(words)

Python: 'PythonCall jl is very useful'

jlstr = pyconvert(String, sentence)

"PythonCall jl is very useful"

4. Python lists and slices

x = pylist([3, 4, 5])

Python: [3, 4, 5]

x[2] = 6

pyslice(0, 2)

Python: slice(0, 2, None)

pylist([1, 2])

Python: [1, 2]

x[pyslice(0,2)] + pylist([1,2])

Python: [3, 4, 1, 2]

5. Using the `math` package

math = pyimport("math")
math.sqrt(16)

Python: 4.0

6. Using the `numpy` package

np = pyimport("numpy")
np.array([1, 2, 3])

Python: array([1, 2, 3])

7. Using a local module

The code for the local module called my_module.py is:

class MyClass:
    def __init__(self, name):
        self.name = name

    def greet(self):
        return f"Hello, {self.name}!"

def greet_fn():
    return "I greet you!"

## Import the my_module module
sys = pyimport("sys")
sys.path.append(".")

my_module = pyimport("my_module") ## import module
MyClass = pyimport("my_module").MyClass ## import MyClass
greet_fn = pyimport("my_module").greet_fn ## import greet_fn

Python: <function greet_fn at 0x7f698a1f49a0>

my_module.MyClass("Bob").greet()

Python: 'Hello, Bob!'

instance = MyClass("Alice")
instance.greet()

Python: 'Hello, Alice!'

greet_fn()

Python: 'I greet you!'

8. Using a `PriceSimulator`

The code for the local module called price_simulator.py is:

import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
import matplotlib as mpl

eNAMES = ['AAA', 'BBB']
SEED_TRAIN = 77777777
x = ['Up', 'Neutral', 'Down']
W_BIAS_CDFS = pd.DataFrame(
  [[.9, 1., 1.], #'Up' cdf
   [.2, .8, 1.], #'Neutral' cdf
   [0., .1, 1.]], #'Down' cdf
  index=x,
  columns=x,
)
W_UP_STEP = 1
W_DOWN_STEP = -1
W_VARIANCE = 2

class PriceSimulator():
  def __init__(self,
    biasCdfs=W_BIAS_CDFS,
    upStep=W_UP_STEP,
    downStep=W_DOWN_STEP,
    variance=W_VARIANCE,
    seed=None):

    self.biasCdfs = biasCdfs
    self.upStep = upStep
    self.downStep = downStep
    self.variance = variance
    self.prng = np.random.RandomState(seed)
    self.bias = 'Neutral'

  def simulate(self): #assume the change in price is normal with mean bias and variance 2
    phat_tt1_dict = {}
    b_tt1_dict = {}
    b_tt1_val_dict = {}
    for e in eNAMES:
      b_t = self.prng.choice(['Down', 'Neutral', 'Up'])
      biasCdf = self.biasCdfs.loc[[b_t]]
      coin = self.prng.random_sample()
      if (coin < float(biasCdf['Up'].iloc[0])):  
        b_tt1 = 'Up' #new bias
        b_tt1_val = self.upStep #bias
      elif (coin >= float(biasCdf['Up'].iloc[0]) and coin < float(biasCdf['Neutral'].iloc[0])): #.
        b_tt1 = 'Neutral' #new bias
        b_tt1_val = 0 #bias
      else:
        b_tt1 = 'Down' #new bias
        b_tt1_val = self.downStep #bias
      self.bias = b_tt1
      # phat_tt1 = self.prng.normal(b_tt1_val, self.variance) #change in price
      phat_tt1_dict[e] = self.prng.normal(b_tt1_val, self.variance) #change in price
      b_tt1_dict[e] = b_tt1
      b_tt1_val_dict[e] = b_tt1_val
    W_tt1 = {
        "p_t": {e: phat_tt1_dict[e] for e in eNAMES},
        "b_t": {e: b_tt1_dict[e] for e in eNAMES}, #just for display
        "b_t_val": {e: b_tt1_val_dict[e] for e in eNAMES} #just for display
    }
    return W_tt1

  def plot_output(self, df1):
    n_charts = 3
    ylabelsize = 16
    mpl.rcParams['lines.linewidth'] = 1.2
    default_colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
    fig, axs = plt.subplots(n_charts, sharex=True)
    fig.set_figwidth(13); fig.set_figheight(9)
    fig.suptitle('Price Simulation', fontsize=20)

    xi = 0
    axs[xi].set_title(f'')
    axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
    for e in eNAMES:
      axs[xi].step(df1[f'p_t_{e}'], random.choice(default_colors), where='post')
    axs[xi].axhline(y=0, color='k', linestyle=':')
    y1ab = '$p_{t}$'
    axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)

    xi = 1
    axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
    for e in eNAMES:
      axs[xi].step(df1[f'b_t_val_{e}'], random.choice(default_colors), where='post')
    axs[xi].axhline(y=0, color='k', linestyle=':')
    y1ab = '$b_{t,val}$'
    axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)

    xi = 2
    axs[xi].set_ylim(auto=True); axs[xi].spines['top'].set_visible(False); axs[xi].spines['right'].set_visible(True); axs[xi].spines['bottom'].set_visible(False)
    for e in eNAMES:
      axs[xi].step(df1[f'b_t_{e}'], random.choice(default_colors), where='post')
    y1ab = '$b_{t}$'
    axs[xi].set_ylabel(y1ab, rotation=0, ha='right', va='center', fontweight='bold', size=ylabelsize)

    axs[xi].set_xlabel('$t$', rotation=0, ha='center', va='center', fontweight='bold', size=ylabelsize)

def create_data(T__sim):
  price_sim = PriceSimulator(seed=SEED_TRAIN)
  PriceData = []
  for i in range(T__sim):
    res = price_sim.simulate()
    entry = [itm[1][e] for itm in list(res.items()) for e in eNAMES]
    PriceData.append(entry)
  labels = [f'{itm[0]}_{e}' for itm in list(res.items()) for e in eNAMES]
  df = pd.DataFrame.from_records(data=PriceData, columns=labels); df[:10]
  return df

## Import the price_simulator module
sys = pyimport("sys")
sys.path.append(".")

price_simulator = pyimport("price_simulator") ## import module
PriceSimulator = pyimport("price_simulator").PriceSimulator ## import PriceSimulator
create_data = pyimport("price_simulator").create_data

Python: <function create_data at 0x7f698238b920>

pydf = create_data(20)

	p_t_AAA	p_t_BBB	b_t_AAA	b_t_BBB	b_t_val_AAA	b_t_val_BBB
0	0.336515	3.108396	Up	Neutral	1	0
1	2.095779	1.136877	Neutral	Neutral	0	0
2	0.722554	-1.143349	Up	Up	1	1
3	3.939653	-0.485803	Neutral	Down	0	-1
4	2.181077	0.421848	Down	Down	-1	-1
5	-2.378513	-0.016214	Up	Up	1	1
6	-3.990976	-0.327373	Neutral	Down	0	-1
7	-3.532132	3.986402	Down	Up	-1	1
8	-3.187230	1.223801	Down	Up	-1	1
9	-0.191991	0.716637	Up	Up	1	1
10	5.916625	0.756212	Down	Down	-1	-1
11	3.954901	2.564203	Down	Up	-1	1
12	0.656361	0.518275	Up	Neutral	1	0
13	0.026968	3.311696	Up	Up	1	1
14	-0.109170	2.463019	Neutral	Up	0	1
15	2.646287	1.938792	Up	Up	1	1
16	5.214935	5.014522	Up	Up	1	1
17	-0.092501	-3.121095	Neutral	Down	0	-1
18	4.212627	-2.043299	Up	Up	1	1
19	1.939671	0.207026	Up	Down	1	-1

typeof(pydf)

Py

_p_t_AAA = pyconvert(Vector, pydf["p_t_AAA"])
_p_t_BBB = pyconvert(Vector, pydf["p_t_BBB"])
_b_t_val_AAA = pyconvert(Vector, pydf["b_t_val_AAA"])
_b_t_val_BBB = pyconvert(Vector, pydf["b_t_val_BBB"])
_b_t_AAA = pyconvert(Vector{String}, pydf["b_t_AAA"])
_b_t_BBB = pyconvert(Vector{String}, pydf["b_t_BBB"])

20-element Vector{String}:
 "Neutral"
 "Neutral"
 "Up"
 "Down"
 "Down"
 "Up"
 "Down"
 "Up"
 "Up"
 "Up"
 "Down"
 "Up"
 "Neutral"
 "Up"
 "Up"
 "Up"
 "Up"
 "Down"
 "Up"
 "Down"

_p1 = plot(
    _p_t_AAA,
    title= "Simulated data from Python code",
    size=(700, 600),
    seriestype=:step,
    ylabel= "price\n"*L"p_t",
    yguidefontrotation=-90,
    left_margin=15Plots.mm,
    color= "red",
    label= "AAA",
    legend=:topright,
)
_p1 = plot!(
    _p_t_BBB,
    seriestype=:step,
    color= "green",
    label= "BBB",   
)

_p2 = plot(
    _b_t_val_AAA,
    seriestype=:step,
    color= "red",
    ylabel= "bias value\n"*L"b_{t,val}",
    yguidefontrotation=-90,
    legend= :none,
)
_p2 = plot!(
    _b_t_val_BBB,
    seriestype=:step,
    color= "green",
    legend= :none,
)

_p3 = scatter(
    _b_t_AAA,
    color= "red",
    ylabel= "bias\n"*L"b_t",
    yguidefontrotation=-90,
    legend= :none,
)
_p3 = scatter!(
    _b_t_BBB,
    xlabel= L"time $t$",
    color= "green",
    legend= :none,
)

plot(_p1, _p2, _p3, layout= @layout([a; b; c]))