py2mat.m and mat2py.m Performance Improvements

Introduction

Last week a Github user, https://github.com/hcommin, posted a pair of issues, #4 and #5 on my book’s Github repo showing ways to make py2mat.m and mat2py.m run much faster.

TL;DR

The method I used to import Python modules numpy, datetime, and scipy.sparse into the two MATLAB scripts was suboptimal. As a result, both scripts suffered unnecessarily large load latencies. Directly using a Python module via MATLAB’s py. is faster than explicitly importing the module. For example, x = py.numpy.array(y) is much faster than np = py.importlib.import_module('numpy'); x = np.array(y) even if np is defined only once at the top of the MATLAB program.

Invoke Python modules directly via py. instead of explicitly importing them

https://github.com/hcommin profiled py2mat.m and mat2py.m and found that most of the time was spent loading Python modules into MATLAB via py.importlib.import_module().

Original, slow versions

py2mat.m and mat2py.m used to import and reference Python modules like this:

py2mat.py:

    Im = @py.importlib.import_module;
    np = Im('numpy');
           :
    x_mat = int64(x_py.astype(np.float64));

mat2py.m:

    Im = @py.importlib.import_module;
    np = Im('numpy');
    sp = Im('scipy.sparse');
    dt = Im('datetime');
    tz = Im('dateutil.tz');
             :
    x_py = np.array(x_mat);

New, fast versions

It turns out that explicitly importing Python modules via py.importlib.import_module()—at least for the modules above—is not only unnecessary, but is also slower than using the modules directly via, for example, py.numpy.* The faster versions of py2mat.m and mat2py.m committed on Sept. 15, 2023 no longer call py.importlib.import_module() and dispense with the np, sp, dt, and t_py aliases. Their equivalent lines of code are

py2mat.py:

    x_mat = int64(x_py.astype('float64'));

mat2py.py:

	x_py = py.numpy.array(x_mat);

Benchmarks

The seemingly small code changes have a large impact on performance, esprecially when the utilities are called frequently. These MATLAB code snippets were use to measure before and after conversion times. The tests were run with MATLAB 2022b on Linux (Ubuntu 2020).

py2mat.py:

x_mat = rand(1000);
x_py = mat2py(x_mat);

tic;
for i = 1:100
    y_mat = old_py2mat(x_py);
end
fprintf('OLD time to convert rand(1000) to MATLAB= %.3f s\n', toc)

tic;
for i = 1:100
    y_mat = py2mat(x_py);
end
fprintf('NEW time to convert rand(1000) to MATLAB= %.3f s\n', toc)

clear x_mat
x_mat.a = {'abc', [12, 13]};
x_mat.b = rand(5);
x_py = mat2py(x_mat);

tic;
for i = 1:100
    y_mat = old_py2mat(x_py);
end
fprintf('OLD time to convert struct to MATLAB    = %.3f s\n', toc)

tic;
for i = 1:100
    y_mat = py2mat(x_py);
end
fprintf('NEW time to convert struct to MATLAB    = %.3f s\n', toc)

Python to MATLAB matrix conversion saw a 3x increase while converting a structured variable from Python to MATLAB was nearly 8x faster:

OLD time to convert rand(1000) to MATLAB= 0.976 s
NEW time to convert rand(1000) to MATLAB= 0.302 s
OLD time to convert struct to MATLAB    = 0.779 s
NEW time to convert struct to MATLAB    = 0.099 s

Next, going in the other direction, from MATLAB to Python variables:

mat2py.py:

x_mat = rand(1000);

tic;
for i = 1:100
    y_py = old_mat2py(x_mat);
end
fprintf('OLD time to convert rand(1000) to Python= %.3f s\n', toc)

tic;
for i = 1:100
    y_py = mat2py(x_mat);
end
fprintf('NEW time to convert rand(1000) to Python= %.3f s\n', toc)

clear x_mat
x_mat.a = {'abc', [12, 13]};
x_mat.b = rand(5);

tic;
for i = 1:100
    y_py = old_mat2py(x_mat);
end
fprintf('OLD time to convert struct to Python    = %.3f s\n', toc)

tic;
for i = 1:100
    y_py = mat2py(x_mat);
end
fprintf('NEW time to convert struct to Python    = %.3f s\n', toc)

The performance boost is even more noticeable in the MATLAB to Python direction:

OLD time to convert rand(1000) to Python= 0.953 s
NEW time to convert rand(1000) to Python= 0.120 s
OLD time to convert struct to Python    = 0.968 s
NEW time to convert struct to Python    = 0.140 s