Recent Posts (page 8 / 8)

by Al Danial

Set up a MATLAB+Python environment

Python MATLAB conda

Part 1 of the Python Is The Ultimate MATLAB Toolbox series.

What you’ll need

  • MATLAB version 2020b or newer (some MATLAB versions older than 2020b support Python, but the farther back you go, the more more limitations and problems you’ll hit). The latest release, 2022a, is definitely recommended.
  • A Python installation supported by your version of MATLAB. I strongly recommend the Anaconda Python distribution 1 for two reasons: 1) it comes with a deep stack of Python modules and external tools useful for scientific computing and 2) its conda command has the ability to create virtual environments with specific shared library versions. Control over shared library versions in Python is invaluable for troubleshooting MATLAB errors (or crashes) when importing complex modules such as NumPy, SciPy, and statsmodels (among others) in MATLAB.

Step 1: pyenv

The first command to issue in a MATLAB console is pyenv. It will either come up empty, or it will print information about the Python installation MATLAB found:

>> pyenv
ans = 

  PythonEnvironment with properties:

          Version: "3.9"
       Executable: "/usr/local/anaconda3/2021.05/envs/matpy/bin/python"
          Library: "/usr/local/anaconda3/2021.05/envs/matpy/lib/libpython3.9.so"
             Home: "/usr/local/anaconda3/2021.05/envs/matpy"
           Status: Loaded
    ExecutionMode: InProcess
        ProcessID: "840228"

If pyenv comes up empty, set 'Version' to the path of the Python executable you want to use:

>> pyenv('Version', "/usr/local/anaconda3/2021.05/envs/matpy/bin/python")

On a Windows computer your entry might be

>> pyenv('Version', "C:/ProgramData/Anaconda3/python.exe")

(Yes, forward slashes work in Windows too.)

The best way to have MATLAB recognize a specific virtual environment is to first activate the virtual environment in a terminal, then start MATLAB from that terminal.

Step 2: Does it work?

A quick way to determine if your MATLAB and Python environments are compatible is to import the NumPy module into MATLAB:

>> np = py.importlib.import_module('numpy')

NumPy and MATLAB have several shared libraries in common so importing NumPy into MATLAB is a good litmus test for overall MATLAB/Python compatibility. Several things may happen:

  1. A long list of function names appears.
  2. Python Error: ImportError
  3. Unable to resolve the name numpy
  4. Segmentation violation detected

The first case indicates the import was successful. If you see the functions, congratulations, you should be able to use MATLAB and Python together effectively.

The last three errors can be caused by a Python installation that is not version-consistent with, or not visible to MATLAB; the Python installation does not have NumPy installed; or you need to create a virtual environment configured with shared library versions that are consistent with MATLAB’s shared libraries.

Possible work-arounds to library conflicts:

  1. Change the ExecutionMode setting to OutOfProcess ExecutionMode: InProcess 

    >> pyenv("ExecutionMode","OutOfProcess")

  2. Create a conda environment with library versions that more closely match MATLAB’s. Unfortunately there’s no obvious way to identify either the libraries or the versions so experimentation is needed. The conda environment that works well for me with MATLAB 2020b is

    Linux (Ubuntu 20.04) + MATLAB 2020b 

    conda create --name matpy python=3.8.8 libgcc-devel_linux-64=8.4.0  \
    libgcc-ng=8.4.0 geopandas matplotlib pulp cartopy austin faker  \
    pint poliastro psycopg2 pymongo pythran redis redis-py simanneal \
    netCDF4 descartes h5py statsmodels pyyaml psutil lxml dask       \
    distributed paramiko sympy requests pyflakes uncertainties seaborn \
    cython pytest scikit-umfpack ipython -c conda-forge

    Windows 10 + MATLAB 2020b 

    conda create --name matpy python=3.8.8 geopandas matplotlib pulp ^
    cartopy faker pint poliastro psycopg2 pymongo pythran redis  ^
    redis-py statsmodels simanneal netCDF4 descartes h5py pyyaml ^
    psutil lxml dask distributed paramiko sympy requests pyflakes ^
    cython uncertainties seaborn pytest ipython -c conda-forge

    macOS (Catalina, Big Sur) + MATLAB 2020b 

    conda create --name matpy python=3.8.8 geopandas matplotlib pulp \
    cartopy austin faker intel-openmp=2021.2.0 llvm-openmp=10.0.0 \
    libllvm10=10.0.1 llvmlite=0.36.0 \
    pint poliastro psycopg2 pymongo pythran redis redis-py simanneal \
    netCDF4 descartes h5py statsmodels pyyaml psutil lxml dask       \
    distributed paramiko sympy requests pyflakes uncertainties seaborn \
    cython pytest ipython -c conda-forge

I’ve found MATLAB 2022a to work cleanly with the base Anaconda 2021.05 installation; no finely-tuned conda environment needed.

  1. free for individual use but requires a purchased license for commercial use ↩︎

by Al Danial

Build gRPC Plugins on Ubuntu 20.04 with Anaconda

gRPC Anaconda Python C++ C# node Objective C PHP Ruby

gRPC Logo

A frustrating aspect of working with gRPC in multiple languages is that you need each language’s gRPC plugin. One can get all supported plugins by building gRPC from source but that’s a challenging endeavour. If you can build gRPC from source, this post won’t interest you. Everyone else, read on.

There’s a shortcut to creating a gRPC build environment with the needed build dependendies: install the Anaconda Python distribution (if you don’t already have it), then create a conda environment that includes Protobuf, Abseil-Cpp, and re2. The resulting conda environment lets you build gRPC plugins for Python, C++, C#, node, Objective C, PHP, and Ruby. The process goes like this:

  1. Install Anaconda if you don’t already have it.

  2. Create a conda environment that includes the gRPC Python package and the dependencies needed to build from source:

     conda create --name grpc_env grpcio protobuf abseil-cpp re2

    I’m explicitly excluding grpcio-tools since the Python plugin will be created by the process described here. Pay attention to the grpcio version before you hit y to begin the installation. In my case the conda create command shows

       Package                Version  Build           Channel                  Size
 ─────────────────────────────────────────────────────────────────────────────────
   Install:
 ─────────────────────────────────────────────────────────────────────────────────
   + _libgcc_mutex            0.1  main            pkgs/main/linux-64     Cached
   + _openmp_mutex            4.5  1_gnu           pkgs/main/linux-64     Cached
   + abseil-cpp        20210324.2  h2531618_0      pkgs/main/linux-64     965 KB
   + c-ares                1.18.1  h7f8727e_0      pkgs/main/linux-64     Cached
   + ca-certificates    2022.3.29  h06a4308_0      pkgs/main/linux-64     Cached
   + certifi            2021.10.8  py39h06a4308_2  pkgs/main/linux-64     Cached
   + grpcio                1.42.0  py39hce63b2e_0  pkgs/main/linux-64       2 MB
                          ... etc ...

    so the gRPC version I’ll get is 1.42.0. (Most likely newer gRPC releases will also work with the 1.42.0 Python package but your chances of success will be higher if the gRPC source bundle matches the gRPC Python package.)

  3. Activate the new conda env:

     conda activate grpc_env

  4. Download the gRPC source tar file matching the version that was installed from the gRPC Github releases area.

  5. Expand the gRPC source tar file in a working directory

     cd /tmp
 tar xf ~/Downloads/grpc-1.42.0.tar.gz

  6. Enter the expanded directory and edit the CMakeLists.txt file

     cd /tmp/grpc-1.42.0/
 vi CMakeLists.txt

    Of course replace vi with your editor of choice.

  7. Insert these two lines at the top of CMakeLists.txt:

     include_directories( "$CONDA_PREFIX/include")
 link_directories(    "$CONDA_PREFIX/lib")

    replacing $CONDA_PREFIX with the actual value of this environment variable. In my setup $CONDA_PREFIX is /usr/local/anaconda3/2021.05/envs/grpc_env so my two lines are

     include_directories( "/usr/local/anaconda3/2021.05/envs/grpc_env/include")
 link_directories(    "/usr/local/anaconda3/2021.05/envs/grpc_env/lib")

  8. The CMakeLists.txt file, at least in gRPC v1.42.0, is missing Protocol Buffer library entries. Fix this is with

     perl -pi -e 's/^(.*?_gRPC_PROTOBUF_LIBRARIES\})/$1\nprotobuf\nprotoc/' CMakeLists.txt

  9. Build the makefile with cmake, then start the build

      mkdir -p cmake/build
  cd cmake/build
  cmake -DgRPC_INSTALL=ON \
        -DgRPC_BUILD_TESTS=OFF \
        -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} \
        -DgRPC_CARES_PROVIDER=kludge \
        -DgRPC_ABSL_PROVIDER=package \
        -DgRPC_PROTOBUF_PROVIDER=kludge \
        -DgRPC_RE2_PROVIDER=kludge \
        -DgRPC_ZLIB_PROVIDER=kludge \
        -DgRPC_SSL_PROVIDER=kludge \
        ../..
    
  make

  10. My build fails at 99% completion. I haven’t bothered investigating further because at this point the build will have added these seven plugins to the current directory:

      grpc_cpp_plugin
  grpc_csharp_plugin
  grpc_node_plugin
  grpc_objective_c_plugin
  grpc_php_plugin
  grpc_python_plugin
  grpc_ruby_plugin

  11. Copy the plugins to the installation directory of your choice (it does not have to be the conda environment’s bin as done here):

      cp grpc_*_plugin ${CONDA_PREFIX}/bin

    These executables contain embedded rpath directories pointing to your ${CONDA_PREFIX}/lib so don’t delete the newly-created grpc_env conda environment as long as you need the plugins. Alternatively, copy the linked shared libraries to a separate directory and add that directory to $LD_LIBRARY_PATH.

  12. Deactivate the grpc_env conda environment when you’ve finished.

      conda deactivate grpc_env

Note: if compiling with GCC, gRPC requires version 5.1 or newer. This means the process above will not work with the stock compiler on Red Hat Enterprise Linux versions below 9.