Build gRPC Plugins on Ubuntu 20.04 with Anaconda

gRPC Anaconda Python C++ C# node Objective C PHP Ruby

gRPC Logo

A frustrating aspect of working with gRPC in multiple languages is that you need each language’s gRPC plugin. One can get all supported plugins by building gRPC from source but that’s a challenging endeavour. If you can build gRPC from source, this post won’t interest you. Everyone else, read on.

There’s a shortcut to creating a gRPC build environment with the needed build dependendies: install the Anaconda Python distribution (if you don’t already have it), then create a conda environment that includes Protobuf, Abseil-Cpp, and re2. The resulting conda environment lets you build gRPC plugins for Python, C++, C#, node, Objective C, PHP, and Ruby. The process goes like this:

Install Anaconda if you don’t already have it.

Create a conda environment that includes the gRPC Python package and the dependencies needed to build from source:

 conda create --name grpc_env grpcio protobuf abseil-cpp re2

I’m explicitly excluding grpcio-tools since the Python plugin will be created by the process described here. Pay attention to the grpcio version before you hit y to begin the installation. In my case the conda create command shows

   Package                Version  Build           Channel                  Size
 ─────────────────────────────────────────────────────────────────────────────────
   Install:
 ─────────────────────────────────────────────────────────────────────────────────
   + _libgcc_mutex            0.1  main            pkgs/main/linux-64     Cached
   + _openmp_mutex            4.5  1_gnu           pkgs/main/linux-64     Cached
   + abseil-cpp        20210324.2  h2531618_0      pkgs/main/linux-64     965 KB
   + c-ares                1.18.1  h7f8727e_0      pkgs/main/linux-64     Cached
   + ca-certificates    2022.3.29  h06a4308_0      pkgs/main/linux-64     Cached
   + certifi            2021.10.8  py39h06a4308_2  pkgs/main/linux-64     Cached
   + grpcio                1.42.0  py39hce63b2e_0  pkgs/main/linux-64       2 MB
                          ... etc ...

so the gRPC version I’ll get is 1.42.0. (Most likely newer gRPC releases will also work with the 1.42.0 Python package but your chances of success will be higher if the gRPC source bundle matches the gRPC Python package.)

Activate the new conda env:
```
 conda activate grpc_env
```
Download the gRPC source tar file matching the version that was installed from the gRPC Github releases area.
Expand the gRPC source tar file in a working directory
```
 cd /tmp
 tar xf ~/Downloads/grpc-1.42.0.tar.gz
```
Enter the expanded directory and edit the CMakeLists.txt file
```
 cd /tmp/grpc-1.42.0/
 vi CMakeLists.txt
```
Of course replace vi with your editor of choice.

Insert these two lines at the top of CMakeLists.txt:

 include_directories( "$CONDA_PREFIX/include")
 link_directories(    "$CONDA_PREFIX/lib")

replacing $CONDA_PREFIX with the actual value of this environment variable. In my setup $CONDA_PREFIX is /usr/local/anaconda3/2021.05/envs/grpc_env so my two lines are

 include_directories( "/usr/local/anaconda3/2021.05/envs/grpc_env/include")
 link_directories(    "/usr/local/anaconda3/2021.05/envs/grpc_env/lib")

The CMakeLists.txt file, at least in gRPC v1.42.0, is missing Protocol Buffer library entries. Fix this is with
```
 perl -pi -e 's/^(.*?_gRPC_PROTOBUF_LIBRARIES\})/$1\nprotobuf\nprotoc/' CMakeLists.txt
```

Build the makefile with cmake, then start the build

  mkdir -p cmake/build
  cd cmake/build
  cmake -DgRPC_INSTALL=ON \
        -DgRPC_BUILD_TESTS=OFF \
        -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} \
        -DgRPC_CARES_PROVIDER=kludge \
        -DgRPC_ABSL_PROVIDER=package \
        -DgRPC_PROTOBUF_PROVIDER=kludge \
        -DgRPC_RE2_PROVIDER=kludge \
        -DgRPC_ZLIB_PROVIDER=kludge \
        -DgRPC_SSL_PROVIDER=kludge \
        ../..
    
  make

My build fails at 99% completion. I haven’t bothered investigating further because at this point the build will have added these seven plugins to the current directory:
```
  grpc_cpp_plugin
  grpc_csharp_plugin
  grpc_node_plugin
  grpc_objective_c_plugin
  grpc_php_plugin
  grpc_python_plugin
  grpc_ruby_plugin
```
Copy the plugins to the installation directory of your choice (it does not have to be the conda environment’s bin as done here):
```
  cp grpc_*_plugin ${CONDA_PREFIX}/bin
```
These executables contain embedded rpath directories pointing to your ${CONDA_PREFIX}/lib so don’t delete the newly-created grpc_env conda environment as long as you need the plugins. Alternatively, copy the linked shared libraries to a separate directory and add that directory to $LD_LIBRARY_PATH.
Deactivate the grpc_env conda environment when you’ve finished.
```
  conda deactivate grpc_env
```

Note: if compiling with GCC, gRPC requires version 5.1 or newer. This means the process above will not work with the stock compiler on Red Hat Enterprise Linux versions below 9.

Build gRPC Plugins on Ubuntu 20.04 with Anaconda

Latest Articles

Categories