Work with Redis in MATLAB Using the redis-py Python Module

Part 7 of the Python Is The Ultimate MATLAB Toolbox series.

Redis, the “remote dictionary server”, is a fast network-based in-memory database for key/value pairs. In addition to high performance and ability to store structured values, Redis implements a publish/subscribe service so that applications can be notified of changes to keys they subscribe to. The MATLAB Production Server has a Redis interface. Python does too—several, actually. In this article I’ll demonstrate a MATLAB interface to Redis via the the redis-py Python module.

Why would a MATLAB user care about Redis?

Redis may prove useful to you if you have a MATLAB application that needs to share values rapidly with other programs on your network. The programs involved use Redis as a remote memory bank for shared variables. Interaction with a Redis server happens with the Redis communication API which is available for more than 50 languages including Python—and therefore, by extension, MATLAB as well. While that sounds complicated to set up, it is often easier than writing other networking code, whether directly with TCP/IP sockets, or with a networking framework such as MPI. It is certainly cleaner and faster than communicating through the file system.

Note: Redis stores all values as character (or byte) arrays. For this reason it is poorly-suited for exchanging numeric arrays as might be needed by a computationally intensive parallel MATLAB program. Stick with MPI from the Parallel Computing Toolbox for such applications (or wait for my Nov. 12, 2022 article on using Dask with MATLAB).

If your MATLAB projects work in isolation, that is, they don’t need to get or provide information to other applications, chances are you won’t need a network-based key/value store. If that’s the case, the rest of this article may not interest you.

Install `redis-py`

You’ll need the redis-py Python module in your installation to call it from MATLAB.

NOTE: There’s a confusing overlap of names for the Python Redis package, the Redis application, and Anaconda’s Redis bundle.

Is it `redis` or `redis-py` ?

The primary Python Redis module is developed at https://github.com/redis/redis-py . Although the name on Github is redis-py, once you install this module, you import it with import redis (not import redis-py)
In Anaconda distributions, this Python module is installed with conda install redis-py (not conda install redis)
In Anaconda distributions, the full-up Redis application including the server and command line tools can be installed with conda install redis
In other Python distributions, the module can be installed with python -m pip install redis (not python -m pip install redis-py)

Check your installation

Check your Python installation to see if it has the redis-py Python module by starting the Python command line REPL and importing it—with import redis, not import redis-py. If it imports successfully you can also check the version number.

> python
>>> import redis
>>> redis.__version__
'3.5.3'

If you get a ModuleNotFoundError, you’ll need to install it using one of the methods listed above.

Start a Redis server

Skip this section if you already have a Redis server running somewhere on your network (or on your own computer).

To begin, install the Redis application by following the Redis installation instructions, using your operating system’s package manager (for example on Ubuntu you’d need apt install redis-tools redis-server) or, if you have the Anaconda Python installation, by installing Redis with conda:

> conda install redis

This provides both the server, redis-server and command line tools such as the redis-cli client program. Next, open a terminal and start the server:

> redis-server  redis.conf --port 12987

The configuration file redis.conf and port value --port 12987 are optional. If you don’t specify them the server will use a default configuration and run on port 6379. You’ll need to create a config file if you want to add password protection and persist the memory database to disk, among many other options.

If all is well you’ll see some ASCII art:

                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 5.0.3 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 12987
 |    `-._   `._    /     _.-'    |     PID: 2453308
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

2453308:M 30 Jul 2022 14:30:20.334 # Server initialized
2453308:M 30 Jul 2022 14:30:20.334 * Ready to accept connections

Connect to a Redis server

We’ll use the redis-cli command line application to test our ability to interact with the Redis server. It starts a read-evaluate-print loop (REPL) that accepts Redis commands:

(matpy) » redis-cli -h localhost -p 12987
localhost:12987>

The prompt above indicates a successful connection to the server running on the same computer. Had the connection failed, perhaps because the server isn’t running, the prompt would look like this:

(matpy) » redis-cli -h localhost -p 12987
Could not connect to Redis at localhost:12987: Connection refused
not connected>

The -h localhost switch is redundant as the default host is the current computer. It appears here explicitly to give a sense of how one would connect to a Redis server on a different computer. Also the -p 12987 can be omitted if your Redis server is using the default port (6379).

Add or update a key/value pair

Within a successfully-connected redis-cli session we can add new keys and values with the SET command:

localhost:12987> set sensor_T712_temperature 23.4
OK
localhost:12987>

Any Redis-capable application on our network can now retrieve the value for sensor_T712_temperature. To update the value, simply set it again:

localhost:12987> set sensor_T712_temperature 37.9
OK
localhost:12987>

Get a key’s value

We can retrieve a key’s value with the GET command

localhost:12987> get sensor_T712_temperature
"37.9"
localhost:12987>

Note that the return value is a character array, not a floating point number.

Batch `redis-cli` commands

Exit the redis-cli REPL session above with <ctrl>-c, <ctrl>-d, or by enterring either exit or quit. The command line client program accepts complete Redis commands without needing to entering the REPL. This makes it possible to write Redis-capable shell or Windows .bat scripts, or in fact be used by any programming language capable of making a system call. We’ll start with the ping command which merely tests the connection. A reply of PONG indicates success:

> redis-cli -h localhost -p 12987 ping
PONG

> redis-cli -h localhost -p 12987 set sensor_T712_temperature 40.2
OK

> redis-cli -h localhost -p 12987 get sensor_T712_temperature
"40.2"

Now that we’ve checked that our Redis server is up and we can interact with it, let’s do the interactions programmatically:

Python: set_get.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


#!/usr/bin/env python
# set_get.py
import redis
R = redis.Redis(host='localhost',port=12987,decode_responses=True)

try:
    R.ping()
except redis.exceptions.ConnectionError as e:
    print(f'Is Redis running?  Unable to connect {e}')
    sys.exit(1)
print(f'Connected to server.')

R.set('sensor_T712_temperature', 39.8)
retrieved_temp = R.get('sensor_T712_temperature')
print(f'sensor_T712_temperature = {retrieved_temp}')
print(f'(as a number)           = {float(retrieved_temp):.3f}')

A note about the decode_responses=True option: without it, programmatic reads of Redis keys and values receive byte arrays. In most cases you’d rather have strings, not byte arrays. You can get a string from a byte array by applying the .decode() method to the byte array, but including the decode_responses=True in the initial connection call spares you from having to add .decode() to every Redis return variable.

In any event, you still have to explicitly typecast the string temperature value if you want to use it numerically.

Running the Python program gives:

> ./set_get.py
Connected to server.
sensor_T712_temperature = 39.8
(as a number)           = 39.800

Now in MATLAB:

MATLAB: set_get.m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


% set_get.m
redis = py.importlib.import_module('redis');
R = redis.Redis(pyargs('host','localhost','port',int64(12987),...
                       'decode_responses',py.True));
try
  R.ping();
  fprintf('Connected to server.\n')
catch EO
  fprintf('Is Redis running?  Unable to connect: %s\n', EO.message)
end

R.set('sensor_T712_temperature', 39.8);
retrieved_temp = R.get('sensor_T712_temperature');
fprintf('sensor_T712_temperature = %s\n', string(retrieved_temp))
as_number = double(string(retrieved_temp));
fprintf('(as a number)           = %6.3f\n', as_number)

>> set_get
Connected to server.
sensor_T712_temperature = 39.8
(as a number)           = 39.800

Set/get structured data

The ‘value’ part of a Redis key/value pair may be a structure. Supported structures are a string, set, sorted set, list, hash, bitmap, bitfield, hyperloglog, stream, and geospatial index. The example below demonstrates setting and getting a hash since conceptually this structure resembles a MATLAB struct.

Add a hash from the command line

The Redis command HSET adds a hash. This command

> redis-cli -h localhost -p 12987 hset newmark alpha 0.25 beta 0.5 gamma 'is unset'

adds a key newmark with fields alpha, beta, and gamma corresponding to a struct like this in MATLAB:

>> newmark
  struct with fields:
    alpha: 0.2500
     beta: 0.5000
    gamma: "is unset"

Bear in mind though the numeric values in the MATLAB struct are saved as strings in the Redis hash. The Redis hash can be retrieved in its entirety with HGETALL

> redis-cli -h localhost -p 12987 hgetall newmark
1) "alpha"
2) "0.25"
3) "beta"
4) "0.5"
5) "gamma"
6) "is unset"

or selectively with HMGET (where M is for “multi-values”) by providing a list of the desired fields:

> redis-cli -h localhost -p 12987 hmget newmark alpha gamma
1) "0.25"
2) "is unset"

Programmatically, the .hgetall() function returns a Python dictionary in both Python and MATLAB. This is convenient when the values are strings but not that helpful if the values have to be typecast to numbers. In that case it’s an equal amount of work to instead calling the .hmget() method to just get back a list of values (once again, byte arrays) and convert those.

Python: get_hash.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


#!/usr/bin/env python
# get_hash.py
import redis
R = redis.Redis(host='localhost',port=12987,decode_responses=True)

try:
    R.ping()
except redis.exceptions.ConnectionError as e:
    print(f'Is Redis running?  Unable to connect {e}')
    sys.exit(1)
print(f'Connected to server.')

values = R.hmget('newmark', ['alpha', 'beta', 'gamma'])
newmark = { 'alpha' : float(values[0]),
            'beta'  : float(values[1]),
            'gamma' : values[2] }
print(newmark)

> get_hash.py
Connected to server.
{'alpha': 0.25, 'beta': 0.5, 'gamma': 'is unset'}

MATLAB: get_hash.m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


% get_hash.m
redis = py.importlib.import_module('redis');
R = redis.Redis(pyargs('host','localhost','port',int64(12987),...
                       'decode_responses',py.True));
try
  R.ping();
  fprintf('Connected to server.\n')
catch EO
  fprintf('Is Redis running?  Unable to connect: %s\n', EO.message)
end

values = R.hmget('newmark', {'alpha', 'beta', 'gamma'});
newmark = struct;
newmark.alpha = double(string(values{1}));
newmark.beta  = double(string(values{2}));
newmark.gamma = string(values{3});
disp(newmark)

>> get_hash
Connected to server.
    alpha: 0.2500
     beta: 0.5000
    gamma: "is unset"

Set/get performance

The following programs set, then get 100,000 keys with integer values from 0 to 99,999. The time to .decode() the returned byte array is also included for the get time since this operation is nearly always necessary. The MATLAB code is more than 2x slower on sets and a baffling 5x slower on gets.

Python: set_get_speed.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


#!/usr/bin/env python
# set_get_speed.py
import redis
import time
R = redis.Redis(host='localhost',port=12987)

try:
    R.ping()
except redis.exceptions.ConnectionError as e:
    print(f'Is Redis running?  Unable to connect {e}')
    sys.exit(1)
print(f'Connected to server.')

N = 100_000

T_s = time.time()
for i in range(N):
    R.set(f'K_%06{i}', i)
T_e = time.time()
print(f'{N} sets for integer values took {T_e-T_s:.3f} s')

T_s = time.time()
for i in range(N):
    v = R.get(f'K_%06{i}')
    v = v.decode()
T_e = time.time()
print(f'{N} gets for integer values took {T_e-T_s:.3f} s')

MATLAB: set_get_speed.m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


% set_get_speed.m
redis = py.importlib.import_module('redis');
R = redis.Redis(pyargs('host','localhost','port',int64(12987),...
                       'decode_responses',py.True));
try
  R.ping();
  fprintf('Connected to server.\n')
catch EO
  fprintf('Is Redis running?  Unable to connect: %s\n', EO.message)
end

N = 100000;

tic
for i = 1:N
    key = sprintf('K_%06d', i-1);
    R.set(key, i-1);
end
fprintf('%d sets for integer values took %.3f s\n', N, toc)

tic
for i = 1:N
    key = sprintf('K_%06d', i-1);
    v = R.get(key);
end
fprintf('%d gets for integer values took %.3f s\n', N, toc)

Python:

> ./set_get_speed.py
Connected to server.
100000 sets for integer values took 8.377 s
100000 gets for integer values took 7.973 s

MATLAB:

>> set_get_speed
Connected to server.
100000 sets for integer values took 21.580 s
100000 gets for integer values took 44.525 s

Despite the lower performance, a MATLAB Redis set followed by a get takes less than a millisecond.

A Redis power-feature is its support for subscriptions. When a program subscribes to a key change, Redis notifies the program each time the key’s value changes. The first step involves notifying Redis that you wish to monitor “keyspace events”, namely, changes to keys as well as key expiration and removal. The most common type of keyspace configuration option is “KEA”. It is set from the command line like so:

redis-cli -h localhost -p 12987 config set notify-keyspace-events KEA

or programmatically like this in Python:

import redis
R = redis.Redis(host='localhost',port=12987,decode_responses=True)
R.config_set('notify-keyspace-events', 'KEA')

or this MATLAB:

redis = py.importlib.import_module('redis');
R = redis.Redis(pyargs('host','localhost','port',int64(12987),...
                       'decode_responses',py.True));
R.config_set('notify-keyspace-events', 'KEA')

The next step is to define a string pattern that matches one or more keys to watch for. For this example, say our Redis instance has a collection of keys storing temperature, humidity, and light data from an array of sensors. Their names might be

sensor_T712_temperature
sensor_T714_temperature
sensor_T716_illumination
sensor_T712_humidity

To have our application to act on changes in any of the temperature keys, our subscription would look like this:

Sub = R.pubsub()
Sub.psubscribe('__keyspace@0__:sensor_*_temperature')

After setting up the subscription, the code waits for notifications. The most direct way of doing this is in a loop that sleeps (or does something else) when no notifications come in:

Python: subscribe.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


#!/usr/bin/env python
# subscribe.py
import time
import sys
import redis
R = redis.Redis(host='localhost',port=12987,decode_responses=True)

try:
    R.ping()
except redis.exceptions.ConnectionError as e:
    print(f'Is Redis running?  Unable to connect {e}')
    sys.exit(1)
print(f'Connected to server.')

# modify configuration to enable keyspace notification
R.config_set('notify-keyspace-events', 'KEA')

Sub = R.pubsub()
keys_to_match = 'sensor_*_temperature'
Sub.psubscribe('__keyspace@0__:' + keys_to_match)
while True:
    try:
        message = Sub.get_message()
    except redis.exceptions.ConnectionError:
        print('lost connection to server')
        sys.exit(1)
    if message is None:
        time.sleep(0.1)
        continue
    keyname = message['channel'].replace('__keyspace@0__:','')
    if keyname == keys_to_match:
        # initial subscription value
        continue
    # gets here if any of the matching keys was updated
    value = R.get(keyname)
    print(f'{keyname} = {value}')

MATLAB: subscribe.m

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


% subscribe.m
redis = py.importlib.import_module('redis');
R = redis.Redis(pyargs('host','localhost','port',int64(12987),...
                       'decode_responses',py.True));
try
  R.ping();
  fprintf('Connected to server.\n')
catch EO
  fprintf('Is Redis running?  Unable to connect: %s\n', EO.message)
end

% modify configuration to enable keyspace notification
R.config_set('notify-keyspace-events', 'KEA');

Sub = R.pubsub();
keys_to_match = "sensor_*_temperature";
Sub.psubscribe("__keyspace@0__:" + keys_to_match);

while 1
   try
       message = Sub.get_message();
   catch EO
       fprintf('lost connection to server: %s\n', EO.message)
       break
   end
   if message == py.None
       pause(0.1)
       continue
   end
   keyname = replace(message{'channel'}, '__keyspace@0__:','');
   if keyname == keys_to_match
       % initial subscription value
       continue
   end
   % gets here if any of the matching keys was updated
   value = string(R.get(keyname));
   fprintf('%s = %s\n', string(keyname), value)
end

When these programs run, they spend most of their time sleeping (line 27 in both programs for time.sleep(0.1) and pause(0.1)). Of course the sleeps should be replaced by calls to actual work if your application has better things to do. Then, when the clients receive a notification from Redis that any of the temperature sensors was updated (line 22 in Python and line 21 in MATLAB), the message variable is populated with the new temperature and the print statements at lines 35 (Python) and 37 (MATLAB) are executed.

The actual duration of the sleeps—or the other work done while not listening for the key changes—is not important. Keyspace change notifications are buffered and the clients will still be informed even if the clients call message = Sub.get_message() long after the keys actually changed.

We can run the Python and MATLAB programs simultaneously and watch them react to command line updates of some keys. A MATLAB session responding to command line key sets such as

> redis-cli -p 12987
127.0.0.1:12987> set sensor_A_temperature 101.5
OK
127.0.0.1:12987> set sensor_B_temperature 101.6
OK
127.0.0.1:12987> set sensor_B_humidity 44.3
OK
127.0.0.1:12987> set sensor_C_temperature 101.7
OK

looks like this

>> subscribe
Connected to server.
sensor_A_temperature = 101.5
sensor_B_temperature = 101.6
sensor_C_temperature = 101.7

Note the program did not respond to a change to sensor_B_humidity since this key name doesn’t match our pattern of sensor_*_temperature.

Table of Contents | Previous: Interact with MongoDB Next : Solve global optimization problems

Work with Redis in MATLAB Using the redis-py Python Module

Why would a MATLAB user care about Redis?

Install `redis-py`

Is it `redis` or `redis-py` ?

Check your installation

Start a Redis server

Connect to a Redis server

Add or update a key/value pair

Get a key’s value

Batch `redis-cli` commands

Set/get structured data

Add a hash from the command line

Set/get performance

Latest Articles

Categories

Work with Redis in MATLAB Using the redis-py Python Module

Why would a MATLAB user care about Redis?

Install redis-py

Is it redis or redis-py ?

Check your installation

Start a Redis server

Connect to a Redis server

Add or update a key/value pair

Get a key’s value

Batch redis-cli commands

Set/get structured data

Add a hash from the command line

Set/get performance

Subscribe to key changes

Latest Articles

Categories

Install `redis-py`

Is it `redis` or `redis-py` ?

Batch `redis-cli` commands