MOVED error with Redis cluster

Introduction

In this article I will try to explain a weird error that I had to deal with, working with Redis. The common message of the error is “(error) MOVED 12933 127.0.0.1:30003”. Now let’s see when this could happened.

Requirements

In order to follow me you maybe will need to have Redis, I will also assume that you are using some UNIX like distro, but you could find out the equivalents command on Windows by yourself. You are a big boy. In total we only need:

  • Redis
  • OS: Ubuntu, Debian, Centos, etc… you got the idea.

Basic setup

Downloading and compiling Redis

Downloading

The current stable version of Redis is 6.2.6, you could download it through this link or making use of curl, like this:

curl -sO https://download.redis.io/releases/redis-6.2.6.tar.gz

Extracting folder

tar -xf redis-6.2.6.tar.gz

Now you should have a folder called redis-6.2.6, let’s position inside of it with cd command:

cd redis-6.2.6

Compiling

Now we are good to go.

make

In case you want to run the tests inside the redis code, you could run make test. There are other parameters that you can pass in the build, that could be interesting to trying out, but not now.

Description

In case you are in hurry, let me just summarize it to you. You will find this error when you try to use a redis client on a redis cluster, without specifying on this redis client that you are actually connecting to a redis cluster and not a normal redis node. Of course, if you just read this sentence could be the case that it makes no sense to you, right? Let’s go deeper then.

Reproducing the error

Let’s try to reproduce this error without any particular library in any language first, just making use of redis-cli.

Setup

Let’s run a Redis cluster locally, it will be easy. If you already compiled the source code of redis in the previous setup, you will notice that there’s a folder called utils. Inside this folder there’s another one called create-cluster, so I’m going to make use of the script inside this folder to create a cluster, is just the easiest way.

In order to run this scrip you need to be position in the same folder as the script, so let’s go into there:

cd utils/create-cluster/

Run the script:

./create-cluster start
./create-cluster create

Yes, like that. Wait what? Is already created? Yep, is code written in C so… A little confusing what they mean by start and create, usually the order is first create and then start, but naming is hard.

In the create command it will ask you Can I set the above configuration? (type ‘yes’ to accept), so type yes, you are following a tutorial that’s the point.

Now we have our cluster locally running.

How to stop and clean this?

./create-cluster stop
./create-cluster clean

Connecting

Let’s connect to the cluster with redis-cli. In the previous step you must notice a line like this one in the logs printed on the ./create-cluster create command:

>>> Performing Cluster Check (using node 127.0.0.1:30001)

In my case the port that I will be trying to connect is 30001, but in your case could be any other.

./src/redis-cli -c -p  30001

VERY IMPORTANT HERE

The -c option is for

-c Enable cluster mode (follow -ASK and -MOVED re-direction).

So right now we are connecting using the cluster mode, so we won’t have any problem, the problem will come later if we try to make operation with a connection that doesn’t have this Cluster mode enabled. I’ll comment more on that later.

Make a PING command just to check if everything looks fine,

127.0.0.1:30001> PING

You should get PONG as an answer.

Reproducing error

Now let’s back to business ), I will set several pairs and later I will try to fetch them

127.0.0.1:30001> SET name Lola
-> Redirected to slot [5798] located at 127.0.0.1:30002
OK
127.0.0.1:30002> SET name1 Lola
-> Redirected to slot [12933] located at 127.0.0.1:30003
OK
127.0.0.1:30003> SET name2 Manolo
-> Redirected to slot [742] located at 127.0.0.1:30001
OK
127.0.0.1:30001> SET name3 Juan
OK
127.0.0.1:30001> SET name4 Juan
-> Redirected to slot [8736] located at 127.0.0.1:30002
OK
127.0.0.1:30002

Take a look at these logs, yes I’m connected to port 30001 but my pairs, not always were set at that node. In my case the value for key name1, on command SET name1 Lola was located in a slot in node 30003. I will explain more in advance, for now let’s just reproduce the error.

Connecting without cluster mode enabled

Let’s try to connect to one of this node without enabling cluster mode, so open another terminal and run a this command:

./src/redis-cli -p 30001

IMPORTANT AGAIN

Here the difference is that the option -c was not supplied.

Let’s make PING, can we? Yes, we can!

127.0.0.1:30001> PING
PONG

THE ERROR

Now comes the cool surprise, at the moment you must be thinking like, “Well I just made PING without problem, so I’m sure that I could retrieve those that I already pushed, right?”

Let’s try to retrieve them with GET command,

127.0.0.1:30001> GET name1

ERROR

127.0.0.1:30001> GET name1
(error) MOVED 12933 127.0.0.1:30003
127.0.0.1:30001>

Enjoy debugging this on your application, is not fun at all really. You start thinking, like, why? I ran the magical command called PING and everything is fine.

Just to clarify, this is NOT an error of the server, this is YOUR FAULT, for setting up bad your redis client.

Rule of thumb

In general if you are using a redis cluster in your server, you should enabled the cluster mode on your client. There’s a lot of alternatives for this, depending on the Programming Language that you are using. Here is a list from the redis website:

  • redis-rb-cluster is a Ruby implementation written by me (@antirez) as a reference for other languages. It is a simple wrapper around the original redis-rb, implementing the minimal semantics to talk with the cluster efficiently.
  • redis-py-cluster A port of redis-rb-cluster to Pyton. Supports majority of redis-py functionality. Is in active development.
  • The popular Predis has support for Redis Cluster, the support was recently updated and is in active development.
  • The most used Java client, Jedis recently added support for Redis Cluster, see the Jedis Cluster section in the project README.
  • StackExchange.Redis offers support for C# (and should work fine with most .NET languages; VB, F#, etc)
  • thunk-redis offers support for Node.js and io.js, it is a thunk/promise-based redis client with pipelining and cluster.
  • redis-go-cluster is an implementation of Redis Cluster for the Go language using the Redigo library client as the base client. Implements MGET/MSET via result aggregation.
  • ioredis is a popular Node.js client, providing a robust support for Redis Cluster.
  • The redis-cli utility implements basic cluster support when started with the -c switch.

Related to Golang, the library go-redis, have also support for this.

Bibligraphy

  1. Redis cluster tutorial
  2. Redis cluster specification
  3. Cluster Nodes

Until next error guys ).