Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -583,7 +583,7 @@ private static String htmlPage(long visibilityMs) {
+ " <h1>Redis Job Queue Demo</h1>\n"
+ " <p class=\"lede\">\n"
+ " Enqueue background jobs and watch a pool of workers drain them through Redis.\n"
+ " Pending jobs sit in a list; each worker uses <code>BRPOPLPUSH</code> to atomically\n"
+ " Pending jobs sit in a list; each worker uses <code>BLMOVE</code> to atomically\n"
+ " claim a job and move it to a processing list. Completed jobs move to a short\n"
+ " history. If a worker hangs past the " + visibilityMs + " ms visibility timeout,\n"
+ " the reclaimer moves its job back to pending so no work is lost.\n"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.Pipeline;
import redis.clients.jedis.args.ListDirection;
import redis.clients.jedis.params.ScanParams;
import redis.clients.jedis.resps.ScanResult;

Expand Down Expand Up @@ -219,7 +220,7 @@ public ClaimedJob claim(long timeoutMs) {
double timeoutSec = Math.max(timeoutMs / 1000.0, 0.1);
String jobId;
try (Jedis jedis = pool.getResource()) {
jobId = jedis.brpoplpush(pendingKey, processingKey, (int) Math.ceil(timeoutSec));
jobId = jedis.blmove(pendingKey, processingKey, ListDirection.RIGHT, ListDirection.LEFT, timeoutSec);
}
if (jobId == null) {
return null;
Expand Down
20 changes: 10 additions & 10 deletions content/develop/use-cases/job-queue/java-jedis/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The flow looks like this:
1. The application calls `queue.enqueue(payload)`
2. The helper writes the job metadata hash and `LPUSH`es the job ID onto the pending list
3. A worker calls `queue.claim(timeoutMs)`
4. The helper runs `BRPOPLPUSH` to atomically move the next pending ID into the processing list and writes a per-claim `claim_token` plus `claimed_at_ms` on the hash
4. The helper runs `BLMOVE` to atomically move the next pending ID into the processing list and writes a per-claim `claim_token` plus `claimed_at_ms` on the hash
5. The worker runs the job and calls `queue.complete(job, result)` or `queue.fail(job, error)`
6. `complete` removes the job from the processing list, writes the result, and `LPUSH`es the ID onto the completed history (with `LTRIM` and an `EXPIRE` on the hash for cleanup)
7. `fail` either retries the job (back to pending) or moves it to the failed list once retries are exhausted
Expand Down Expand Up @@ -82,7 +82,7 @@ public class Main {
}
```

Jedis operations are synchronous. The helper acquires a `Jedis` connection per call using `pool.getResource()` inside a try-with-resources block, so connections are returned to the pool even on errors. The blocking `claim()` method holds its own connection for the duration of the `BRPOPLPUSH` wait, which is fine because every other call uses a different connection.
Jedis operations are synchronous. The helper acquires a `Jedis` connection per call using `pool.getResource()` inside a try-with-resources block, so connections are returned to the pool even on errors. The blocking `claim()` method holds its own connection for the duration of the `BLMOVE` wait, which is fine because every other call uses a different connection.

### Data model

Expand Down Expand Up @@ -116,7 +116,7 @@ queue:jobs-jedis:job:9a4f...
The implementation uses:

* [`LPUSH`](/content/commands/lpush.md) to add new job IDs to the pending list
* [`BRPOPLPUSH`](/content/commands/brpoplpush.md) to atomically claim a job into the processing list
* [`BLMOVE`](/content/commands/blmove.md) to atomically claim a job into the processing list (the modern replacement for the deprecated [`BRPOPLPUSH`](/content/commands/brpoplpush.md))
* [`LREM`](/content/commands/lrem.md) to remove a claimed job from the processing list on complete or fail
* [`LTRIM`](/content/commands/ltrim.md) to cap the completed and failed history lists
* [`HSET`](/content/commands/hset.md) / [`HGETALL`](/content/commands/hgetall.md) for job metadata
Expand Down Expand Up @@ -152,16 +152,16 @@ public String enqueue(Map<String, Object> payload) {

The payload is stored as a JSON string so the queue can carry arbitrary nested structures without forcing every field into a hash.

## Claiming jobs with BRPOPLPUSH
## Claiming jobs with BLMOVE

A worker blocks until a job is available, then atomically pops it from the pending list and pushes it onto the processing list. `BRPOPLPUSH` does both in a single Redis call:
A worker blocks until a job is available, then atomically pops it from the pending list and pushes it onto the processing list. `BLMOVE` does both in a single Redis call (it's the modern replacement for `BRPOPLPUSH`, which is deprecated in Redis 6.2+):

```java
public ClaimedJob claim(long timeoutMs) {
double timeoutSec = Math.max(timeoutMs / 1000.0, 0.1);
String jobId;
try (Jedis jedis = pool.getResource()) {
jobId = jedis.brpoplpush(pendingKey, processingKey, (int) Math.ceil(timeoutSec));
jobId = jedis.blmove(pendingKey, processingKey, ListDirection.RIGHT, ListDirection.LEFT, timeoutSec);
}
if (jobId == null) {
return null;
Expand Down Expand Up @@ -265,7 +265,7 @@ public List<String> reclaimStuck() {
}
```

The Lua script also handles a narrower race: a worker that crashed between `BRPOPLPUSH` and writing `claimed_at_ms`. Those jobs are reclaimed after `2 × visibility_ms` using `enqueued_at_ms` as a fallback timer, so they aren't stranded forever.
The Lua script also handles a narrower race: a worker that crashed between `BLMOVE` and writing `claimed_at_ms`. Those jobs are reclaimed after `2 × visibility_ms` using `enqueued_at_ms` as a fallback timer, so they aren't stranded forever.

## Stats and history

Expand Down Expand Up @@ -306,7 +306,7 @@ The completed and failed lists are capped via `LTRIM` so they never grow unbound

Before running the demo, make sure that:

* Redis 6.2 or later is running locally on the default port (6379). Earlier versions still work, since the helper uses commands that have existed since Redis 2.6.
* Redis 6.2 or later running locally on the default port (6379). `BLMOVE` requires Redis 6.2+; on older servers, replace the call with `BRPOPLPUSH`.
* Java 17 or later (the demo uses text-block-free string concatenation but still relies on a modern JDK).
* Jedis 5.x is on the classpath. The smallest workable classpath is the Jedis jar plus its two transitive dependencies, `commons-pool2` and `slf4j-api`.

Expand Down Expand Up @@ -393,7 +393,7 @@ The demo only reclaims when you click the button. In production, run `reclaimStu

### Size `JedisPool` for your worker count

`JedisPool` is thread-safe and connections are released back to the pool when the try-with-resources block exits. The demo bumps `maxTotal` to 32 to support the blocking `BRPOPLPUSH` call held by each worker plus the per-request connections used by the HTTP handlers. As a rule of thumb, `maxTotal` should be at least *workers + concurrent HTTP request threads + reclaimer threads + headroom*.
`JedisPool` is thread-safe and connections are released back to the pool when the try-with-resources block exits. The demo bumps `maxTotal` to 32 to support the blocking `BLMOVE` call held by each worker plus the per-request connections used by the HTTP handlers. As a rule of thumb, `maxTotal` should be at least *workers + concurrent HTTP request threads + reclaimer threads + headroom*.

### Use a separate Redis database or key prefix per queue

Expand Down Expand Up @@ -433,7 +433,7 @@ redis-cli --scan --pattern 'queue:jobs-jedis:*' | xargs redis-cli DEL
This example uses the following Redis commands:

* [`LPUSH`](/content/commands/lpush.md) to enqueue a job ID.
* [`BRPOPLPUSH`](/content/commands/brpoplpush.md) to atomically claim a job into the processing list.
* [`BLMOVE`](/content/commands/blmove.md) to atomically claim a job into the processing list.
* [`LREM`](/content/commands/lrem.md) to remove a job from the processing list on complete or fail.
* [`LRANGE`](/content/commands/lrange.md) and [`LLEN`](/content/commands/llen.md) to read queue depth and list contents.
* [`LTRIM`](/content/commands/ltrim.md) to cap the completed and failed history.
Expand Down
13 changes: 6 additions & 7 deletions content/develop/use-cases/job-queue/php/JobQueue.php
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ class JobQueue
// and moves any job past the visibility timeout back to the pending
// list. A job is past the timeout if either:
// - claimed_at_ms is set and (now - claimed_at_ms) > visibility_ms, OR
// - claimed_at_ms is missing (worker crashed between BRPOPLPUSH and
// - claimed_at_ms is missing (worker crashed between BLMOVE and
// the metadata write) and (now - enqueued_at_ms) > 2 * visibility_ms.
// Runs in one round trip so a concurrent worker can't claim a
// half-reclaimed job.
Expand Down Expand Up @@ -266,17 +266,16 @@ public function enqueue(array $payload): string
/**
* Block until a job is available, then atomically claim it.
*
* Uses BRPOPLPUSH to wait for a pending job and move it to the
* Uses BLMOVE to wait for a pending job and move it to the
* processing list in a single Redis call. Returns null if nothing
* arrives before $timeoutMs.
*/
public function claim(int $timeoutMs = 1000): ?ClaimedJob
{
// Predis brpoplpush takes whole seconds (server-side blocking
// semantics). Round up so a sub-second timeout still blocks at
// least once.
$timeoutSec = max(1, (int) ceil($timeoutMs / 1000));
$jobId = $this->redis->brpoplpush($this->pendingKey, $this->processingKey, $timeoutSec);
// BLMOVE takes a fractional timeout in seconds. Keep a 0.1s minimum so
// a tiny timeoutMs does not turn into 0, which would block forever.
$timeoutSec = max($timeoutMs / 1000.0, 0.1);
$jobId = $this->redis->blmove($this->pendingKey, $this->processingKey, 'RIGHT', 'LEFT', $timeoutSec);
if ($jobId === null || $jobId === false || $jobId === '') {
return null;
}
Expand Down
20 changes: 10 additions & 10 deletions content/develop/use-cases/job-queue/php/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ The flow looks like this:
1. The application calls `$queue->enqueue($payload)`
2. The helper writes the job metadata hash and `LPUSH`es the job ID onto the pending list
3. A worker calls `$queue->claim($timeoutMs)`
4. The helper runs `BRPOPLPUSH` to atomically move the next pending ID into the processing list and writes a per-claim `claim_token` plus `claimed_at_ms` on the hash
4. The helper runs `BLMOVE` to atomically move the next pending ID into the processing list and writes a per-claim `claim_token` plus `claimed_at_ms` on the hash
5. The worker runs the job and calls `$queue->complete($job, $result)` or `$queue->fail($job, $error)`
6. `complete` removes the job from the processing list, writes the result, and `LPUSH`es the ID onto the completed history (with `LTRIM` and an `EXPIRE` on the hash for cleanup)
7. `fail` either retries the job (back to pending) or moves it to the failed list once retries are exhausted
Expand Down Expand Up @@ -110,7 +110,7 @@ Because PHP's built-in dev server runs each HTTP request in a fresh process, per
The implementation uses:

* [`LPUSH`](/content/commands/lpush.md) to add new job IDs to the pending list
* [`BRPOPLPUSH`](/content/commands/brpoplpush.md) to atomically claim a job into the processing list
* [`BLMOVE`](/content/commands/blmove.md) to atomically claim a job into the processing list (the modern replacement for the deprecated [`BRPOPLPUSH`](/content/commands/brpoplpush.md))
* [`LREM`](/content/commands/lrem.md) to remove a claimed job from the processing list on complete or fail
* [`LTRIM`](/content/commands/ltrim.md) to cap the completed and failed history lists
* [`HSET`](/content/commands/hset.md) / [`HGETALL`](/content/commands/hgetall.md) for job metadata
Expand Down Expand Up @@ -148,15 +148,15 @@ public function enqueue(array $payload): string

The payload is stored as JSON so the queue can carry arbitrary nested structures without forcing every field into a hash. The `flattenFields()` helper turns the associative `$meta` array into the variadic `field, value, field, value` argument list that Predis's `hset()` expects in Predis 3.x.

## Claiming jobs with BRPOPLPUSH
## Claiming jobs with BLMOVE

A worker blocks until a job is available, then atomically pops it from the pending list and pushes it onto the processing list. `BRPOPLPUSH` does both in a single Redis call:
A worker blocks until a job is available, then atomically pops it from the pending list and pushes it onto the processing list. `BLMOVE` does both in a single Redis call (it's the modern replacement for `BRPOPLPUSH`, which is deprecated in Redis 6.2+):

```php
public function claim(int $timeoutMs = 1000): ?ClaimedJob
{
$timeoutSec = max(1, (int) ceil($timeoutMs / 1000));
$jobId = $this->redis->brpoplpush($this->pendingKey, $this->processingKey, $timeoutSec);
$timeoutSec = max($timeoutMs / 1000.0, 0.1);
$jobId = $this->redis->blmove($this->pendingKey, $this->processingKey, 'RIGHT', 'LEFT', $timeoutSec);
if ($jobId === null || $jobId === false || $jobId === '') {
return null;
}
Expand Down Expand Up @@ -186,7 +186,7 @@ public function claim(int $timeoutMs = 1000): ?ClaimedJob

The `claim_token` is the worker's proof of ownership for this attempt. Every subsequent state change (complete, fail) checks it before touching the processing list, so a worker that hung past the visibility timeout cannot interfere with the new claimant.

Predis exposes `BRPOPLPUSH` directly and accepts a whole-second timeout; sub-second blocking would need either a custom command or a non-blocking poll loop.
Predis exposes `BLMOVE` directly and accepts a fractional timeout in seconds, so the demo keeps a 0.1 second minimum instead of rounding up to a whole second.

## Completing jobs

Expand Down Expand Up @@ -270,7 +270,7 @@ public function reclaimStuck(): array
}
```

The Lua script also handles a narrower race: a worker that crashed between `BRPOPLPUSH` and writing `claimed_at_ms`. Those jobs are reclaimed after `2 × visibility_ms` using `enqueued_at_ms` as a fallback timer, so they aren't stranded forever.
The Lua script also handles a narrower race: a worker that crashed between `BLMOVE` and writing `claimed_at_ms`. Those jobs are reclaimed after `2 × visibility_ms` using `enqueued_at_ms` as a fallback timer, so they aren't stranded forever.

## Stats and history

Expand Down Expand Up @@ -305,7 +305,7 @@ The completed and failed lists are capped via `LTRIM` so they never grow unbound

## Prerequisites

* Redis 6.2 or later running locally on the default port (6379). Earlier versions still work, since the helper uses commands that have existed since Redis 2.6.
* Redis 6.2 or later running locally on the default port (6379). `BLMOVE` requires Redis 6.2+; on older servers, replace the call with `BRPOPLPUSH`.
* PHP 8.1 or later, with the `posix` and `pcntl` extensions enabled (both ship with the official PHP binary on macOS and most Linux distros).
* The Predis client (3.x). Install it with [Composer](https://getcomposer.org/):

Expand Down Expand Up @@ -450,7 +450,7 @@ redis-cli --scan --pattern 'queue:jobs:*' | xargs redis-cli DEL
This example uses the following Redis commands:

* [`LPUSH`](/content/commands/lpush.md) to enqueue a job ID.
* [`BRPOPLPUSH`](/content/commands/brpoplpush.md) to atomically claim a job into the processing list.
* [`BLMOVE`](/content/commands/blmove.md) to atomically claim a job into the processing list.
* [`LREM`](/content/commands/lrem.md) to remove a job from the processing list on complete or fail.
* [`LRANGE`](/content/commands/lrange.md) and [`LLEN`](/content/commands/llen.md) to read queue depth and list contents.
* [`LTRIM`](/content/commands/ltrim.md) to cap the completed and failed history.
Expand Down
2 changes: 1 addition & 1 deletion content/develop/use-cases/job-queue/php/demo_server.php
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ function render_page(int $visibilityMs): string
<h1>Redis Job Queue Demo</h1>
<p class="lede">
Enqueue background jobs and watch a pool of workers drain them through Redis.
Pending jobs sit in a list; each worker uses <code>BRPOPLPUSH</code> to atomically
Pending jobs sit in a list; each worker uses <code>BLMOVE</code> to atomically
claim a job and move it to a processing list. Completed jobs move to a short
history. If a worker hangs past the {$visibilityMs} ms visibility timeout,
the reclaimer moves its job back to pending so no work is lost.
Expand Down
Loading
Loading