Cloudflare Queues

# Cloudflare Queues

**Status**: Production Ready ✅ | **Last Verified**: 2025-12-27

**Dependencies**: cloudflare-worker-base (for Worker setup)

**Contents**: [Quick Start](#quick-start-10-minutes) • [Critical Rules](#critical-rules) • [Top Errors](#top-3-critical-errors) • [Use Cases](#common-use-cases) • [When to Load References](#when-to-load-references) • [Limits](#limits--quotas)

---

## Quick Start (10 Minutes)

### 1. Create Queue

```bash
bunx wrangler queues create my-queue
bunx wrangler queues list
```

### 2. Producer (Send Messages)

**wrangler.jsonc:**

```jsonc
{
  "name": "my-producer",
  "main": "src/index.ts",
  "queues": {
    "producers": [
      {
        "binding": "MY_QUEUE",
        "queue": "my-queue"
      }
    ]
  }
}
```

**src/index.ts:**

```typescript
import { Hono } from 'hono';

type Bindings = {
  MY_QUEUE: Queue;
};

const app = new Hono<{ Bindings: Bindings }>();

app.post('/send', async (c) => {
  await c.env.MY_QUEUE.send({
    userId: '123',
    action: 'process-order',
    timestamp: Date.now(),
  });

  return c.json({ status: 'queued' });
});

export default app;
```

### 3. Consumer (Process Messages)

**wrangler.jsonc:**

```jsonc
{
  "name": "my-consumer",
  "main": "src/consumer.ts",
  "queues": {
    "consumers": [
      {
        "queue": "my-queue",
        "max_batch_size": 10,
        "max_retries": 3,
        "dead_letter_queue": "my-dlq"
      }
    ]
  }
}
```

**src/consumer.ts:**

```typescript
import type { MessageBatch } from '@cloudflare/workers-types';

export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      console.log('Processing:', message.body);
      // Your logic here
    }
    // Implicit ack: returning successfully acknowledges all messages
  },
};
```

**Deploy:**

```bash
bunx wrangler deploy
```

**Load**: `references/setup-guide.md` for complete 6-step setup with DLQ configuration

---

## Critical Rules

### Always Do ✅

1. **Configure Dead Letter Queue** for production queues
2. **Use explicit ack()** for non-idempotent operations (DB writes, API calls)
3. **Validate message size** before sending (<128 KB)
4. **Use sendBatch()** for multiple messages (more efficient)
5. **Implement exponential backoff** for retries
6. **Set appropriate batch settings** based on workload
7. **Monitor queue backlog** and consumer errors
8. **Use ctx.waitUntil()** for async cleanup in consumers
9. **Handle errors gracefully** - log, alert, retry
10. **Let concurrency auto-scale** (don't set max_concurrency unless needed)

### Never Do ❌

1. **Never assume message ordering** - not guaranteed FIFO
2. **Never rely on implicit ack for non-idempotent ops** - use explicit ack()
3. **Never send messages >128 KB** - will fail
4. **Never delete queues with active messages** - data loss
5. **Never skip DLQ configuration** in production
6. **Never exceed 5000 msg/s per queue** - rate limit error
7. **Never process messages synchronously in loop** - use Promise.all()
8. **Never ignore message.attempts** - use for backoff logic
9. **Never set max_concurrency=1** unless you have a very specific reason
10. **Never forget to ack()** in explicit acknowledgement patterns

---

## Top 3 Critical Errors

### Error #1: Message Too Large

**Problem**: Message exceeds 128 KB limit

**Solution**: Store large data in R2, send reference

```typescript
// ❌ Wrong
await env.MY_QUEUE.send({ data: largeArray }); // >128 KB fails

// ✅ Correct
const message = { data: largeArray };
const size = new TextEncoder().encode(JSON.stringify(message)).length;

if (size > 128000) {
  const key = `messages/${crypto.randomUUID()}.json`;
  await env.MY_BUCKET.put(key, JSON.stringify(message));
  await env.MY_QUEUE.send({ type: 'large-message', r2Key: key });
} else {
  await env.MY_QUEUE.send(message);
}
```

### Error #2: Throughput Exceeded

**Problem**: Exceeding 5000 messages/second per queue

**Solution**: Use sendBatch() and rate limiting

```typescript
// ❌ Wrong
for (let i = 0; i < 10000; i++) {
  await env.MY_QUEUE.send({ id: i }); // Too fast!
}

// ✅ Correct
const messages = Array.from({ length: 10000 }, (_, i) => ({
  body: { id: i },
}));

// Send in batches of 100
for (let i = 0; i < messages.length; i += 100) {
  await env.MY_QUEUE.sendBatch(messages.slice(i, i + 100));
}
```

### Error #3: Entire Batch Retried When One Message Fails

**Problem**: Single message failure causes all messages to retry

**Solution**: Use explicit acknowledgement

```typescript
// ❌ Wrong - implicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
        message.body.id,
        message.body.amount
      ).run();
    }
    // If any fails, ALL retry!
  },
};

// ✅ Correct - explicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      try {
        await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
          message.body.id,
          message.body.amount
        ).run();

        message.ack(); // Only ack on success
      } catch (error) {
        console.error(`Failed: ${message.id}`, error);
        // Don't ack - will retry independently
      }
    }
  },
};
```

**Load `references/error-catalog.md` for all 10 errors including DLQ configuration, auto-scaling issues, message deletion prevention, and detailed solutions.**

---

## Common Use Cases

### Use Case 1: Basic Message Queue

**When**: Simple async job processing (emails, notifications)

**Quick Pattern**:

```typescript
// Producer
await env.MY_QUEUE.send({ type: 'email', to: 'user@example.com' });

// Consumer (implicit ack - for idempotent operations)
export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      await sendEmail(message.body.to, message.body.content);
    }
  },
};
```

**Load**: `templates/queues-producer.ts` + `templates/queues-consumer-basic.ts`

### Use Case 2: Database Writes (Non-Idempotent)

**When**: Writing to database, must avoid duplicates

**Load**: `templates/queues-consumer-explicit-ack.ts` + `references/consumer-api.md`

### Use Case 3: Retry with Exponential Backoff

**When**: Calling rate-limited APIs, temporary failures

**Load**: `templates/queues-retry-with-delay.ts` + `references/error-catalog.md` (Error #2, #3)

### Use Case 4: Dead Letter Queue Pattern

**When**: Production systems, need to capture permanently failed messages

**Load**: `templates/queues-dlq-pattern.ts` + `references/setup-guide.md` (Step 4)

### Use Case 5: High Throughput Processing

**When**: Processing thousands of messages per second

**Quick Pattern**:

```jsonc
{
  "queues": {
    "consumers": [{
      "queue": "my-queue",
      "max_batch_size": 100,        // Large batches
      "max_batch_timeout": 5,       // Fast processing
      "max_concurrency": null       // Auto-scale
    }]
  }
}
```

**Load**: `references/best-practices.md` → Optimizing Throughput

---

## When to Load References

**Load `references/setup-guide.md` when**:
- User needs complete setup walkthrough (queue → producer → consumer → DLQ)
- First time setting up Cloudflare Queues
- Need production configuration examples
- Want complete end-to-end example

**Load `references/error-catalog.md` when**:
- Encountering any of the 10 documented errors
- Troubleshooting queue issues
- Messages not being delivered/processed
- Need prevention checklist

**Load `references/producer-api.md` when**:
- Need complete producer API reference
- Using send() or sendBatch() methods
- Need message format specifications
- Handling large messages or batches

**Load `references/consumer-api.md` when**:
- Need complete consumer API reference
- Using explicit ack(), retry(), or retryAll()
- Need batch processing patterns
- Implementing complex consumer logic

**Load `references/best-practices.md` when**:
- Optimizing queue performance
- Production deployment guidance
- Monitoring and observability setup
- Security and reliability patterns

**Load `references/wrangler-commands.md` when**:
- Need CLI commands reference
- Managing queues (create, delete, list)
- Controlling delivery (pause, resume)
- Debugging queue issues
- Real-time monitoring and performance analysis

**Load `references/typescript-types.md` when**:
- Need complete TypeScript type definitions
- Working with Queue, MessageBatch, or Message interfaces
- Implementing type-safe producers or consumers
- Using generic types for message bodies
- Need type guards for message validation

**Load `references/production-checklist.md` when**:
- Preparing for production deployment
- Need pre-deployment verification checklist
- Want detailed explanations of production best practices
- Setting up monitoring, DLQ, or error handling
- Planning load testing or security review

**Load `references/pull-consumers.md` when**:
- Need to consume messages from non-Workers environments
- Integrating with existing backend services (Node.js, Python, Go)
- Implementing pull-based polling instead of push-based consumption
- Working with containerized or legacy applications

**Load `references/http-publishing.md` when**:
- Publishing messages from external systems via HTTP
- Integrating webhooks from third-party services
- Need to send messages without deploying Workers
- Implementing CI/CD pipeline notifications

**Load `references/r2-event-integration.md` when**:
- Triggering queue messages on R2 bucket events
- Implementing automated image/document processing
- Setting up event-driven data pipelines
- Need R2 object upload/delete notifications

---

## Agents & Commands

**Available Agents**:
- **queue-debugger** - 9-phase diagnostic analysis for queue issues (systematic troubleshooting)
- **queue-optimizer** - Performance tuning and cost optimization (batch size, concurrency, retries)

**Available Commands**:
- **/queue-setup** - Interactive wizard for complete queue setup
- **/queue-troubleshoot** - Quick diagnostic for common issues
- **/queue-monitor** - Real-time metrics and status display

---

## Limits & Quotas

**Critical limits:**

- **Message size**: 128 KB max per message
- **Throughput**: 5,000 messages/second per queue
- **Batch size**: 100 messages max per sendBatch()
- **Consumer CPU time**: 30 seconds (default), 300 seconds (max with config)
- **Max retries**: Configurable (default 3)
- **Queue name**: Max 63 characters, lowercase/numbers/hyphens

**Load `references/best-practices.md` for handling limits and optimization strategies.**

---

## Configuration Reference

**Producer**: Add queue binding to `wrangler.jsonc` queues.producers array with `binding` and `queue` fields.

**Consumer**: Configure in `wrangler.jsonc` queues.consumers array with `queue`, `max_batch_size` (1-100), `max_batch_timeout` (0-60s), `max_retries`, `dead_letter_queue`, and optionally `max_concurrency` (default: auto-scale).

**CPU Limits**: Increase `limits.cpu_ms` from default 30,000ms if processing takes longer.

**Load `references/setup-guide.md` for complete configuration examples and `templates/wrangler-queues-config.jsonc` for production-ready config.**

---

## Using Bundled Resources

### References (references/)

- **setup-guide.md** - Complete 6-step setup (queue → producer → consumer → DLQ → deploy → production config)
- **error-catalog.md** - All 10 errors with solutions + prevention checklist
- **producer-api.md** - Complete producer API (send, sendBatch, message format)
- **consumer-api.md** - Complete consumer API (ack, retry, batch processing)
- **best-practices.md** - Performance, monitoring, security, reliability patterns
- **wrangler-commands.md** - CLI reference (create, delete, list, pause, resume)
- **typescript-types.md** - Complete TypeScript type definitions for Queue, MessageBatch, Message
- **production-checklist.md** - Pre-deployment verification and best practices
- **pull-consumers.md** - Pull-based consumers for non-Workers environments (HTTP polling)
- **http-publishing.md** - Publishing messages via HTTP API from external systems
- **r2-event-integration.md** - R2 event notifications triggering queue messages

### Templates (templates/)

- **queues-producer.ts** - Basic producer with single and batch sending
- **queues-consumer-basic.ts** - Implicit ack consumer (idempotent operations)
- **queues-consumer-explicit-ack.ts** - Explicit ack consumer (non-idempotent)
- **queues-retry-with-delay.ts** - Exponential backoff retry pattern
- **queues-dlq-pattern.ts** - Dead letter queue setup and consumer
- **wrangler-queues-config.jsonc** - Complete production configuration

---

## TypeScript Types

Use `@cloudflare/workers-types` package for complete type definitions: `Queue`, `MessageBatch<Body>`, `Message<Body>`, `QueueSendOptions`.

**Load `references/typescript-types.md` for complete type reference with interfaces, generics, type guards, and usage examples.**

---

## Monitoring & Debugging

**Key Commands**: `wrangler queues info` (status), `wrangler tail` (logs), `wrangler queues pause-delivery/resume-delivery` (control).

**Load `references/wrangler-commands.md` for complete CLI reference with real-time monitoring, debugging workflows, and performance analysis commands.**

---

## Production Checklist

**12-Point Pre-Deployment Checklist**: DLQ configuration, message acknowledgment strategy, size validation, batch optimization, concurrency settings, CPU limits, error handling, monitoring, rate limiting, idempotency, load testing, and security review.

**Load `references/production-checklist.md` for complete checklist with detailed explanations, code examples, and deployment workflow.**

---

## Related Skills

- **cloudflare-worker-base** - Worker setup and configuration
- **cloudflare-d1** - Database integration for queue consumers
- **cloudflare-r2** - Store large message payloads
- **cloudflare-workflows** - More complex orchestration needs

---

## Official Documentation

- **Cloudflare Queues**: https://developers.cloudflare.com/queues/
- **Configuration**: https://developers.cloudflare.com/queues/configuration/
- **Wrangler Commands**: https://developers.cloudflare.com/workers/wrangler/commands/#queues
- **Limits**: https://developers.cloudflare.com/queues/platform/limits/
- **Troubleshooting**: https://developers.cloudflare.com/queues/observability/troubleshooting/
- **Best Practices**: https://developers.cloudflare.com/queues/configuration/best-practices/

---

**Questions? Issues?**

1. Check `references/error-catalog.md` for all 10 errors and solutions
2. Review `references/setup-guide.md` for complete setup walkthrough
3. See `references/best-practices.md` for production patterns
4. Check official docs: https://developers.cloudflare.com/queues/