Rate Limiting

Some tools are expensive or risky to run often: sending email, calling a paid API, uploading files. @RateLimit caps how many times a caller can invoke a tool within a time window. When the cap is reached, further calls fail with an error until the window resets.

import { ToolDecorator as Tool, RateLimit, ExecutionContext } from '@nitrostack/core';

export class EmailTools {
  @Tool({ name: 'send_email', description: 'Send an email' })
  @RateLimit({ requests: 10, window: '1m' }) // 10 calls per minute, per caller
  async sendEmail(input: EmailInput, ctx: ExecutionContext) {
    await this.emailService.send(input);
    return { success: true };
  }
}

How it works

Each call increments a counter for the caller. While the counter is within the limit, the handler runs; once it passes the limit, NitroStack throws an error and the handler doesn't run. The counter resets when the window ends.

Options

@RateLimit takes two required options and three optional ones.

Time windows

A window is a whole number followed by a unit: s for seconds, m for minutes, h for hours, or d for days. Any other format, such as '1 minute' or '1.5h', throws an error when the server starts.

@RateLimit({ requests: 5, window: '30s' })     // 5 calls per 30 seconds
@RateLimit({ requests: 1000, window: '1h' })   // 1,000 calls per hour
@RateLimit({ requests: 50000, window: '7d' })  // 50,000 calls per week

When the limit is reached

The tool call fails with Rate limit exceeded. Maximum <requests> requests per <window>. The model sees that message, so it can tell the user to wait. Set message to say something more specific:

@RateLimit({
  requests: 10,
  window: '1m',
  message: 'You can send up to 10 emails a minute. Try again shortly.',
})

The error passes through any exception filters on the tool, like any other error.

Rate limiting topics

Related