How I Solved Rate-Limit Exceptions in DSPy

The Problem

While working on a DSPy project that used OpenAI’s API, I kept running into rate-limit exceptions that crashed my runs.

Here’s the error I was seeing:

openai.error.RateLimitError: You have hit the rate limit...

Not great when you’re in the middle of chaining modules or running evaluations.

My Quick Fix

Instead of switching libraries or adding a dependency, I wrote a simple retry helper that uses exponential backoff.

Here’s the code:

import openai
import time

def retry_with_backoff(func, *args, retries=5, base_delay=1, **kwargs):
    """
    Retry the given function with exponential backoff on RateLimitError.
    
    Parameters:
        func: The function to call (e.g., module.predict)
        args: Positional arguments for the function
        kwargs: Keyword arguments for the function
        retries: Number of retries
        base_delay: Base delay in seconds (doubles each retry)

    Returns:
        The result of func(*args, **kwargs), or raises after max retries.
    """
    for i in range(retries):
        try:
            return func(*args, **kwargs)
        except openai.error.RateLimitError:
            wait_time = base_delay * (2 ** i)
            print(f"Rate limit hit. Retrying in {wait_time} seconds...")
            time.sleep(wait_time)
    raise Exception("Failed after retries.")

And here’s how I use it with DSPy:

result = retry_with_backoff(my_dspy_module.predict, inputs)

Done

Why It Works

Rate-limit errors mean you’re sending too many requests too fast. The API just wants you to slow down and try again. This backoff logic gives it time.

There are libraries like tenacity, and some wrappers built into the openai lib, but this lightweight version works perfectly for my prototyping needs.

What I Learned

• DSPy doesn’t yet have native rate-limit handling (as of writing)
• Backoff strategies are simple and effective
• Writing tiny utilities like this makes debugging smoother in the long run

Related

• DSPy GitHub
• OpenAI Rate Limits Docs

Let me know if you’ve found a cleaner way to handle this — or if DSPy adds built-in support for it later. Meanwhile, this little helper saved me a bunch of crashes.