introduction
Last week, I spent three days debugging a custom model routing workflow I built for an internal developer tool. This tool supported cross-functional teams across engineering and customer support, so any disruptions to its functionality delayed access to generative AI-powered support for customer inquiries, making the debugging process high-priority beyond just fixing a broken workflow. The setup was designed to route user prompts to available model endpoints, with an explicit fallback to a secondary endpoint if the primary started experiencing high load. But every time we sent a batch of concurrent test requests, the logs showed errors that looked like standard rate limiting being triggered, even though we’d configured per-endpoint rate limits correctly and double-checked our throttle settings. What made the issue more confusing was that queueing upstream looks nothing like rate limiting in the logs: the upstream service’s own logs didn’t mention rate limits at all — instead, they showed queued requests that had timed out. During a peak test run ahead of a company-wide demo, this failure mode became particularly pronounced: a group of team members ran simultaneous validation requests, and the fallback never activated, leaving all test requests to time out instead of rerouting to the secondary endpoint. The logs showed 503 Service Unavailable errors, but our routing layer only logged 429 errors from a prior test, making it impossible to quickly tie the two issues together.
We started our troubleshooting by cross-referencing our routing layer logs with the upstream endpoint’s detailed access logs. We quickly realized that the 429 Too Many Requests errors we’d been seeing weren’t coming from the upstream model service. Instead, the upstream platform was holding excess concurrent requests in an internal queue, and only returning 503 Service Unavailable errors after the queue’s timeout threshold was hit. Our fallback logic had been programmed to activate only when we received a 429 response, so we were missing the window to shift traffic to the secondary endpoint before queued requests started timing out for end users.
To fix this, we needed a reliable way to track pending in-flight requests per endpoint in real time, so we could trigger the fallback before the upstream queue backed up to its timeout limit. I’d used comparable open-source projects for similar workflow orchestration before, but none had a clear, actionable breakdown of how to map pending request counts to fallback triggers. After searching through community developer resources, I found a guide that walked through exactly this kind of queue calculation and fallback setup: FastGPT documentation. The guide outlined a straightforward formula to count pending requests: subtract the number of completed requests from the total requests sent to the endpoint over a rolling window, then use that count to trigger the fallback when the pending number exceeded a predefined threshold aligned with the upstream queue’s documented maximum capacity. We implemented this formula in our routing layer, updating our fallback logic to activate as soon as the pending request threshold was hit, rather than waiting for error codes to appear.
Before committing the updated routing layer code, we compiled a short verification checklist to catch similar issues in future deployments:
1. Confirm fallback logic triggers on both documented upstream error codes and elevated in-flight request counts
2. Cross-reference the pending request calculation formula against upstream service queue documentation to align thresholds appropriately
3. Validate that logging captures both in-flight request totals and upstream response codes for every routed request
4. Test concurrent request flows to confirm fallback activates before timeouts occur, rather than only after errors are returned
Final Thought
The biggest caveat we ran into during this process is that this approach relies entirely on accurate in-flight request tracking from our own routing layer. If the upstream platform silently modifies its queue behavior or drops requests without logging, our local count will be inaccurate, leading to either premature fallback triggers or missed queue backlogs. We also learned that different upstream endpoints may have varying queue limits and timeout settings, so the threshold for triggering fallbacks needs to be adjusted per endpoint, rather than using a one-size-fits-all value. Additionally, we added dedicated logging for pending request counts alongside our existing error logs, which has made it far easier to identify queue backlogs before they lead to user-facing timeouts. Since implementing these changes, our concurrent request throughput has stayed consistent, and end users no longer see unexpected timeout errors.
