Higress AI Routing Issues: 404 Errors On /v1/chat/completions

by Alex Johnson 62 views

Introduction: The Problem of Path Rewriting

AI routing is a crucial component in modern applications, especially those dealing with AI services. However, when the routing logic inadvertently alters the request path, it can lead to frustrating issues. This article addresses a specific problem: Higress, an API gateway, is automatically rewriting the path of /v1/chat/completions requests to /chat/completions. This seemingly minor change results in a 404 error for the client, disrupting the expected flow of information. We'll dive into the details of the issue, explore the root cause, and discuss potential solutions. Understanding how this path rewriting occurs and how it impacts your application is essential for maintaining a stable and functional AI service.

Let's be clear; this isn't just a simple inconvenience. It's a breakdown in the expected behavior of the system. The client expects to send a request to /v1/chat/completions, and the gateway should ideally pass this request along without modification, or with modifications that are expected and handled appropriately by the backend service. Instead, the gateway is changing the path, which means the backend is not receiving the correct information. The consequences range from failed API calls to incomplete data processing, impacting the user experience. The challenge here is to identify why the gateway is changing the path and find a reliable method to preserve it.

This highlights the importance of carefully configuring and monitoring your API gateway. A misconfiguration can lead to unexpected behavior and significant downtime. As you work with Higress or similar API gateways, be sure to have a strong understanding of how they handle path matching, rewriting, and forwarding. This awareness can help you quickly pinpoint the source of such problems, allowing you to resolve the situation and keep your AI services running smoothly. A well-configured system ensures that the routing rules align with your intended API design, reducing the likelihood of such errors and guaranteeing that your clients can successfully access the resources they require.

Detailed Issue Description and Steps to Reproduce

Let's delve deeper into what's happening. The user reports a scenario where the application sends a request to /v1/chat/completions. However, after passing through the Higress gateway, the path is modified to /chat/completions. Consequently, the client receives a 404 Not Found error. The problem stems from the gateway rewriting the URL path unexpectedly.

The user provided valuable diagnostic information: A routing configuration screenshot illustrates the route setup within Higress. Additionally, the user includes the curl command used for the test, along with the server response, which clearly shows the 404 error. Most importantly, the user shared detailed plugin logs that pinpoint the moment when the path is rewritten. The logs show several crucial steps in the request's journey. At first, the ai-proxy plugin logs show [OverwriteRequestPath] originPath=/apigateway/v1/chat/completions, mappedPath=/chat/completions, indicating that the path is, indeed, being rewritten. Then, the ai-route-rewrite plugin logs record the new path as /chat/completions. This detailed evidence helps narrow down the problem to a specific area within the Higress configuration.

The steps to reproduce the issue are as follows:

  1. Configure AI Service Provider: Set up the AI service provider, which acts as the backend service that will handle the requests.
  2. Configure Routing: Set up the routing rules to match any path prefix, such as /. This configures the gateway to intercept requests matching certain patterns.
  3. Send Request: Make a request to the gateway using the path /v1/chat/completions. You can use curl or any other HTTP client.
  4. Observe the 404 Error: Check if the client receives a 404 error, confirming the path rewriting problem.

This simple and direct method allows anyone to replicate the issue and understand the underlying problem. It underscores the importance of proper testing and validation when setting up your API gateway to ensure it behaves as expected and forwards requests accurately. By following these steps, you can quickly determine whether your setup faces this path rewriting problem and take the necessary steps to fix it.

Analysis: Pinpointing the Root Cause

So, what's causing this unexpected path rewriting? Based on the logs, the ai-proxy plugin seems to be the culprit. The logs show that it is actively changing the path as part of its processing logic. This suggests the plugin has been configured to perform such rewriting, perhaps with the intention of simplifying the path for the backend service or to add some level of abstraction. The fact that the path is rewritten before the ai-route-rewrite plugin processes it further supports this conclusion.

The user's hypothesis that the ai-proxy plugin is the issue is further strengthened by the log entries showing the overwrite of the path. When the ai-proxy plugin rewrites the path from /apigateway/v1/chat/completions to /chat/completions, this leads the downstream service to report a