The new routing layer can allocate inference requests based on the optimal model, priority, health status, and cost, providing a path for predictable AI expenditures.
Sunnyvale, California, USA, November 5th / PRNewswire / -- Rafay Systems, an infrastructure orchestration and monetization platform for neoclouds, telecommunications companies, and enterprises, today announced the addition of new capabilities that allow operators to control where each inference request is executed and what the cost is. The Token Router is a new routing layer that can send each request to model deployments within the cluster based on priority, health status, and cost, or to external providers such as OpenAI and Anthropic, as well as any endpoints that are compatible with OpenAI. Token Budgets also allows administrators to set daily or monthly expenditure limits for organizations, projects, or individual users, and to enforce these restrictions before requests are accepted.
The timing of this release reflects the changes in the consumption patterns of AI. Each task in the Agentic workflow consumes more of token, so even though the price of each token has decreased, the total cost AI is still rising. At the same time, the team is migrating high-throughput tasks to open-source models in order to control expenses. Operators now need to enable both cutting-edge models and open-source models to operate in parallel, while also managing the costs of each model.
By using Token Factory and Token Router together, operators can provide both cutting-edge models and open-source models through a single endpoint, and automatically switch traffic between them. Enterprises can initially allow their teams to use the cutting-edge models within the daily budget. Once the daily budget is exhausted, they can switch to the open-source models running on their own GPU. Applications that call these interfaces do not need to make any modifications.
These new capabilities are targeted at two types of users: one group provides open-source models as a service, and the other group consists of enterprises that develop their own models and manage their application operations internally.
One endpoint covers all models.
Token Router is located between the application and all inference execution locations. The operator manages it through a API, provides users with a set of credentials, and monitors token expenditures in a single view. Behind this stable endpoint, the operator can change the source of the model without requiring users to modify the code.
- External models: Token Factory can now be provided together with model deployments within the cluster, along with OpenAI, Anthropic, and any endpoints that are compatible with OpenAI. They all share the same API interface, cost tracking, and failover control mechanisms.
- Priority Routing: Each backend maintains an ordered list of provider groups. Traffic will first be directed to the group with the highest priority; if the primary group is unavailable, it will be switched to the next one. Common patterns include using self-hosted solutions first and then switching to cloud services during peak traffic times; using cutting-edge models initially and reverting to in-cluster models as a backup; as well as implementing multi-provider resilience across different providers.
- Perception of health status for routing and load balancing: Requests are evenly distributed among all healthy providers within the same group, which includes a combination of those deployed internally in the cluster and those API external to it. Providers that fail the health check will automatically be removed from the rotation and will rejoin once they have recovered.
- Measurement of perceived cost: Operators can set group rates for inputs, outputs, cache inputs, and cache writes token. Each request is billed according to the actual group rate of the service provided, and even if traffic is rerouted, showback, chargeback, and the billing can still remain accurate.
Not just stopping requests, but also managing the budget for directing traffic.
Request rate limits cannot make the expenses of AI predictable, as the cost of calling a small model may be just a fraction of a few cents, while the cost of initiating a long inference request for an advanced model is much higher. Token Budgets sets a limit directly in terms of amount:
- Three levels of management: Administrators can set budgets by organization, per project, and per user, which can be calculated on a daily or monthly basis, using the currencies supported by the service provider. Users' budgets will follow them across projects.
- Before acceptance: Each request is checked against all applicable budgets before it is accepted. When the budget is exhausted, the next step is determined by the operator's policy: rejecting subsequent requests, continuing to provide services, or reverting to a lower-cost model. Other teams can still continue to work within their respective limits.
- Default Values and Overwriting: New projects and new users will inherit the default budget. Administrators can increase the budget for more demanding workloads without having to modify the application or endpoints.
- Reminder and self-service visualization: Threshold reminders will notify administrators before approaching the upper limit. Developers can view their expenses and remaining budget in Rafay Developer Hub. The budget will be automatically reset daily or monthly.
With budget-based routing, Token Router can also respond to these budgets. When a tenant, team, or API key approaches or reaches their budget, Token Router will switch their traffic to a lower-cost group, such as a cheaper price or a self-hosted model, instead of directly rejecting the request. Users can still receive responses, and their expenses are kept within the limits set by the administrator.
Designed for neoclouds and enterprises token factory
neoclouds, which provides services for open-source models, can now add cutting-edge models to its product catalog on the very first day and offer them through the same API and credentials that customers are already using. Providers can sell daily quotas of access to these cutting-edge models to each customer, and any remaining traffic for the day is routed to the open-source models hosted on their own GPU. Customers get the models they need when they need them most, while providers maintain a high utilization rate of computing power. Each token is billed based on the actual rate of the services provided by the models. Tenant administrators can manage budgets for their projects and users, so providers do not need to oversee every individual developer.
For enterprises that operate token and factory, teams are allowed to use cutting-edge models without incurring unlimited billing. The platform team can set a daily quota for each team or application to use these models. Developers can use the cutting-edge models until the quota is exhausted; thereafter, Token Router will switch requests to open-source models running on the company's own GPU for the remaining time of the day. Applications continue to call the same endpoint, and the quota will be reset the next day. A dashboard with user filtering and CSV export features is helpful for internal showback and chargeback.
Rafay Systems Chief Product Officer Mohan Atreya stated: "Every operator we collaborate with hopes to provide the best models, without having to hand over their profit margins or budgets to others. Token Router and Token Budgets have turned this trade-off into a strategy. Operators decide which model will serve each request and what its cost will be, without the need for applications to change. neoclouds has found a way to maintain the utilization of their own computing power while selling access to cutting-edge models, and businesses in turn obtain predictable AI expenditures."
Rafay Token Factory There are also these new features
- Fence provider: Any fence service that implements LiteLLM Generic Guardrail API can inspect prompt words and responses in routing traffic, and apply the same policies to both models within the cluster and external models.
- Cache input token. Pricing: Operators can set a separate price for token provided by the prefix cache, and pass on the savings brought by caching to agentic, AI, and retrieval workloads.
- API key Rotation Strategy: The operations team can enforce the key expiration, automatic rotation, and set overlapping windows, while also issuing a reminder before the key expiration.
Availability
Token Router, Token Budgets as well as the aforementioned capabilities are now available to Rafay Token Factory customers. For more information or to request a demonstration, please visit https :// rafay.co / request-demo.
About Rafay Systems
Rafay Systems is a leading software provider that supports the operators of AI clouds, including neoclouds, telecommunications providers, enterprises, and sovereign AI operators. Rafay Platform enables organizations to operate their GPU and computing infrastructure through self-service automation, governance, and multi-tenancy. Its capabilities cover bare metal deployment, infrastructure lifecycle management, virtual machines, Kubernetes, GPU PaaS, AI development environments, as well as AI models that are deployed as token metered inference services using Rafay. By simplifying the orchestration and operations on this technology stack, Rafay helps operators improve GPU utilization rates and transform raw infrastructure into monetizable, producible AI services while maintaining security, consistency, and control. Rafay is a verified partner of NVIDIA AI Cloud Ready and initiative. For more information, please visit rafay.co.
Media contact: Angela Shugarts | Rafay Systems | angela @ rafay.co | 303-868-6294
OpenAI and Anthropic are trademarks of their respective owners. All other trademarks are the property of their respective owners. Source: Rafay Systems.
