LLM SERVING PLATFORM
From a question
to a real model response.
A small, working inference platform. Start with Chat; open Admin when you want to manage model metadata or inspect the gateway.
Open Chat →01 / TRY IT
Talk to the model
Type a question and send. The default backend and model are already configured.
Start a conversation →02 / OPERATE ITManage and inspect
Register a version, update its stage, or inspect routing and response-cache statistics.
Open Admin →What happens behind the scenes?
Browser → API gateway → GPU inference → Streamed answer
The gateway handles routing and response caching. Ray Serve and vLLM run the model on a rented GPU. A deployed website does not imply that the GPU is always online.
Read the architecture and test evidence on GitHub ↗