Coordinate requests
The alpha contains an HTTP proxy and gRPC interfaces, with model-aware server selection, load counters and session mapping. These provide a starting point for evaluating request routing across Ollama instances.
Local inference · open-source alpha
olol
Your local inference fleet.
An open-source starting point for coordinating multiple Ollama instances. Bring request routing, model availability and cluster status into one place, then evaluate it against your own machines and workload.
Current alpha foundation
Start with the fleet, not another endpoint.
The public implementation brings together the components of an inference gateway. It is a foundation for early evaluation, not a claim of validated availability, compatibility or performance.
See the fleet
Model registries, node discovery, status endpoints and a console view bring cluster information together. Evaluate how that view tracks the models and machines in your environment.
Work in the open
Python source, command-line tools and synchronous and asynchronous interfaces are available under the MIT license. The current package is alpha; compatibility and operating behavior need validation for each setup.
Forward-looking discussion
Next release — under discussion
These are directions for a next release, not capabilities offered by the current alpha. Scope and delivery remain subject to implementation and validation; no version, date or performance improvement is promised.
Real layer handoff
Explore a runtime path that passes actual intermediate model state between machines, with explicit boundaries for tokenization, tensor exchange and output decoding. A partition plan alone is not distributed execution.
Capability-aware placement
Base placement on measured node capabilities and exact model requirements. Make memory limits, model identity and any change in quantization visible instead of treating an estimate as available capacity.
Predictable failure behavior
Define what happens when a node disappears, a request stalls or a model is unavailable. Evaluate health transitions, timeouts, backpressure and recovery without silently changing the requested work.
Routing independent requests across machines and splitting one model across machines are different capabilities. Real cross-host layer execution remains a development direction, not a shipping claim.
Evidence before promises
What an early evaluation needs to prove.
Does the supported path work?
Pin the Ollama and Python versions, then exercise installation, model discovery, routing and streaming with real requests.
What changes under load?
Measure latency, throughput and failure behavior against a declared baseline. Report the hardware, model and workload behind each result.
Where is the trust boundary?
The current alpha assumes trusted networking. A broader deployment needs validated controls for node enrollment, transport, access and resource limits.
Early evaluation
Bring your machines. Bring a concrete workload.
Tell us which models you run, where they run and what gets in the way. We can discuss an evaluation scope and the evidence needed to decide whether olol fits your fleet.