We put a capable open model on an ordinary Windows machine to see how far local AI has come. The interesting part was not the speed. It was what it opens up commercially.
Most conversations about AI in the channel still assume the cloud. A customer signs up to a platform, data goes off to a provider, and the commercial model is somebody else's subscription. That is a reasonable default, and for a lot of work it is still the right answer.
But it is worth checking your assumptions occasionally. So we installed Ollama on a standard Windows desktop with an RTX 4070 and ran Qwen 3.5 9B locally, using the GPU for inference, to see what a business-grade machine can now do without any connection to a cloud provider.
- Model
- Qwen 3.5 9B
- Runtime
- Ollama on Windows
- Hardware
- RTX 4070, consumer GPU
- Generation speed
- Around 66 tokens per second
Around 66 tokens per second is comfortably faster than most people read. It is not frontier-model quality, and it is not trying to be. It is a competent general model producing usable output on a machine that would not look out of place under a desk in any office in the country.
Verbose output from the run. Eval rate 66.21 tokens per second on a consumer GPU.
Why this matters commercially
The technical achievement is not the story. Running a model locally has been possible for a while. What has changed is the threshold. The hardware required has dropped to something a business already owns or can buy for a few hundred pounds, and the setup is now a download rather than a project.
That shifts the question from whether this can be done to where it makes more sense than the alternative. Three situations stand out.
Where data genuinely cannot leave
Some customers have contractual, regulatory or client-imposed restrictions that make cloud AI a difficult conversation. Legal, healthcare, defence supply chain, financial services, parts of the public sector. For those customers a local model is not a cheaper option. It is the only option that gets past their own compliance team.
Where the workload is repetitive and high volume
Document classification, ticket triage summaries, data cleansing, extracting structured fields from unstructured text. Individually low value, collectively enormous. Per-token pricing on high-volume repetitive work adds up quickly, and it is exactly the sort of work a mid-sized open model handles competently.
Where the data is internal and always on
Retrieval over internal documentation, historical tickets, contracts, engineer notes and process guides. This is the use case most MSPs recognise instantly, because they are sitting on years of it and nobody can find anything.
Private AI is not going to replace the big providers. It is going to sit alongside them and take the work that never needed to leave the building.
What local AI does not solve
This is where the channel conversation usually goes wrong, so it is worth being blunt.
- Local is not automatically secure. The model stops sending data to a third party. It does not secure the machine, control who can query it, log what was asked, or stop someone copying the index. Everything you would normally do around access control and data governance still applies.
- Quality is lower than frontier models. For complex reasoning, long documents and anything customer-facing that has to be right first time, the large cloud models are still meaningfully better. Pretending otherwise sets a customer up to be disappointed.
- Someone has to own it. A local model is infrastructure. It needs a machine, updates, monitoring and a person whose job it is. That is an operating cost, not a one-off install.
- Licensing needs checking. Open weights do not always mean unrestricted commercial use. Read the licence before you build a service on it.
None of that makes it a bad idea. It makes it a service, which is the point.
The opportunity for MSPs and IT providers
If you are an MSP or IT provider, the interesting question is not whether you should run a local model internally. It is whether private AI becomes a line on your price list.
The shape is familiar. Hardware, deployment, configuration, integration with the customer's document estate, ongoing management, and a monthly fee attached to a machine sitting in the customer's own building. It looks a lot like every other managed service you already sell, and it lands in a conversation customers are already having internally.
The businesses that get there first will not be the ones with the best technical understanding of the models. They will be the ones who packaged it, priced it and trained their salespeople to open the conversation. That gap, between the technology being available and the proposition being sellable, is the same gap we see across the channel on almost every new product line.
This is the work, not the theory
Getting from a technology that works to a service you can sell involves proposition design, pricing, packaging, delivery model, onboarding and a sales team that knows how to raise it. That sequence is what our workshops on practical AI and new propositions are built around.
What we are testing next
This was a first pass. The next round is aimed at the things that would have to hold up before any of it is worth selling:
- Retrieval over a real document set, and whether the answers stand up
- An internal knowledge assistant against historical tickets and process documentation
- Customer and CRM data analysis where the data is commercially sensitive
- Comparing open models and working out the real hardware requirement at each level
- Honest boundaries on where cloud remains the better answer
The same model through a normal chat interface. No cloud service behind it.
We will publish what we find, including the parts that do not work.