What Is Ollama and Why SpeedMVPs Uses It
Ollama was created to make local LLM deployment as simple as running a Docker container. Before Ollama, running an open-source LLM locally required navigating Python environments, CUDA drivers, quantisation tooling, and serving configuration. Ollama abstracts all of this into a single binary and a CLI that mirrors Docker's interface. ollama pull llama3 downloads the model, ollama run llama3 starts an interactive session, and ollama serve starts an OpenAI-compatible HTTP API. SpeedMVPs uses Ollama in every AI project as the local development environment for LLM calls. This allows the entire development team to run the AI layer locally without API keys, without cost, and without sending data to external services during development. The OpenAI-compatible API means that code written against Ollama's local endpoint works against OpenAI, Anthropic, or any other provider in production with a configuration change. For production, Ollama is the right tool when the client has a hard requirement for air-gapped deployment. Defence and government clients, NHS trusts with strict data processing policies, financial services firms with client data confidentiality obligations, and MHRA-regulated medical device software companies all have use cases where sending inference requests to a commercial API is either prohibited or extremely difficult to justify legally. Ollama on private cloud infrastructure solves this with minimal operational complexity compared to alternatives like vLLM.
Setting Up Ollama in a Production AI Project
For local development, Ollama installs on macOS, Windows, and Linux with a single download. After installation: ```bash ollama pull llama3.1:8b ollama serve # starts HTTP API on localhost:11434 ``` The local API is OpenAI-compatible. In your Next.js project, use the official OpenAI SDK pointed at the Ollama endpoint: ```ts import OpenAI from 'openai' const client = new OpenAI({ baseURL: process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434/v1', apiKey: 'ollama', // required by SDK but ignored by Ollama }) const response = await client.chat.completions.create({ model: 'llama3.1:8b', messages: [{ role: 'user', content: userMessage }], }) ``` In Docker Compose for local development, add Ollama as a service: ```yaml services: ollama: image: ollama/ollama ports: - '11434:11434' volumes: - ollama_data:/root/.ollama ``` For production on a private cloud server (AWS EC2, Azure VM, bare metal), deploy Ollama on a GPU-enabled instance. GPU acceleration dramatically improves throughput. The instance needs sufficient VRAM for the model: Llama 3.1 8B requires approximately 6GB VRAM in 4-bit quantisation, Llama 3.1 70B requires approximately 40GB. For cloud deployments, put Ollama behind a reverse proxy (nginx or Caddy) with authentication to prevent unauthorised access.
Key Features and Capabilities
The model library includes all major open-source LLMs: Llama 3.1 (8B, 70B, 405B), Mistral (7B, Large), Phi-3 (mini, medium), Qwen 2.5, Code Llama, and many more. New models appear in the Ollama library shortly after public release. OpenAI API compatibility means existing code using the OpenAI SDK works with Ollama with a base URL change. This portability is central to SpeedMVPs' development workflow: develop locally against Ollama, deploy to production against OpenAI or Anthropic, and have the option to deploy to a private Ollama instance if requirements change. Multimodal support is available on models like LLaVA and Llama 3.2 Vision, which accept image inputs alongside text. This extends Ollama's capability to image analysis and document processing tasks. Model management handles quantisation automatically. When you pull a model, Ollama downloads a quantised GGUF version appropriate for your hardware. This makes models significantly smaller and faster without requiring manual quantisation steps. Concurrent requests are supported: Ollama can handle multiple simultaneous requests, queuing them and batching where the model supports it. For multi-user applications, Ollama's throughput is limited by the underlying hardware but is sufficient for many private deployment scenarios. For EU AI Act compliance, using a self-hosted open-source model means your AI supply chain is entirely within your control. You can document exactly which model weights you are running, what training data was used (if known), and what your test and evaluation results show, all requirements for high-risk AI system documentation under the Act.
Real-World Workflow: Ollama in an AI MVP
SpeedMVPs used Ollama in production for a document intelligence product built for a UK financial services firm. The product analysed client-submitted financial documents for anomalies, extracted key figures, and generated structured reports. Compliance requirements meant no client financial data could be sent to external API providers. The architecture deployed Ollama on an AWS EC2 g4dn.xlarge instance (1x NVIDIA T4 GPU, 16GB VRAM) within the client's existing VPC. The instance had no public internet access: all ingress was via the internal load balancer and all egress was blocked at the security group level. Ollama ran Mistral 7B in 4-bit quantisation, which fit comfortably in 4GB VRAM. The application server (a Node.js service on ECS Fargate) called the Ollama API via the internal VPC endpoint using the standard OpenAI SDK. From the application code perspective, the only difference from a commercial API integration was the base URL. Throughput was sufficient for the expected volume of 100-150 documents per day. Average processing time per document was 15-30 seconds depending on length, which was acceptable for a background processing workflow. The client paid no per-token API costs and had complete control over model versioning and updates.
Cost and Pricing Considerations
Ollama itself is free and open source. The cost is the underlying compute infrastructure. For local development, it runs on existing developer hardware at no incremental cost. For production, GPU instance costs are the primary expense. AWS g4dn.xlarge (1x T4 GPU) costs approximately USD 0.526/hour on-demand, around USD 380/month. A g4dn.2xlarge (1x T4, more RAM and CPU) is approximately USD 0.752/hour. Reserved pricing for 1-year commitments reduces these costs by approximately 40%. For very high throughput requirements, vLLM is a more efficient serving option than Ollama on the same hardware, handling higher concurrent request volumes through advanced batching. vLLM requires more configuration but may be worth the operational investment at scale. Compare the private compute cost against commercial API costs for your expected volume. At high token volumes, self-hosted models are substantially cheaper. At low volumes, the fixed instance cost exceeds commercial API pay-per-use pricing. The break-even point depends on your specific model and volume; SpeedMVPs can help you model this during the discovery phase.
Alternatives to Ollama
For local development alternatives, LM Studio provides a GUI-based model management tool for macOS and Windows. Jan.ai is another desktop application. Both are simpler for non-developers but lack the API server functionality that makes Ollama useful for development environments. For production private inference, vLLM is the highest-performance open-source serving framework, supporting tensor parallelism across multiple GPUs and higher concurrent request throughput than Ollama. It is more complex to configure but appropriate for high-scale private deployments. Hugging Face TGI (Text Generation Inference) is another production-grade serving option with Hugging Face model hub integration. It requires more configuration than Ollama but has better performance characteristics for high-concurrency scenarios. For managed private deployment without self-hosting, Hugging Face Inference Endpoints in a private cloud region provide dedicated model instances in your chosen AWS or Azure region, with Hugging Face managing the serving infrastructure. This splits the operational burden without sending data through a shared public API. For teams without a hard air-gap requirement, commercial providers (OpenAI, Anthropic, Mistral) with appropriate DPAs are simpler to operate and maintain, and allow you to use frontier models that are not yet available in open-source form.