What Is Inference: A Plain-English Definition
Inference is the operational use of a trained AI model. Once a model has been trained, its weights are fixed. Every time you use it, you are running inference: passing new inputs through the frozen model weights to produce an output. The word 'inference' comes from the statistical concept of drawing conclusions from observations. In traditional statistics, inference means using data to draw conclusions about a population. In the context of AI models, inference means using the model to draw a conclusion about a new input, predicting its class, generating a response, scoring its relevance, or producing any other output the model was trained to generate. For language models, inference works by passing a tokenised prompt through the model's neural network layers. At each layer, the attention mechanisms compute relationships between tokens. The final layer produces probability distributions over the vocabulary, and the model samples from these distributions to select the next token. This repeats until the model generates a stop token or reaches the maximum output length. A critical distinction for product builders is that inference is not lookup. The model does not have a database of answers that it retrieves. It generates a new output each time by running the input through its learned representations. This is why the same prompt can produce slightly different outputs when run multiple times, why temperature settings affect output variability, and why larger models with richer representations generally produce better outputs than smaller ones. Inference happens across AI modality types. LLM inference processes text and generates text. Vision model inference processes images and generates classifications, bounding boxes, or descriptions. Audio model inference processes audio waveforms and generates transcriptions, translations, or new audio. Each modality has different compute requirements and latency profiles.
How Inference Works
Understanding inference in production requires understanding its three key dimensions: latency, throughput, and cost. Latency is the time from sending a request to receiving the complete response. For streaming LLM APIs, the relevant metric is often time to first token (TTFT), which determines when the user sees the first word, and time per output token (TPOT), which determines how quickly the response fills in. For non-streaming synchronous use cases, total latency is what matters. LLM inference latency varies from under one second for small models on fast hardware to 30-60 seconds for long outputs from large models. Throughput is the number of inference requests a system can handle per second. This is relevant when building systems that need to process large volumes in parallel, such as batch processing of documents or serving a large number of concurrent users. Cloud inference APIs handle throughput scaling automatically within rate limits. Self-hosted inference requires careful capacity planning. Cost is per-token for most hosted LLM inference APIs. OpenAI charges separately for input tokens (the prompt) and output tokens (the response). Output tokens are typically more expensive because they require serial generation one token at a time, while input tokens can be processed in parallel. Understanding your expected token volumes is essential for product economics. Consider a concrete UK example. A SaaS startup builds a product that analyses customer support tickets and generates suggested responses for human agents to review. Each analysis involves a 1,500-token input (system prompt plus ticket content) and a 400-token output (analysis and suggested response). Using GPT-4o at USD 2.50 per million input tokens and USD 10 per million output tokens, each inference costs approximately USD 0.0038 plus USD 0.004, around USD 0.008 per ticket. At 5,000 tickets per day, that is USD 40 per day in inference cost, or around GBP 1,200 per month. Knowing this number early lets the team design appropriate pricing and assess product economics before building.
Why Inference Matters for AI Product Development
Inference matters for product development because it is the primary ongoing cost and the primary source of latency in AI products. Getting inference right, in terms of cost, speed, and reliability, is the difference between a product with healthy economics and one that haemorrhages money at scale. Inference cost affects pricing strategy directly. If your product's primary feature is LLM-powered analysis, and each analysis costs USD 0.05 in inference, then your per-user marginal cost has a hard floor. Pricing needs to cover this with an appropriate margin. Teams that do not model inference costs before launch frequently discover their economics do not work when they try to scale. Inference latency affects user experience. Users tolerate 1-3 seconds of latency for AI features relatively well. Beyond 5-10 seconds, without streaming output, satisfaction drops significantly. Designing for latency means choosing models with appropriate speed characteristics, implementing streaming where possible, setting user expectations clearly, and building fallback behaviour for slow responses. Inference reliability affects product stability. Cloud LLM APIs have published uptime SLAs, but in practice all AI inference endpoints experience occasional elevated latency, error rate spikes, and brief outages. Production AI products need retry logic with exponential backoff, circuit breakers that prevent cascade failures, and graceful degradation paths for when inference is unavailable. For UK teams, inference through cloud APIs also has data residency implications under UK GDPR. Understanding where each provider runs inference, which data centres process your prompts, and what data retention policies apply is part of your GDPR compliance posture. Major providers offer EU data residency options, but these typically cost more and may have different rate limits than US endpoints.
Common Use Cases in Production AI Products
Real-time inference powers conversational AI products, copilot features, and live document analysis where users expect immediate responses. Streaming output, where the model's response appears word by word as it is generated, is the standard pattern for user-facing real-time inference because it significantly improves perceived responsiveness even when total latency is unchanged. Batch inference processes large volumes of inputs without real-time latency requirements. Nightly classification of support tickets, bulk analysis of uploaded documents, and scheduled report generation are common batch inference patterns. Batch processing typically costs less per token than real-time inference because providers can schedule it efficiently, and it can be parallelised across many simultaneous requests. Background inference supports features that run asynchronously while users do other things. A document uploaded for analysis might trigger a background inference job that runs for 30-60 seconds while the user navigates elsewhere in the product, with results available when they return. This removes latency from the user's critical path. Embedding inference converts text to vector representations for storage in vector databases. Unlike generative inference, embedding inference is much cheaper, faster, and deterministic. Embedding models like text-embedding-3-small process thousands of tokens per second at very low cost, making them practical for indexing large document collections. Multimodal inference processes combined text and image inputs. For products that need to analyse documents with charts and images, verify identity documents, or process product photos alongside descriptions, multimodal inference via APIs like GPT-4o Vision or Google Gemini handles the combined processing in a single API call.
Related Concepts
Large language models are the most commonly discussed inference targets for product teams. Understanding LLM inference characteristics, context window limits, token counting, and cost per token, is foundational for AI product economics and performance planning. Tokenisation determines how text is split into units for LLM inference and how costs are calculated. Every inference call is billed in tokens, not words or characters. Understanding that English text averages roughly four characters per token, that code is tokenised differently from prose, and that some languages are significantly more token-expensive than others helps predict costs accurately. Foundation models are the trained systems that inference runs on. The choice of foundation model determines the capability ceiling, the cost, and the latency profile of inference. Smaller models like GPT-4o-mini are faster and cheaper but less capable. Larger models are more capable but slower and more expensive. The right model for a given inference task depends on the required quality level and the acceptable cost and latency. Fine-tuning is the process of adapting a model through additional training. A fine-tuned model is then used for inference just like the base model, but potentially at different cost and latency profiles depending on the provider. Understanding the relationship between fine-tuning and inference costs is important for evaluating whether fine-tuning a smaller, cheaper model can match the inference quality of a larger, more expensive base model for your specific task. AI orchestration frameworks manage multiple inference calls, passing outputs from one to the next, managing state, and handling errors. In complex AI products, a single user action may trigger multiple inference calls to different models. The orchestration layer coordinates these calls and assembles the final output, making efficient inference management central to both product performance and cost control.