What Monitoring Is and What It Covers
Monitoring is the practice of collecting predefined metrics and setting alerts when those metrics cross defined thresholds. It answers the question: is the system working as expected? You monitor HTTP error rates and alert when they exceed 1%. You monitor database connection pool usage and alert when it approaches the limit. You monitor disk space and alert before it fills. You monitor API response time and alert when it degrades. Monitoring is reactive and predefined: you have identified the things that can go wrong and created instruments to detect them. The power of monitoring is simplicity and speed: a well-configured monitoring system can alert you to a known failure mode within seconds of it occurring. The limitation is that monitoring only catches what you anticipated. If a new failure mode emerges that you did not anticipate and did not define a metric for, monitoring will not detect it. For traditional SaaS applications where failure modes are well-understood, comprehensive monitoring may be sufficient. For AI products with non-deterministic LLM behaviour, monitoring alone is not enough.
What Observability Adds
Observability is the property of a system that makes it possible to understand its internal state from its external outputs. The term comes from control theory and was applied to software systems to describe the ability to ask arbitrary questions about what a system is doing without having pre-instrumented those specific questions. An observable system lets you investigate a user's complaint that their AI assistant gave them incorrect information by retrieving the trace of their session, seeing the exact prompt sent to the LLM, the context retrieved from the vector database, the model's response, and every intermediate step. You did not need to have anticipated this specific failure and pre-built a monitor for it. The observability data, structured logs, traces, and detailed metrics, is rich enough to answer novel questions. The three pillars of observability, metrics, logs, and traces, work together to provide this capability. Metrics show you that something is wrong. Logs provide the detailed events around the time of the problem. Traces show the path a specific request took through your distributed system.
Why AI Products Need Both
AI products need monitoring for the same reasons as any software product: to be alerted when servers are down, error rates are high, or performance has degraded to an unacceptable level. These are the known failure modes. They need observability for the category of problems unique to AI: output quality degradation that does not manifest as a server error, prompt injection attempts, unexpected LLM model behaviour following a provider-side model update, RAG retrieval returning irrelevant context, and the subtle drift where user satisfaction with AI outputs declines over weeks without any single alertable event. A user whose AI document summarisation returns technically valid but superficially unhelpful summaries is not generating error logs. Monitoring will not catch this. An observable system, with captured prompt inputs and outputs, user feedback signals, and quality evaluation pipelines, will surface this pattern across users and over time. The combination of monitoring for hard failures and observability for quality and behaviour investigation is the complete picture for production AI products.
Practical Implementation for Small Teams
For an early-stage AI startup, the implementation of monitoring and observability should be practical and incremental rather than an attempt to build a complete system from day one. Start with monitoring: configure uptime checks for your application's health endpoint, set up error rate and latency alerts in your cloud provider's native monitoring or in a tool like Grafana Cloud, and subscribe to your LLM provider's status page. This takes a day and catches the majority of hard failures. Then layer observability incrementally. Add structured logging for LLM API calls capturing prompt template, model, token counts, latency, and error type. Add request tracing for your critical AI workflows using OpenTelemetry. Add a LLM-specific observability tool such as LangFuse to track prompt versions and output quality. Each addition expands your visibility into how the system is behaving. Avoid the trap of setting up a comprehensive observability platform before you have real traffic and real failure modes to investigate: the operational overhead of a full Prometheus, Grafana, Tempo, Loki stack before you have meaningful load is a distraction from building the product.
Alerting: Getting the Balance Right
The primary practical failure of monitoring in real systems is not insufficient alerting but excessive alerting. Alert fatigue, where so many alerts fire that engineers stop responding to them attentively, is the most common monitoring dysfunction. The solution is to alert only on signals that require human action and to clearly distinguish severity levels. Critical alerts require immediate response regardless of time: service is down, error rate is critically high, security incident detected. Warning alerts require investigation during business hours: elevated error rate, degraded latency, unusual cost spike in LLM usage. Informational notifications require no action: scheduled maintenance completed, deployment successful. For AI products, tune thresholds to reflect actual user impact. A 2% LLM error rate that is fully handled by retry logic with no user-visible effect should be a warning, not a critical alert. A 15% error rate where users see failures should be critical. Review and tune your alert thresholds regularly as you learn which conditions genuinely require urgent response versus those that resolve themselves.
SLIs, SLOs, and Observability
Service Level Indicators and Service Level Objectives connect observability data to business commitments. An SLI is a measurable indicator of service quality, such as the fraction of requests completing in under 2 seconds or the fraction of LLM inference calls returning a valid response. An SLO is the target value for an SLI, such as 95% of requests should complete in under 2 seconds or 99% of LLM calls should return a valid response. Observability tooling measures the SLIs from real traffic data. Monitoring alerts when SLIs trend toward violating SLOs, using error budget concepts to distinguish normal variance from concerning trends. For AI products, defining SLOs that include AI-specific quality indicators, such as user acceptance rate for AI outputs, is more meaningful than purely infrastructure SLOs. An AI product whose infrastructure is healthy but whose AI outputs are being consistently rejected by users has a real quality problem that traditional infrastructure SLOs would not capture.