How Semantic Search Works
Semantic search relies on embeddings: dense numerical vectors that capture the semantic meaning of text. An embedding model processes an input text and produces a fixed-size vector, typically 768 to 3,072 dimensions depending on the model. The key property of a good embedding model is that semantically similar texts produce vectors that are close to each other in the vector space, as measured by cosine similarity or dot product, regardless of whether those texts share literal words. At query time, the user's search query is embedded using the same model, producing a query vector. The system then searches through all document vectors to find the ones closest to the query vector. This proximity reflects semantic similarity: documents that are conceptually related to the query rank highly even if they use different vocabulary. In practice, this retrieval is performed by a vector database optimised for approximate nearest-neighbour search across millions or billions of vectors efficiently. The result is a ranked list of semantically relevant documents returned in milliseconds.
Semantic Search vs Keyword Search
Keyword search, also called lexical search, matches documents that contain the exact words in the query, typically with weighting for term frequency, inverse document frequency, and other statistical signals. BM25 is the dominant keyword search algorithm underlying Elasticsearch, OpenSearch, and many other search systems. It is fast, interpretable, and excellent at exact match recall. It fails when users use different vocabulary from the documents, when the query expresses a concept not directly named in the relevant documents, or when multi-word phrases carry meaning beyond their individual words. Semantic search handles these cases because it operates on meaning, not tokens. However, it introduces its own failure modes: it can miss exact matches that the user genuinely wanted, it is sensitive to the quality and domain coverage of the embedding model, and it can retrieve topically related but factually different documents. The best production search implementations use hybrid search, combining lexical and semantic approaches and re-ranking results from both. This is now standard in production RAG systems and modern search products.
Embedding Models for Semantic Search
The quality of semantic search depends directly on the quality of the embedding model. Several factors affect model selection. Domain coverage matters: a general-purpose embedding model trained on web text may not produce well-calibrated similarity scores for highly technical content, legal documents, or medical records. Domain-specific models or models fine-tuned on in-domain data produce better recall for specialist content. The model must be the same for indexing and querying: documents and queries must be embedded by the same model for vector similarity to be meaningful. Switching models requires re-embedding your entire document corpus. Dimension size is a practical consideration: higher-dimension embeddings capture more nuance but cost more to store and search. Leading options in 2025 include OpenAI's text-embedding-3-large (3,072 dimensions, excellent general-purpose performance), Cohere Embed v3 (strong multilingual capability and domain adaptation support), and open-source options like E5-large and BAAI/bge-large that can be self-hosted to keep data within UK or EU infrastructure, relevant for GDPR compliance.
Semantic Search in RAG Systems
Retrieval-augmented generation systems use semantic search as the retrieval component. When a user asks a question, the question is embedded and used to retrieve the most semantically relevant chunks from a document corpus. These chunks are then included in the LLM's context as grounding material. The quality of the semantic search directly determines what information the LLM has available when generating its response. Poor retrieval means the LLM lacks the right context and either hallucinates or produces generic responses. Strong retrieval ensures the model has exactly the information it needs. Chunking strategy, the way documents are split before embedding, significantly affects retrieval quality. Chunks must be long enough to carry sufficient semantic meaning for accurate embedding but short enough that a retrieved chunk is tightly relevant to the query rather than containing mostly unrelated content. Typical chunk sizes range from 256 to 1,024 tokens, with 512 being a common starting point. Overlapping chunks, where consecutive chunks share a small boundary, improve recall of information that falls near chunk boundaries.
GDPR and Data Residency Considerations
Semantic search systems that index personal data carry GDPR compliance obligations. Under UK GDPR and EU GDPR, embedding personal data and storing those embeddings in a vector database constitutes processing personal data. The same rules apply as for any personal data: you need a lawful basis for processing, data must be retained only as long as necessary, and individuals have rights including the right to erasure. The right to erasure creates a specific technical challenge: deleting a person's data requires finding and removing all embeddings derived from documents containing that person's data. This is more complex than deleting a database row. Vector databases vary in their support for targeted deletion. Architecture decisions made at MVP stage, such as tagging embeddings with a user or data subject identifier, significantly reduce the effort required to fulfil erasure requests later. For NHS Digital connected products handling patient data, or products handling financial data subject to FCA oversight, data residency requirements may mandate that embeddings are stored within UK or EU infrastructure, which affects the choice between cloud-hosted vector databases and self-hosted alternatives.
Implementing Semantic Search with SpeedMVPs
SpeedMVPs builds semantic search and RAG systems as part of AI MVP delivery. A typical implementation includes embedding model selection matched to the client's content domain, vector database setup using Pinecone, Weaviate, or pgvector depending on scale and infrastructure preferences, chunking strategy design and testing, and hybrid search combining BM25 and vector retrieval with a re-ranking step where accuracy requirements are high. GDPR considerations are built in from the start: embedding metadata design includes the fields needed for efficient data subject request handling. For NHS Digital and healthcare clients, self-hosted embedding models and UK-region vector database instances are standard. All code and infrastructure configuration are transferred to the client on delivery. Projects from GBP 8,000, 2-3 week delivery from our Hemel Hempstead team.