Project Overview and Business Context
The client was a UK fashion e-commerce retailer with approximately 8,000 active product listings across multiple categories. Their existing product recommendations were rule-based: bestsellers, new arrivals, and manually curated collections. These were not personalised to individual shoppers and performed roughly the same for all users regardless of their demonstrated preferences. The brief was to build a personalisation system that adapts product recommendations to each shopper's observed behaviour in the current session and across past sessions, surfacing relevant products from the full catalogue rather than a manually curated subset. The technical challenge in personalisation is matching shopper intent signals, which are sparse and noisy, to a large product catalogue in real time without introducing latency that harms the shopping experience. Semantic embeddings address the catalogue side: representing each product as a dense vector that captures its meaning beyond keyword overlap. Behaviour signal processing addresses the intent side: converting clicks, views, add-to-basket events, and purchase history into a real-time preference signal. GDPR compliance is a material consideration for e-commerce personalisation. Storing individual browsing behaviour and using it to personalise content requires a lawful basis. The client's approach uses legitimate interests for session-level personalisation (no consent required for temporary session signals) and explicit consent (cookie banner) for cross-session preference storage. Data minimisation principles mean that only behavioural signals necessary for personalisation are stored, and raw event data is aggregated and retained for only 90 days.
Technical Architecture and Stack Decisions
The architecture has three main layers: product indexing, behaviour signal processing, and real-time ranking. The product indexing layer runs as a weekly batch job on Vercel serverless functions. For each active product, it concatenates the product title, description, category tags, and material attributes into a structured text string and passes it to the OpenAI text-embedding-3-small model to generate a 1536-dimensional embedding vector. Vectors are stored in Pinecone with product metadata (price, category, stock status, image URL, product URL). In-stock products with updated attributes are re-embedded weekly; embeddings are stable for unchanged products. The behaviour signal layer processes three event types: product views (weighted 1), add-to-basket events (weighted 3), and purchase events (weighted 5). These events are collected client-side via a JavaScript snippet and posted to a Next.js API route that aggregates them into a session preference vector. The preference vector is an average of the embedding vectors of recently interacted products, weighted by event type and recency. This gives a single dense vector representing the shopper's current session intent. The real-time ranking layer queries Pinecone with the session preference vector to retrieve the top 20 most similar products by cosine similarity, filtered to in-stock products and excluding already-purchased or already-viewed items. Results are re-ranked by a diversity algorithm that ensures the top 10 recommendations span at least 3 different sub-categories. The personalised recommendation set is delivered to the frontend as a JSON response in under 80ms for 95% of requests.
Key AI and ML Components
OpenAI text-embedding-3-small is the core AI component. Product embeddings capture semantic relationships between products that keyword-based systems miss: a shopper who views a linen blazer sees recommendations for linen trousers and tailored shorts, not just other blazers, because the embedding space groups linen tailoring products by material and style context rather than product category alone. This cross-category relevance is the primary value of embedding-based personalisation versus category-filtered recommendations. The preference vector aggregation is the second key component. Averaging embedding vectors of interacted products to produce a session-level preference representation is a well-established technique that requires no model training and updates in real time as the shopper browses. The recency weighting (more recent interactions weighted higher) ensures the preference vector reflects the shopper's current intent even if it differs from their historical purchase pattern. A shopper who typically buys casual clothing but is currently browsing formal wear for an event will see formal recommendations within a few product views. The diversity re-ranking algorithm is deterministic, not AI-driven. Ensuring recommendation diversity prevents the personalisation engine from showing ten near-identical products to a shopper whose recent interactions are all within one narrow sub-category, which would reduce the recommendation set's utility.
Challenges Solved and How
Cold start for new visitors with no session history is handled by reverting to popularity-based recommendations (bestsellers weighted by category) for the first three product views, then transitioning to embedding-based personalisation once a preference signal has accumulated. The three-view threshold was calibrated from A/B testing data: below three interactions, the preference vector is too noisy to outperform popularity-based defaults. GDPR consent management for cross-session personalisation required a clear cookie consent flow. Session-level personalisation uses only in-session memory and requires no consent. Cross-session personalisation (remembering preferences from a previous visit) stores a pseudonymous visitor ID and a compact preference profile in a first-party cookie, requiring explicit consent under UK PECR. The consent banner is integrated with the personalisation system: when a visitor declines cookie consent, the system falls back to session-only personalisation. Embedding freshness for a fast-moving fashion catalogue was addressed by the weekly re-embedding run and an on-demand re-embedding trigger for new product additions, ensuring that new arrivals enter the recommendation pool within hours of going live rather than waiting for the next weekly batch.
Outcome and Measurable Results
The client ran an A/B test for eight weeks with 50% of traffic receiving personalised recommendations and 50% receiving the previous rule-based recommendations. Click-through rate on recommendation widgets increased by 34% in the personalised variant. Conversion rate from recommendation click to purchase was 4.1% for personalised recommendations versus 2.8% for rule-based, a 46% improvement. Average order value for sessions that included a recommendation click was 12% higher in the personalised variant, attributed to the cross-category recommendations surfacing complementary items the shopper would not have found through navigation alone. Return visitor rate over a 30-day window increased by 8% in the personalised variant, suggesting that relevant recommendations improved overall shopping satisfaction beyond the immediate session.
Lessons for Similar Projects
Embeddings over keywords for catalogue-rich personalisation. A keyword-based recommendation system will show a shopper who bought a red dress more red dresses. An embedding-based system shows them complementary items across categories that share aesthetic and contextual relevance. The semantic understanding is what makes personalisation feel intelligent rather than mechanical. Start with session-level personalisation before adding cross-session complexity. Session-level personalisation is effective, GDPR-simple, and requires no consent infrastructure. Validate the lift with an A/B test before investing in the cross-session storage and consent layer. Diversity algorithms matter as much as relevance algorithms. A personalisation engine that shows ten identical products loses the shopper's attention on the second scan. Build diversity constraints into the ranking layer from the start. Make re-embedding fast and cheap. A fashion catalogue changes rapidly. A weekly batch re-embedding is a minimum; an on-demand trigger for new product additions is essential.