What This Template Covers
The technical interview scorecard for AI engineers covers four evaluation areas that together distinguish engineers with genuine production AI experience from those with theoretical or superficial AI knowledge. The LLM knowledge assessment section covers the core concepts that any competent AI engineer building with large language models should understand deeply: tokenisation and context windows, prompt engineering principles, model selection trade-offs, inference optimisation, and the practical limitations of current LLM capabilities. The RAG architecture section covers the specific technical design skills required for retrieval-augmented generation systems, which are the most common architecture for enterprise AI products. A strong AI engineer should be able to design a RAG system from scratch, explain the trade-offs in each design decision, and diagnose common RAG failure modes. The system design rubric provides a structured framework for evaluating a candidate's ability to design a production AI system end to end: from data ingestion and model integration to deployment, monitoring, and cost management. The red and green signal checklist provides a rapid reference for assessing candidate responses in real time. Green signals indicate genuine depth. Red signals indicate surface-level knowledge that will not hold up in a production environment.
How to Use This Template Step by Step
Step one: decide the format and duration of the technical assessment. For senior AI engineer roles, a 90-minute technical interview is standard. Allocate: 20 minutes for LLM knowledge questions, 30 minutes for RAG architecture design, 30 minutes for system design, and 10 minutes for candidate questions. For a shorter interview (60 minutes), prioritise the RAG architecture section and one LLM knowledge area. Step two: prepare the LLM knowledge questions. Select two to three questions from each sub-area of the LLM knowledge section. The goal is not to quiz on memorised facts but to probe conceptual understanding and practical judgment. Good LLM knowledge questions ask candidates to explain a decision they would make and the reasoning behind it, rather than to recite a definition. Step three: prepare the RAG architecture design question. Use a realistic scenario from your product domain. For example: "We need to build a system that allows users to query a knowledge base of 10,000 internal documents. The system should return accurate, cited answers in under five seconds. Walk me through how you would design this." This open-ended format reveals how the candidate structures a complex problem, what questions they ask to clarify requirements, and how deep their knowledge of RAG components is. Step four: prepare the system design question. Use a real requirement from your product or a hypothetical that maps closely to your technical needs. The system design section should probe: the candidate's approach to architecture decisions (how they select between options), their understanding of production concerns (monitoring, cost, failure handling), and their communication of technical decisions to non-technical stakeholders. Step five: use the red and green signal checklist during the interview. Keep the checklist visible as a quick reference. When a green signal appears (specific naming of design trade-offs, honest acknowledgement of uncertainty, detailed account of a production failure and its resolution), note it. When a red signal appears (claiming AI can do things current models cannot reliably do, inability to discuss failure modes, generic answers to specific questions), note it. Step six: complete the scorecard within 30 minutes of the interview ending. Rate each section on the scoring rubric. Calculate the total score. Compare across candidates using the total score and the pattern of strong and weak sections. A candidate who scores strongly on LLM knowledge but weakly on system design is a different risk profile from one who scores the opposite way.
Section-by-Section Walkthrough
The LLM knowledge section should cover at minimum: context window management (how the candidate handles long documents or conversations that exceed the context window), prompt architecture (how they structure system prompts, few-shot examples, and output schemas), model selection (how they choose between models given constraints of cost, latency, and capability), and hallucination management (how they design systems to detect and handle incorrect model outputs). Strong answers in the LLM knowledge section are specific and practical. A candidate describing context window management should name specific techniques: chunking strategies, sliding window approaches, hierarchical summarisation, or retrieval-based context selection. Weak answers are generic: "we would use a large context window model." The large context model is one tool. A strong engineer knows its limitations and the conditions under which alternative approaches are superior. The RAG architecture section should evaluate five sub-components: document ingestion and preprocessing (how documents are parsed, cleaned, and prepared for embedding), chunking strategy (how documents are divided into retrievable segments and why), embedding and indexing (which embedding model, which vector database, which indexing approach), retrieval design (dense retrieval, sparse retrieval, or hybrid, and the re-ranking approach), and generation with citations (how the retrieved context is provided to the LLM and how citations are produced and verified). Strong RAG answers discuss trade-offs explicitly. "We would use smaller chunks for high-precision retrieval but risk losing context, or larger chunks for better context but noisier retrieval" shows genuine understanding. "We would embed the documents and store them in a vector database" shows only surface-level familiarity. The system design section should assess three dimensions: architecture breadth (does the candidate identify all the major components a production system needs?), architecture depth (can they elaborate on the design of any specific component with genuine specificity?), and production maturity (do they address monitoring, failure handling, cost management, and scaling concerns without prompting?). The red and green signal checklist is most valuable as a real-time tool. Keep it on a physical card or a second screen during the interview. Green signals: the candidate corrects a misconception in your question without prompting, the candidate describes a production failure they experienced and what they learned, the candidate asks clarifying questions before answering the design problem. Red signals: the candidate claims AI can do X reliably when current models do not, the candidate cannot explain a specific trade-off they supposedly made in a past project, the candidate defaults to the same model or tool for every use case without discussing alternatives.
Common Mistakes This Template Prevents
The most common AI engineer hiring mistake is evaluating on AI tool usage rather than AI engineering competence. A candidate who has used ChatGPT extensively, built several Cursor-assisted side projects, and deployed a Streamlit demo is not necessarily an AI engineer. The scorecard's specific questions about RAG design, production failure handling, and cost management distinguish tool users from engineers who can build reliable production systems. The second mistake is asking questions that reveal whether the candidate has memorised recent AI paper abstracts rather than whether they can build systems. Questions like "what are the latest advances in chain-of-thought prompting?" are not useful hiring signals. Questions like "describe a prompt engineering decision you made that changed the quality of your system's outputs, and how you measured the improvement" reveal practical engineering judgment. The third mistake is not assessing production experience explicitly. AI is a field where academic credentials and side project portfolios may not reflect the ability to build, deploy, and maintain a system that works reliably for real users. Ask specifically about production deployments: "Tell me about an AI feature you shipped to production. What was the deployment process, what went wrong, and how did you handle it?" The fourth mistake is not assessing communication skills for AI engineers. AI engineers who cannot explain their architectural decisions to product managers, founders, or customers create information silos that slow the team. Include at least one question that requires the candidate to explain a technical AI concept to a non-technical audience.
Customisation Tips for Different Project Types
For teams building AI agents and multi-agent systems, add a section specifically covering agent architecture. Ask: "How would you design an AI agent that needs to use multiple tools (web search, code execution, database queries) to answer a user's question? What are the failure modes and how do you handle them?" Strong candidates will discuss orchestration patterns, tool definition and error handling, state management across tool calls, and the conditions under which an agent should escalate to a human. For teams building in a regulated sector (fintech, healthtech), add compliance-aware AI assessment questions. Ask: "How would you design an AI feature for a credit assessment tool to ensure it does not discriminate against protected groups?" or "How would you implement explainability for an AI diagnosis support tool used by clinicians?" Strong candidates will reference fairness metrics, bias testing, the ICO's guidance on AI, and the EU AI Act's explainability requirements without needing to be prompted. For CTOs who are hiring the first AI engineer and are not deeply technical themselves, include a technical advisor in the interview process. A freelance AI engineer or a senior technical contact can conduct the RAG and system design sections while you focus on the communication, problem-solving approach, and cultural fit dimensions. Do not attempt to assess technical AI depth if you do not have the technical background to evaluate the answers. For senior AI engineer or AI lead roles, add a team and project leadership assessment. Ask about technical roadmap planning, managing junior engineers in an AI team, evaluating build-versus-buy decisions for AI components, and communicating AI strategy to non-technical leadership. The senior AI engineer role requires both deep technical competence and the ability to operate as a technical leader in a cross-functional environment.