What This Template Covers
The CI/CD pipeline checklist covers six pipeline stages that together define a production-grade delivery process for an AI product. The workflow structure section defines the trigger conditions (push to main, pull request to main, scheduled evaluation runs), the job dependencies (which jobs must complete before others start), and the environment matrix (different configurations for staging and production deployments). The static analysis stage covers type checking, linting, and formatting checks. These are fast checks that catch a large class of errors before the more expensive test stages run. The test stage covers unit tests, integration tests, and end-to-end tests. For AI products, this stage also includes the non-AI test coverage: the logic that processes AI inputs and outputs, the API endpoints, and the data layer. The LLM evaluation stage is specific to AI products. It runs the AI components against a curated evaluation dataset and reports quality metrics. This stage can block deployment if quality falls below a defined threshold. The build stage covers production build, asset optimisation, and any compilation steps required before deployment. The deployment stage covers staging deployment, smoke tests against the staging environment, production deployment using a zero-downtime strategy (blue-green, rolling, or canary), and post-deployment verification. The notification and rollback section covers how the team is notified of pipeline results and how a failed deployment is reversed.
How to Use This Template Step by Step
Step one: define your trigger strategy. At minimum, the pipeline should run on every push to the main branch and on every pull request that targets main. Add a scheduled trigger for comprehensive LLM evaluation runs that are too slow to run on every commit: daily or weekly depending on your evaluation dataset size and the cost of model inference during evaluation. Step two: set up the static analysis jobs. These should run first because they are fast and catch errors cheaply. Required checks: TypeScript type checking (tsc --noEmit), ESLint with your project's configured rules, and Prettier format checking. These checks should run in parallel and complete in under two minutes combined. Step three: configure the test suite. Separate unit tests from integration tests. Unit tests (testing individual functions and components in isolation) should run in under five minutes. Integration tests (testing database interactions, API endpoints, and external service mocks) take longer. For AI products, add a test category for the non-AI business logic that processes and validates AI inputs and outputs. Use mocked AI responses for these tests to avoid test flakiness from variable AI outputs. Step four: set up the LLM evaluation stage. Create an evaluation script that: loads your evaluation dataset, runs each input through the AI component, scores each output against the expected output, and reports aggregate metrics (accuracy, F1, task completion rate, or whichever metrics are defined in your PRD). Set a minimum passing threshold for each metric. The evaluation stage should run against the current code and compare to the last passing evaluation to detect regressions. Step five: configure the build step. For a Next.js application, this is npm run build or pnpm build. The build step should fail if there are TypeScript errors, missing environment variables, or any compile-time issue. Store build artifacts for the deployment step. Step six: set up staging deployment. Deploy to staging automatically on every successful pipeline run from the main branch. Run a smoke test suite against staging that verifies: the application is accessible, authentication works, the database connection is healthy, and the AI component returns a valid response for a test input. Fail the pipeline if smoke tests fail. Step seven: configure production deployment. Use a zero-downtime deployment strategy. For Vercel or Railway, this is handled automatically. For self-hosted infrastructure, implement either blue-green deployment (deploy to inactive environment, switch traffic) or rolling deployment (replace instances one at a time). Add a manual approval gate before production deployment if the team prefers that control.
Section-by-Section Walkthrough
The GitHub Actions workflow structure section uses a jobs object with explicit needs relationships. The recommended job order is: static-analysis (no dependencies, runs immediately), test (needs: static-analysis), evaluate-llm (needs: test, can run in parallel with other test jobs), build (needs: test), deploy-staging (needs: build), and deploy-production (needs: deploy-staging). The static analysis job should use a cached node_modules installation to keep it fast. Use the actions/cache action with a cache key based on the hash of the package lock file. Restoring from cache typically reduces installation time from 60 to 90 seconds to 5 to 10 seconds. The test job should set up a test database (PostgreSQL in a service container is standard for GitHub Actions) and run integration tests against it. Use a separate test database for each pipeline run to prevent test interference. Seed the test database with deterministic fixtures using a dedicated seeding script. The LLM evaluation job needs careful design to be useful rather than just slow. Key decisions: how large is the evaluation dataset (start with 50 to 100 examples, grow as needed), what is the maximum cost budget per evaluation run (set this as a guardrail), and what is the threshold for blocking deployment versus warning? For evaluation runs that are too expensive to run on every commit, use a separate scheduled workflow that runs nightly and posts results to a dashboard. The deployment jobs should use GitHub Environments for production deployments, which enables: environment-specific secrets, deployment protection rules (requiring reviewer approval), and deployment history tracking. This is particularly valuable for regulated products in fintech or healthtech where deployment approvals may need to be documented. The notification section should post pipeline results to Slack or Teams. Failed pipelines should generate an alert that includes: which job failed, the error message, a link to the full logs, and which commit triggered the failure. Successful production deployments should also be announced with the deployment URL and a brief summary of what was deployed.
Common Mistakes This Template Prevents
The most common CI/CD mistake for AI products is not having an automated LLM evaluation step at all. Teams that rely on manual testing of AI outputs after each change are unable to deploy quickly because manual testing does not scale. Automated evaluation with defined quality thresholds allows rapid iteration with confidence. This checklist makes LLM evaluation a first-class pipeline stage, not an afterthought. The second mistake is running all pipeline stages sequentially when many can run in parallel. A pipeline that takes 30 minutes because steps that could run simultaneously are running one after another slows development and discourages frequent commits. The job dependency structure in this template is designed to maximise parallelism. The third mistake is not having a rollback plan. Every deployment should have a defined rollback procedure that can be executed in under five minutes. For Vercel and Railway, rollback to the previous deployment is one click. For custom infrastructure, document the exact rollback steps and test them before they are needed in a crisis. The fourth mistake is using production secrets in the CI/CD pipeline in a way that could expose them in logs. All secrets should be stored in the CI/CD platform's secret management (GitHub Actions Secrets, not environment variables) and referenced by name. Log outputs from deployment steps should be reviewed to ensure secrets are not being printed.
Customisation Tips for Different Project Types
For products with a Python backend alongside a TypeScript frontend, add a separate Python CI job that runs mypy type checking, ruff linting, and pytest test suite. Run Python and TypeScript jobs in parallel to minimise total pipeline time. The LLM evaluation job can be written in Python if the evaluation logic is Python-native. For products that fine-tune or retrain models as part of the development cycle, add a model training pipeline separate from the deployment pipeline. Model training jobs are typically too slow and expensive to run on every commit. Trigger training jobs manually or on a weekly schedule, and gate model updates on evaluation results before promoting to production. For enterprise products with compliance requirements (fintech under FCA oversight, healthtech under NHS Digital requirements), add a compliance check stage. This can include: dependency vulnerability scanning (npm audit, Snyk), SAST (static application security testing) with tools like Semgrep, secrets scanning to ensure credentials are not committed to the repository, and licence compliance checking for open-source dependencies. For products deployed to AWS or GCP rather than managed platforms like Vercel, the deployment stage needs to include infrastructure-as-code steps (Terraform plan and apply, or AWS CDK deploy). Add a Terraform plan step to the CI pipeline that runs on pull requests so reviewers can see infrastructure changes before they are merged.