CI/CD Pipelines Intermediate

Continuous integration (CI) is the practice of merging every developer’s work into a shared mainline frequently — ideally many times a day — and verifying each merge with an automated build and test run. The goal is to keep the codebase in a known-good, always-integrated state so problems surface within minutes, not weeks.

CI is a discipline, not just a tool: it requires a shared repository, a self-testing build, and a team commitment to fix a broken build immediately.

Before CI, teams worked in isolation for weeks and merged everything at the end — a painful event called integration hell or merge hell. The longer branches live apart, the more they diverge, and the harder conflicts become to resolve.

The cost of fixing a defect rises sharply the later it is found. A bug caught seconds after a commit is cheap; the same bug found in production may be orders of magnitude more expensive. Frequent integration keeps each change small, so conflicts are tiny and feedback is fast.

A CI pipeline is an automated sequence triggered by a code change. A classic linear shape is: checkout the source → build (compile/transpile and resolve dependencies) → test (run the automated suites) → package (produce a deployable artifact such as a jar, container image or zip).

Each step runs only if the previous one succeeded. If any step fails, the pipeline stops and reports a red build, giving the team immediate, actionable feedback.

CI relies on layered automated tests. The test pyramid recommends many fast unit tests at the base, fewer integration tests in the middle, and a small number of slow end-to-end (e2e) tests at the top.

Unit tests check a single function or class in isolation. Integration tests verify that components work together (e.g. code plus a real database). End-to-end tests exercise the whole system through its public interface, as a user would. Inverting the pyramid — many slow e2e tests, few unit tests — produces brittle, slow suites.

A build artifact is the immutable, deployable output of the build — for example a compiled binary, a JAR/WAR, a Python wheel, or a container image. A key CI principle is build once, deploy many: you build the artifact a single time and promote that exact same artifact through every environment, so what you tested is what you ship.

Artifacts are stored in an artifact repository (such as a container registry, Nexus, Artifactory or a package registry), versioned and addressable so any pipeline run can fetch a known build.

Both extend CI, but they differ in one decisive step. Continuous delivery keeps every change releasable: the pipeline automatically builds, tests and prepares each change for production, but the final release to production requires a manual approval (a human clicks a button).

Continuous deployment removes that manual gate: every change that passes the automated pipeline is automatically released to production, with no human intervention. Continuous deployment requires continuous delivery, plus enough confidence in your tests and safeguards to ship automatically.

Pipeline-as-code means defining your build and release process in a version-controlled file that lives alongside the application code — typically a YAML file (e.g. .gitlab-ci.yml, a GitHub Actions workflow, or a Jenkinsfile).

Because the pipeline is code, it is reviewed, versioned, diffed and rolled back like any other change. The pipeline evolves with the application in the same commit, and anyone can read exactly how the software is built and shipped.

Pipelines are structured hierarchically. A stage is a phase such as build, test or deploy; stages usually run in order. A job is a unit of work within a stage that runs on a single runner; jobs in the same stage can run in parallel. A step is an individual command or action inside a job, executed in sequence.

This structure lets you fan out work — for example, run lint, unit and integration jobs in parallel within one test stage to finish faster.

The pipeline orchestrator decides what to run; the actual commands execute on a runner (GitLab) or agent (Jenkins, Azure Pipelines) — a machine, container or virtual machine that picks up jobs and runs them.

Runners can be hosted/shared (provided and managed by the CI platform) or self-hosted (you run them on your own infrastructure, useful for special hardware, private network access or compliance). Each job typically runs in a clean, isolated environment so builds are reproducible.

Software is usually shipped through a series of environments: dev (fast feedback), staging (production-like, for final validation), and prod (real users). Each environment mirrors production more closely as you move up.

Promotion is moving the same tested artifact from one environment to the next after it passes the required checks. You never rebuild between environments — you promote the identical artifact, so a passing staging test gives real confidence for production.

Pipelines need credentials — registry passwords, cloud keys, signing keys. These secrets must never be committed to the repository or printed in logs. Instead, store them in the CI platform’s encrypted secret store (or an external vault) and inject them as environment variables at runtime.

Good practice: scope secrets to the environments that need them, mask them in logs, rotate them regularly, and prefer short-lived tokens (e.g. OIDC federation) over long-lived static keys.

A recreate deployment stops all instances of the old version, then starts the new version. It is simple but causes downtime during the switch.

A rolling deployment replaces instances gradually — a few at a time — so the service stays available throughout. At any moment both old and new versions may be serving traffic, so the new version must be compatible with the old. Rolling updates avoid downtime but make rollback slower than an instant switch.

A blue-green deployment keeps two identical production environments: one live (blue) and one idle (green). You deploy the new version to the idle environment, test it, then switch all traffic over at once. Rollback is instant — just switch traffic back to the previous environment.

A canary deployment releases the new version to a small subset of users or servers first, watches metrics and error rates, and only then gradually shifts the rest of the traffic. It limits the blast radius of a bad release.

A feature flag (feature toggle) is a runtime switch that turns a piece of functionality on or off without deploying new code. This decouples deployment (shipping code) from release (exposing the feature to users).

Flags let you merge unfinished work safely behind a disabled toggle (supporting trunk-based development), roll a feature out to a percentage of users, run A/B experiments, and instantly kill a misbehaving feature without a rollback deploy.

When a deployment goes wrong, you need to recover fast. Automated rollback means the pipeline (or platform) detects a failed release — via health checks, error-rate spikes or failing smoke tests — and automatically reverts to the last known-good version without a human in the loop.

Because CI follows build once, deploy many, the previous artifact is still available, so reverting is fast and reliable. A related practice is roll forward: ship a quick fix instead of reverting. Either way, the key is fast, automated detection and recovery.

A quality gate is an automated pass/fail check that a change must clear before it advances. Common gates include a minimum test coverage threshold, clean linting/static analysis, and security scanning.

Embedding security into the pipeline is DevSecOps — “shift left” on security. Typical scans include SAST (static analysis of source), dependency/SCA scanning for vulnerable libraries, secret scanning, and container image scanning. A failing gate stops the pipeline, so problems are blocked before they reach production.

Fast feedback is the whole point of CI, so pipeline speed matters. Caching reuses expensive results between runs — for example downloaded dependencies (node_modules, the Maven .m2 folder) or Docker layers — so they are not rebuilt every time.

Other speed techniques: run independent jobs in parallel, only rebuild what changed (incremental builds), fail fast by running quick checks first, and split slow test suites across runners. The aim is a feedback loop short enough that developers wait for the build before moving on.

Trunk-based development is a branching model where developers integrate into a single shared branch (the trunk or main) at least daily, using very short-lived branches or committing directly. It is the natural partner of CI: small, frequent merges keep the trunk continuously integrated and always releasable.

It contrasts with long-lived feature branches, which drift apart and cause big merges. To keep the trunk green while integrating incomplete work, teams hide it behind feature flags and rely on a strong automated test suite to protect every commit.