Why Are Manual Deployments Technical Debt?

Manual deployments are technical debt because every hand-executed release is a bet that the person running it remembers every step, executes them in the right order, and avoids human error under time pressure.

Every team I’ve joined has had some version of the same problem. Deployments involve a senior engineer running a script from their laptop. There’s a wiki page somewhere describing the steps, but it’s outdated. Two people on the team know how to actually push to production, and when one of them goes on vacation, releases stall.

This is not a process. It’s a liability. Manual deployments are technical debt that compounds quietly until it causes an outage. At JPMorgan, I saw firsthand what happens when a platform serving millions of daily transactions relies on manual release steps. The risk profile is unacceptable. A single missed configuration change or skipped validation step can cascade into a production incident that affects customers across Visa, Amex, and Mastercard networks simultaneously.

The business case for automation is straightforward: every manual deployment is a roll of the dice. You are betting that the person running the release remembers every step, executes them in the correct order, and does not introduce human error under time pressure. That bet fails often enough to justify the investment in proper CI/CD infrastructure.

What Does a Production Pipeline Look Like?

A production-grade CI/CD pipeline consists of four core phases, specifically build, test, scan, and deploy, each acting as a gate that blocks promotion if it fails.

A production-grade CI/CD pipeline is more than a build script that runs on merge. It is a sequence of discrete stages, each with a clear purpose and failure behavior. When I established the fully automated pipeline at JPMorgan for the Chase Offers platform, we structured it around four core phases: build, test, scan, and deploy. Each phase acts as a gate. If any gate fails, the pipeline stops and nothing reaches production.

Build

The build stage compiles the application, resolves dependencies, and produces a deployable artifact. For containerized services, this means generating a Docker image with a deterministic tag tied to the commit SHA. Reproducibility matters here. You need to know exactly which code produced which artifact, and you need the ability to rebuild any previous version from source. At Notary Everyday, we containerized the entire platform during the infrastructure rebuild, ensuring every service produced versioned images that could be traced back to their originating commit.

Test

Automated tests run against the freshly built artifact. This includes unit tests, integration tests, and any contract tests that verify API boundaries. The critical detail is coverage thresholds. A pipeline that runs tests but accepts 30% coverage provides a false sense of security. At JPMorgan, we enforced coverage minimums and treated test failures as hard blockers. No exceptions, no “we’ll fix it later” overrides. The pipeline either passes or the code does not ship.

Scan

Security and dependency scanning catch vulnerabilities before they reach production. This is where tools like Snyk, Trivy, or SonarQube analyze the artifact for known CVEs, license violations, and code quality issues. I will cover this in more depth in the security section below, but the key point is that scanning must be a blocking stage. Advisory-only scans that produce reports nobody reads are theater, not security.

Deploy

The deploy stage pushes the validated artifact to the target environment. In a mature pipeline, this happens without human intervention. The artifact that passed build, test, and scan stages is the same artifact that reaches production. No manual approvals between staging and prod, no last-minute config changes, no “let me just tweak this one thing.” The pipeline is the authority.

What Quality Gates Prevent Bad Deploys?

Quality gates are checkpoints at every pipeline stage that enforce standards and answer one question: is this artifact safe to promote to the next environment?

A pipeline without quality gates is just a fast way to ship broken code. Gates are the checkpoints that enforce standards at every stage.

At JPMorgan, we implemented layered quality gates across the Chase Offers pipeline. Code could not merge to the main branch without passing linting, type checking, and unit tests. It could not progress to staging without passing integration tests against real downstream services. And it could not reach production without passing security scans and a final smoke test suite that validated core transaction flows.

The gates I recommend for any production pipeline include:

Linting and formatting - Enforced code style eliminates noisy diffs and catches syntax errors early
Type checking - For TypeScript or statically-typed languages, the compiler is your first line of defense
Unit test pass rate - 100% of tests must pass, with a coverage floor that the team agrees on
Integration test pass rate - Tests that verify behavior across service boundaries
Security scan - Blocking on critical and high-severity vulnerabilities
Image size limits - Prevents accidentally bundling development dependencies into production containers
Smoke tests - Post-deploy verification that core user paths are functional

When building Pinpoint’s CI/CD plugin, I designed the system to integrate these gates across GitLab CI, GitHub Actions, and Jenkins. The plugin triggers expert QA testers on every code push, adding a human verification layer on top of automated gates. The philosophy is the same regardless of the CI platform: make the pipeline the single authority on what ships, and make every gate a hard blocker.

Which Zero-Downtime Deployment Strategy Should You Use?

The right strategy depends on your constraints: rolling deployments for most services, blue-green for critical services requiring instant rollback, and canary releases for high-risk changes that need controlled observation.

Shipping code to production should never require a maintenance window. Users should not experience service interruptions because your team is releasing a new version. This is not aspirational; it is a baseline expectation for any service with production traffic.

There are three deployment strategies I have used in production environments, each suited to different constraints:

Rolling Deployments

New instances replace old instances incrementally. At any given moment during the deployment, both the old and new versions are serving traffic. This is the default strategy for most container orchestrators like ECS and Kubernetes. At JPMorgan, we used rolling deployments for the Chase Offers API, configuring minimum healthy targets to ensure at least 75% of capacity remained available throughout the rollout. The load balancer drains connections from old instances before terminating them, so in-flight requests complete without errors.

Blue-Green Deployments

You maintain two identical environments. The “blue” environment runs the current production version. The “green” environment receives the new version. Once the green environment passes all health checks, traffic switches from blue to green in a single cut. If anything goes wrong, you switch back instantly. The tradeoff is cost: you are paying for two full environments. But for critical services where rollback speed matters more than infrastructure spend, this approach is worth the premium.

Canary Releases

A small percentage of traffic routes to the new version while the majority continues hitting the stable release. You monitor error rates, latency, and business metrics on the canary population. If the new version behaves well, you gradually increase its traffic share until it serves 100%. If it misbehaves, you pull the canary and route all traffic back to the stable version. This strategy provides the highest confidence for risky changes because you observe real production behavior on a controlled subset of traffic before full rollout.

At Notary Everyday, we combined rolling deployments with automated health checks during the infrastructure rebuild. The pipeline validates that every new task definition registers healthy targets before proceeding, and it automatically rolls back if health checks fail within the stabilization window. No human needs to watch the deploy. The pipeline handles promotion and rollback decisions on its own.

How Should Security Validation Fit Into the Pipeline?

Security validation must be a blocking gate woven into every pipeline stage, with the same authority as test failures, rather than an advisory scan bolted onto the end.

Security cannot be an afterthought bolted onto the end of the pipeline. It must be woven into every stage. When I established the CI/CD pipeline at JPMorgan, security validation was not optional or advisory. It was a blocking gate with the same authority as test failures.

A comprehensive security stage includes several layers:

Dependency scanning - Analyzing third-party libraries for known CVEs. Your application might be secure, but if it pulls in a transitive dependency with a critical vulnerability, you inherit that risk. Tools like Snyk and Dependabot automate this, but they need to be blocking, not advisory.
Container image scanning - Examining the base image and all installed packages for vulnerabilities. A common mistake is using a base image that has not been updated in months, shipping dozens of known vulnerabilities into production without realizing it.
Static analysis - Scanning source code for patterns that indicate security issues: hardcoded credentials, SQL injection vectors, insecure deserialization, and other OWASP Top 10 concerns.
Secrets detection - Catching API keys, tokens, and passwords that accidentally end up in source code or configuration files. This should run on every commit, not just at build time.
License compliance - Verifying that all dependencies use licenses compatible with your organization’s policies. This matters more in enterprise environments where legal compliance is non-negotiable.

Building Pinpoint’s CI/CD plugin gave me a cross-platform perspective on security integration. The plugin needed to work identically across GitLab CI, GitHub Actions, and Jenkins, each with different syntax and capabilities for handling secrets, caching scan results, and reporting findings. The principle remained constant regardless of platform: security scans produce pass/fail outcomes, and failures block the pipeline. Reports are useful for remediation, but the gate decision is binary.

In regulated environments like financial services, the security stage also produces audit artifacts. Every scan result, every dependency list, every vulnerability assessment gets stored as a pipeline artifact with a retention policy. When auditors ask “how do you know this release was secure?” you point them at the pipeline artifacts, not at a spreadsheet someone filled out manually.

What Practical Lessons Hold True Across Different Environments?

Five patterns consistently hold true after building and maintaining pipelines across JPMorgan, Notary Everyday, and Pinpoint: start simple, treat pipeline code like application code, make failures actionable, measure pipeline health, and never allow security bypasses.

Start with the simplest pipeline that blocks on failure. A three-stage pipeline (build, test, deploy) that actually stops bad code is better than a twelve-stage pipeline where half the stages are advisory. You can add sophistication later. Blocking behavior is the foundation.
Treat pipeline code like application code. Your CI/CD configuration lives in version control, gets reviewed in pull requests, and follows the same quality standards as the application it deploys. Pipeline drift, where the configuration quietly falls out of sync with actual requirements, is a real and common problem.
Make pipeline failures actionable. A red build with a cryptic error message is worse than no pipeline at all, because it trains the team to ignore failures. Every failure should clearly indicate what broke, where, and how to fix it. Invest in error messages the way you invest in user-facing copy.
Measure pipeline health. Track build duration, failure rate, and time-to-recovery. A pipeline that takes 45 minutes to run will be bypassed. A pipeline that flakes on 10% of runs will be ignored. Performance and reliability of the pipeline itself are first-order concerns.
Never allow manual overrides for security gates. The moment someone can skip the security scan “just this once,” the gate loses all value. At JPMorgan, the integrated security validation had no bypass mechanism. If the scan failed, the code did not ship. Full stop.

The goal is not a perfect pipeline on day one. The goal is a pipeline that enforces a minimum quality bar and grows more capable over time. Start with hard blockers on build failures and test failures. Add security scanning once the basics are solid. Layer in performance testing and canary analysis as the team matures. Each addition compounds the reliability of your release process.

Ready to eliminate manual deployments? Schedule a conversation to discuss building CI/CD pipelines for your team.

Building CI/CD Pipelines from Zero to Production