Deploying a GitLab CI pipeline for infrastructure code testing
Infrastructure as code has reshaped how Australian IT teams manage environments, from small home labs in suburban garages to enterprise estates running out of data centres in Sydney and Melbourne. Yet plenty of engineers still treat infrastructure code differently from application code, skipping the kind of automated testing that catches broken playbooks before they hit production. A well-designed GitLab CI pipeline brings consistency, repeatability, and confidence to the deployment process, especially when managing VMware environments at scale.
This walkthrough covers the practical steps for building a pipeline that validates Ansible roles, Terraform plans, and PowerCLI scripts before they touch live infrastructure. Whether you are based in Brisbane working on a hybrid cloud project or refining automation in Perth, the same principles apply. The goal is to create feedback loops that are fast, visible, and integrated with the tools your team already uses every day.
Why infrastructure code needs automated testing
When a Terraform module silently drops a security group rule or an Ansible task fails halfway through a configuration, the consequences ripple through the entire stack. Manual review works for small projects, but it does not scale once repositories grow past a few dozen files. Automated testing in GitLab CI turns subjective review into objective checks that run on every merge request.
Australian teams operating under the Privacy Act and APRA CPS 234 obligations have additional reasons to test rigorously. A pipeline that scans for misconfigurations before deployment helps demonstrate due diligence and keeps audit trails tidy. It also reduces the time spent fixing issues after the fact, which matters when engineering hours cost north of AUD 150 in most capital cities.
Core components of a GitLab CI pipeline
A functional infrastructure testing pipeline rests on three pillars: runners, stages, and artifacts. Runners are the machines that execute jobs, and they can be shared across the GitLab instance or dedicated to a single project. Stages group related jobs together, such as linting, planning, and applying changes, while artifacts pass files between stages for inspection.
For infrastructure testing, shared SaaS runners work for lightweight jobs like YAML validation, but heavier workloads such as Terraform plan against real VMware vCentres require self-hosted runners. Many Australian organisations build a VMware home lab on retired enterprise hardware to host these runners, gaining isolation without recurring cloud costs.
Structuring the .gitlab-ci.yml file
The .gitlab-ci.yml file sits at the root of the repository and defines every stage, job, and variable the pipeline needs. A clean structure begins with a default section that sets base images and caching rules, followed by stage declarations and individual job definitions. Variables such as Ansible verbosity levels or Terraform input files are best placed in a central block.
A typical setup includes stages named validate, plan, test, and deploy. The validate stage runs yamllint, ansible-lint, and terraform fmt to catch syntax errors early. The plan stage generates execution plans without applying them, while the test stage spins up disposable environments using Docker or local virtual machines to verify behaviour.
Comparing common testing tools
Different tools serve different purposes in an infrastructure pipeline, and choosing the right combination depends on what you are validating. The overview below summarises the most common options used alongside GitLab CI.
| Tool | Primary use | Language | Integration method |
|---|---|---|---|
| ansible-lint | Ansible playbook style and correctness | Python | CLI in CI job |
| terraform validate | Terraform syntax and internal checks | Go | CLI in CI job |
| Checkov | IaC security and compliance scanning | Python | Container image |
| kitchen-terraform | End-to-end Terraform testing with Test Kitchen | Ruby | Docker driver |
| molecule | Ansible role testing across multiple scenarios | Python | Docker or podman |
Checkov has become popular with Australian teams working under the Essential Eight framework because it flags resources that violate hardening baselines. Kitchen-terraform and molecule both fill the gap between static checks and full integration tests, allowing roles and modules to be exercised against real or simulated infrastructure.
Running tests in real and simulated environments
Simulated environments are perfect for early feedback. Docker containers, Vagrant boxes, and lightweight Linux VMs can stand in for production targets during unit and integration tests. They start quickly, cost nothing to run, and reset between jobs. For VMware-specific code, simulating ESXi behaviour with nested virtualisation on a home lab runner offers a reasonable middle ground.
Real environment tests carry more weight but demand more resources. A pipeline that connects to a vCentre in an ap-southeast-2 AWS region or an on-premises datacentre in Sydney will catch issues that simulation misses, such as network latency, storage contention, and permission boundaries. Limiting these jobs to nightly schedules or merge requests labelled ready-for-integration keeps costs predictable.
Managing secrets and self-hosted runners
Secrets such as vCentre credentials, API tokens, and SSH keys must never appear in plaintext within the repository. GitLab CI offers protected and masked variables, and combining them with a Vault server provides rotation and audit logging. Self-hosted runners add another layer of concern because they often sit inside trusted network zones.
Practical steps for tightening runner security include:
- Rotating registration tokens every 90 days
- Using runner tags to scope job execution to specific projects
- Storing all credentials in GitLab masked variables or an external Vault
- Enabling structured logging for clearer audit trails
Common mistakes include hardcoding passwords in .gitlab-ci.yml, granting runners overly broad IAM permissions, and leaving runner registration tokens in shell history. Rotating tokens quarterly, restricting runner tags to specific projects, and enabling structured logging all reduce the blast radius if a runner is compromised.
Troubleshooting pipeline failures in practice
When a job fails, the first stop is the job trace output, which shows the exact command, exit code, and any captured logs. Filtering for terms like ERROR, failed, or specific module names often surfaces the root cause within seconds. For intermittent failures, enabling job retries with a delay can smooth over transient network issues, especially when runners communicate with remote vCentres over consumer NBN links.
Patterns worth tracking across your pipeline history:
- Terraform state lock contention during parallel jobs
- Ansible gather_facts timeouts on slow wide-area networks
- PowerShell execution policy violations on Windows runners
- Outdated Docker images breaking tool compatibility
Caching dependencies, pinning tool versions, and using explicit image tags rather than latest prevent most of these issues. Local communities like the Melbourne DevOps meetup and the Brisbane Atlassian user group also share war stories that can shortcut your own debugging sessions.
For broader guidance on VMware automation and home lab design, the Virtxpert site collects guides, scripts, and tooling that complement the pipeline approach outlined above. Start small with a single stage, expand as confidence grows, and let the pipeline catch the mistakes that manual review always misses. Drop the first job into a merge request today and watch how quickly the rest of the workflow falls into place.