How to use Git submodules for modular Ansible collections

Ansible collections often grow faster than the playbooks that consume them. Keeping VMware modules, Linux roles, inventory plugins and internal utilities in one repository can make testing convenient at first, but it eventually creates tangled release cycles. Git submodules offer a practical boundary: each collection remains an independent Git project while an automation repository records the exact commit it needs.

This approach suits infrastructure teams managing vSphere, network appliances, cloud services and home labs from a common automation codebase. It also works well when an organisation wants reusable collections without immediately publishing them to Ansible Galaxy or maintaining a private Galaxy server.

Approach	Best fit	Main strength	Main risk
Git submodule	Controlled internal collections	Reproducible commit pinning	Developers must update submodules deliberately
Git subtree	Shared code with simpler cloning	No separate checkout workflow	History and updates can become confusing
Galaxy or private Hub	Published, versioned collections	Familiar dependency management	Requires packaging and registry processes
Monorepo	Closely coupled automation	Simple local development	Weak separation between collection releases

Define the repository boundaries

Create one repository for the consuming automation and separate repositories for collections that have their own lifecycle. For example, a platform repository might contain playbooks and inventories, while virtxpert.vmware contains VMware-specific modules, roles and plugins.

A collection repository should use the standard source layout:

ansible_collections/
└── virtxpert/
    └── vmware/
        ├── galaxy.yml
        ├── plugins/
        ├── roles/
        ├── tests/
        └── README.md

The galaxy.yml file defines the namespace, collection name and version. Keep that metadata in the collection repository, rather than duplicating it in the parent project. This makes the collection buildable and testable on its own.

Add the collection as a submodule

From the root of the automation repository, add the collection at the path Ansible expects when using a local collection source:

git submodule add \
  git@github.com:example/virtxpert.vmware.git \
  collections/ansible_collections/virtxpert/vmware

git commit -m "Add VMware collection submodule"

Git writes the repository URL and path to .gitmodules, while the parent repository records a specific commit from the collection. That commit reference is the important part: a fresh checkout receives the same collection revision used by the rest of the team.

Use relative URLs when the parent and collection repositories live beside one another on an internal Git server. This can simplify migration between GitHub Enterprise, GitLab and other platforms. SSH URLs are convenient for engineers, but CI runners may need deploy keys or HTTPS credentials.

For background on the person behind Virtxpert and its infrastructure focus, see Jonathan’s background.

Clone and update the working copy

A normal clone does not automatically populate submodules. New contributors should use:

git clone --recurse-submodules git@github.com:example/automation.git

If the repository has already been cloned, initialise the nested checkout with:

git submodule update --init --recursive

To update a collection, enter its directory, fetch the desired branch or tag, and return to the parent repository:

cd collections/ansible_collections/virtxpert/vmware
git fetch --tags origin
git checkout v1.4.0
cd ../../../../..
git add collections/ansible_collections/virtxpert/vmware
git commit -m "Update VMware collection to v1.4.0"

The parent project does not store the collection’s files in its own history. It stores the selected commit, so an update is an explicit reviewable change. Avoid casually using git submodule update --remote in production automation unless the resulting commit is inspected and committed deliberately.

Configure Ansible collection discovery

Ansible must be able to discover collections from the submodule path. A project-local ansible.cfg keeps the behaviour predictable:

[defaults]
collections_paths = ./collections:~/.ansible/collections

With the directory structure shown above, playbooks can use fully qualified collection names:

- name: Gather vSphere details
  hosts: localhost
  gather_facts: false
  collections:
    - virtxpert.vmware

  tasks:
    - name: Create an example VM
      virtxpert.vmware.vm:
        name: lab-router
        state: present

Fully qualified names reduce collisions with similarly named modules and make code searches easier. In larger Australian MSP environments, this separation is useful when a Melbourne delivery team and a Brisbane operations team maintain different playbook repositories but consume the same approved collection commit.

Manage dependencies separately

A collection can declare Python and Ansible requirements in galaxy.yml, while the automation repository can maintain a requirements.yml for external collections. Do not assume that adding a submodule installs its dependencies.

For example:

collections:
  - name: community.vmware
    version: ">=4.0.0,<5.0.0"

Install external dependencies in a controlled virtual environment, then run the playbooks with the submodule present. Pin versions where repeatability matters, especially for VMware APIs, network modules and cloud SDKs. Australian organisations working against Essential Eight controls may also need an auditable record of the exact collection, Python package and execution-environment versions.

Build reliable CI checks

CI should initialise submodules before linting or testing:

git submodule sync --recursive
git submodule update --init --recursive
ansible-galaxy collection list
ansible-lint
ansible-test sanity --docker default

Run collection-specific tests inside the submodule and integration tests from the parent repository. A pull request that changes the submodule pointer should show the old and new commit, collection version and relevant test results.

Use an execution environment or container image to pin Ansible Core, Python and system packages. This avoids the common problem where a developer in Sydney has a newer local SDK than a CI runner hosted in a different environment. It also makes scheduled tests easier to run against vSphere, REST endpoints or mock services.

Make local development safe

Submodules work best when the collection has a clear contribution process. Developers can temporarily check out a branch inside the submodule, test changes from the parent repository, then push the collection branch and update the parent pointer. Do not commit a submodule pointer that refers only to an unpushed local commit.

For a low-power home lab, this model keeps the control repository lightweight while collection development remains isolated. A small NUC or mini PC running Git, Ansible and a virtualised test target can provide a useful sandbox; the low-power lab guide provides relevant planning ideas.

Use protected branches and tags for collection releases. A production repository should normally point to a signed or reviewed tag commit, while a development branch can track an explicitly approved commit. Document the update command in CONTRIBUTING.md so engineers in Adelaide, Perth or Canberra follow the same process.

Troubleshoot common failures

If Ansible reports that a collection cannot be found, check the current directory, collections_paths and the nested directory structure. The path must lead to ansible_collections/<namespace>/<name>, not merely to a repository containing galaxy.yml.

If the submodule appears empty, run git submodule update --init --recursive. If CI fails during checkout, confirm that the runner can access the submodule repository and that its token or deploy key has permission. Shallow clones can also cause problems when scripts expect tags or parent commits, so use a full enough fetch depth for version checks.

Finally, review submodule changes as dependency changes, not ordinary file edits. Record why a commit was selected, run the relevant playbook tests, and keep rollback simple by reverting the parent repository’s pointer. With those habits, modular Ansible collections remain reusable without sacrificing reproducibility or operational control.

Adopt a small proof of concept first: place one internal collection behind a submodule, add CI initialisation, pin its commit, and document the update workflow before migrating the rest of the automation estate.