Automating Docker Host Setup with Ansible for Dockhand Management

I use Dockhand to manage Docker containers across multiple physical servers. When I added a new host, instead of configuring everything manually, I created Ansible roles to automate the entire setup. This post covers the automation—not the Caddy CA/ACME setup, which I’ve already documented separately.

What Dockhand Does

Dockhand is a central management tool for Docker containers across multiple hosts. It provides:

Container Management: Start, stop, update, and monitor containers remotely
Image Updates: Automatic detection and updating of container images across all hosts
Unified Interface: Manage your entire Docker infrastructure from one place
Visual Docker Compose Editor: Edit and deploy compose files through a web UI
Deploy Stacks from Git: Deploy Docker Compose stacks directly from git repositories
Auto-Deploy via Webhooks: Automatically redeploy stacks when you push to git
Monitoring & Logs: View real-time logs and resource usage per container
Image Registry Support: Connect to multiple registries to track image updates

To use Dockhand, each Docker host needs a secure remote API endpoint. That’s what this automation handles.

Architecture

The setup automates three key components:

Docker Daemon Configuration: Each host runs Docker with registry mirroring (Harbor for DockerHub). Hawser Agents: Lightweight remote Docker API wrapper. The Caddy sidecar proxies this securely. Caddy Sidecar Proxy: One Caddy container per host that retrieves TLS certificates from a central CA and proxies the Docker API to the network. Dockhand connects securely to this endpoint.

Ansible Automation

I created three reusable roles to automate everything.

Role 1: Docker Daemon Configuration

Configures the Docker daemon with registry mirroring and exposes the API locally. This way Docker automatically pulls from my internal Harbor’s mirror via registry-mirrors in daemon.json and Dockhand detects updates correctly.

Docker daemon configuration template (templates/daemon.json.j2):

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  },
  "registry-mirrors": ["https://registry-proxy.local/v2/dockerhub-proxy"],
  "hosts": ["fd://", "unix:///var/run/docker.sock"]
}

The registry-mirrors entry points to Harbor’s DockerHub proxy, reducing bandwidth and improving pull reliability.

The trick: systemd can override the ExecStart command, conflicting with daemon.json flags. Fix this with a systemd override file.

Systemd override (/etc/systemd/system/docker.service.d/override.conf):

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --config-file /etc/docker/daemon.json

This ensures the daemon only reads configuration from daemon.json.

Role 2: Caddy Sidecar Proxy

Deploys Caddy on each host to handle TLS termination and expose services securely.

Caddyfile template (templates/Caddyfile.j2):

{
    default_bind {{ caddy_default_bind }}
    log {
        output stdout
        format console
    }
}

# Internal TLS snippet with local ACME CA
(internal_tls) {
    tls {
        issuer acme {
            dir https://{{ caddy_acme_ca }}
            trusted_roots /etc/caddy/root.crt
        }
    }
}

# Host-specific service entries
{% if caddy_services is defined %}
{% for service in caddy_services %}
{{ service.domain }} {
    import internal_tls
    reverse_proxy localhost:{{ service.port }}
}
{% endfor %}
{% endif %}

Why This Configuration

The original Caddyfile used a global acme_ca directive, which caused unexpected behavior. Caddy has built-in “smart” logic that backfires on private TLDs:

When you set acme_ca globally for domains like .local, Caddy reasons: “This is a private TLD, so I’ll never get a valid certificate from a standard CA. The domain must need internal CA handling, so I’ll use my Local Authority instead.”

This fallback behavior created conflicts between the intended CA and Caddy’s auto-generated certificates.

The solution: Use an explicit tls block with issuer acme instead. This gives Caddy a direct command with no room for interpretation. The snippet pattern (internal_tls) makes the configuration reusable across all services without repetition.

Key improvements:

No global conflicts - Caddy doesn’t apply fallback logic to every domain
Explicit issuer - The issuer acme block is a direct instruction, not a suggestion
Certificates from your CA - All services get signed by your internal CA (homelab_ca - ECC Intermediate), not Caddy Local Authority
Per-host binding - default_bind lets each host use its own IP address

Reference: Caddy GitHub Issue #7147

Service Configuration & Host Variables

Each host in the [caddy_sidecar_hosts] inventory group needs a host_vars file with:

caddy_default_bind: The IP address that Caddy binds to (usually the host’s primary IP)
caddy_services: List of domains and ports to proxy
caddy_acme_ca: The ACME directory of your internal CA

Example for docker-host-01.local (inventories/host_vars/docker-host-01.local.yml):

caddy_default_bind: "192.168.1.81"
caddy_acme_ca: "ca.local/acme/directory"

caddy_services:
  - domain: "hawser.docker-host-01.local"
    port: 2376
  - domain: "monitoring.local"
    port: 9100

Important notes:

Each service in caddy_services automatically receives a TLS certificate from your internal CA
Services bind to the IP address in caddy_default_bind
The FQDN for Hawser is automatically derived: hawser.
You can add as many additional services as needed without modifying templates

Container Health Check & Retry Logic

When Caddy containers start or when configuration changes, the Ansible role implements a robust health check and retry system:

Container Readiness Check (after Docker Compose starts)
- Waits up to 30 seconds
- Verifies: Container exists AND is running
- Only triggered if the Compose file changed
caddy config formatter (Caddyfile check)
- 3 retry attempts with 2-second delay between retries
- Auto-format Caddyfile correctly
Caddy Container Restart (Apply Configuration)
- Triggered by changes in Caddyfile or docker-compose.yml.
- Ensures ACME certificate initialization, as a standard caddy reload might not always trigger a new challenge for new domains.
- Provides a clean state, which is more reliable for first-time certificate issuance.

The Ansible task structure for auto-format and container restart looks like:

- name: Validate Caddyfile syntax with caddy fmt
  command: "docker exec  caddy fmt --overwrite /etc/caddy/Caddyfile"
  when: caddyfile_template.changed
  register: caddy_fmt_result
  retries: 3
  delay: 2
  until: caddy_fmt_result.rc == 0

- name: Restart Caddy container
  community.docker.docker_container:
    name: ""
    restart: yes
  when: caddyfile_template.changed or compose_template.changed or compose_recreate.changed

Role 3: Hawser Remote Agent

Deploys the Hawser container that exposes the Docker API remotely using the Sidecar Proxy pattern.

Docker Compose template (templates/docker-compose.yml.j2):

services:
  hawser:
    image: ghcr.io/finsys/hawser:latest
    container_name: hawser
    restart: unless-stopped
    ports:
      - "127.0.0.1:2376:2376"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    env_file: .env
    environment:
      AGENT_NAME: "{{ inventory_hostname_short }}"

Hawser Configuration via .env

Instead of passing the Hawser token through environment variables in docker-compose.yml, store it in a local .env file on the target host:

TOKEN=your_secret_hawser_token

The Hawser token and other sensitive data should be encrypted with Ansible Vault:

ansible-vault encrypt_string 'YOUR_HAWSER_TOKEN_HERE' --name 'hawser_token'

This creates an encrypted string that you can paste into your host_vars file.

The role automatically sets:

hawser_fqdn: "hawser." - Used for TLS verification
AGENT_NAME: "" - Agent identification in logs

The Sidecar Proxy Pattern

Hawser deliberately listens only on localhost:2376, without TLS. Security is provided by the Caddy Sidecar:

┌────────────────────────────────────────────────────────┐
│  External Network (192.168.1.82:443)                   │
│  hawser.docker-host-02.local:443 (HTTPS)              │
│           ↓                                            │
│  ┌──────────────────────────────────────────────────┐  │
│  │  Caddy Sidecar (TLS-Proxy)                       │  │
│  │  • Domain: hawser.docker-host-02.local           │  │
│  │  • Cert: homelab_ca - ECC Intermediate          │  │
│  │  • Binds to: 192.168.1.82:443                    │  │
│  │  • Proxies → localhost:2376                      │  │
│  └──────────────────────────────────────────────────┘  │
│           ↓                                            │
│  ┌──────────────────────────────────────────────────┐  │
│  │  Hawser Agent (HTTP only)                        │  │
│  │  • Port: 127.0.0.1:2376                          │  │
│  │  • No TLS, not exposed directly                  │  │
│  │  • Token Authentication                         │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘

Why this architecture:

Hawser stays internal - Never exposed directly to the network
Centralized TLS - One CA, one certificate authority, one sidecar per host
Simpler security - Hawser only needs token auth and localhost binding; Caddy handles TLS
Better observability - TLS termination happens in one place (Caddy)
High availability - The sidecar can be restarted independently of Hawser
Clear responsibility - Hawser manages Docker API; Caddy manages network security

Certificate Issuer Verification

After deployment, the Ansible role automatically verifies that Hawser has a valid TLS certificate from your internal CA.

The verification command:

openssl s_client -connect hawser.docker-host-02.local:443 \
  -servername hawser.docker-host-02.local </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer

Expected output: Internal CA name

issuer=CN = homelab_ca - ECC Intermediate

Why this matters:

If Caddy falls back to its Local Authority, the certificate won’t be trusted by Dockhand (or your client systems). The verification happens 3 times with 10-second delays to account for CA certificate generation timing:

- name: Verify Hawser certificate issuer
  shell: |
    openssl s_client -connect :443 \
      -servername  </dev/null 2>/dev/null \
      | openssl x509 -noout -issuer
  register: cert_issuer
  retries: 3
  delay: 10
  until: "'homelab_ca' in cert_issuer.stdout"
  failed_when: "'homelab_ca' not in cert_issuer.stdout"

Important notes on SNI:

The -servername parameter uses SNI (Server Name Indication), which tells Caddy which certificate to return
Both -connect (address) and -servername (domain) must be correct for the right certificate to be selected
This is why the FQDN must match exactly between Caddy configuration and the verification command

Configuring Dockhand

Once the hosts are automated, configure Dockhand to connect:

Add each Docker host with the Caddy endpoint (e.g., https://hawser.docker-host-02.local:443)
Use the Hawser token for authentication
Add root CA certificate
Dockhand can now manage containers and monitor image updates remotely

Debugging & Important Commands

When troubleshooting the setup, these commands are invaluable:

Caddy Logs
docker logs -f caddy_sidecar_proxy
TLS Check (via FQDN)
openssl s_client -connect hawser.docker-host-02.local:443 -servername \ hawser.docker-host-02.local </dev/null 2>/dev/null | openssl x509 -noout -issuer
Format Caddyfile
docker exec caddy_sidecar_proxy caddy fmt --overwrite /etc/caddy/Caddyfile
Container Status
docker inspect caddy_sidecar_proxy --format ''