Docker Compose Profiles Upgrade and Migration: A Practical Implementation Guide

Intro

When you need to upgrade Docker Compose and migrate an existing project that uses Compose profiles, the work is not just about swapping binaries or editing YAML. It is a controlled change to how services are selected, started, and connected. A profile is a named set of services in a docker-compose.yml that is only activated when you explicitly target it via the --profile flag or the COMPOSE_PROFILES environment variable. Upgrades and migrations become risky when you assume the new Compose version parses profiles the same way, when data lives on the wrong storage layer, or when no one has defined what a successful cutover looks like.

This guide is written for developers, DevOps engineers, and technical startup teams who need to upgrade Docker Compose Profiles without breaking running services. It covers version inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and a pre-cutover operations checklist. Every procedure includes concrete commands, expected output, failure signals, and a decision point so you can observe before changing, limit the blast radius, and recover quickly if the result is not what you expected.

The operational goal is simple: separate observation from intervention, capture current state and timestamps first, protect credentials and private material, change one scoped item only when its blast radius and recovery path are understood, and verify the outcome. A simple restart test can reveal data persistence problems before they become production incidents.

Version and Environment Inventory

Before touching any configuration file, record exactly what you have. This inventory protects you from hidden incompatibilities and gives you a rollback target. Run the following read-only commands as a user with just enough Docker permissions; avoid sudo until you know you need it.

Start with the Docker and Compose versions:

docker --version
# Example output: Docker version 24.0.7, build afdd53b

docker compose version
# Example output: Docker Compose version v2.23.0-desktop.1

docker compose config --profiles
# Lists all profiles defined in the current working directory's compose file(s).

If docker compose config --profiles fails with a YAML parse error, fix the YAML syntax before changing anything else. Confirm the current project actually has profiles:

docker compose config --profiles
# Example output if profiles exist:
#   - dev
#   - debug

If the output is empty, your project is not using profiles yet; migration will introduce them. If the command is not recognized, your Compose version is too old (profiles were introduced in Compose v2.2.0, but newer patch versions fixed profile-resolution bugs). Upgrade the Compose plugin first. On Linux with Docker Desktop, you can update the plugin via the Desktop UI, or on a standalone Docker Engine install you can download the latest plugin binary to ~/.docker/cli-plugins/.

Next, see what is actually running in the project context:

docker compose ps
# Example output:
# NAME                COMMAND                  SERVICE             STATUS              PORTS
# web-1               "python app.py"          web                 running             0.0.0.0:8000->8000/tcp

If you see containers you did not expect, stop and investigate. A profile might have been activated previously via an environment variable, or a developer might have started the service manually.

Check the current profile activation source. In your shell or in a .env file next to the compose file:

printenv | grep COMPOSE_PROFILES
# Example output if set: COMPOSE_PROFILES=dev

If COMPOSE_PROFILES is set globally in your shell profile, it might activate profiles for every compose project in that terminal, causing confusion. Prefer project-local .env files or explicit --profile flags for reproducibility.

Finally, inspect data storage. List named volumes and bind mounts used by the project:

docker compose ps -q | xargs -I{} sh -c 'docker inspect -f "{{.Name}} {{range .Mounts}}{{.Type}}:{{.Source}}->{{.Destination}} {{end}}" {}'
# Example output for a named volume and a bind mount:
# web-1 volume:app_data->/var/lib/app
# web-1 bind:/home/user/app/html->/usr/share/nginx/html

A named volume like app_data is managed by Docker and is easier to preserve across container recreation. A bind mount maps a host path and is useful for local development, but it can break on another host or container context if the path does not exist or permissions differ. Record which services use bind mounts; if any bind-mounted directory contains version-specific files or lock files, make sure it is writable by the container user after the upgrade.

Persistence test before any change: Create or identify a test file inside each data directory, recreate the container, and check whether the file survives.

# For a running container named app, write a marker
docker compose exec app sh -c 'echo canary-$(date +%s) > /var/lib/app/canary.txt'
docker compose stop app
docker compose up -d app
docker compose exec app cat /var/lib/app/canary.txt
# If the file content is returned, the data is on a volume or bind mount.
# If the file is missing, the service wrote to the container's writable layer, which is lost on recreation.

If data disappears after recreation, update the service definition to use a named volume or bind mount before attempting any profile migration. Do not upgrade profiles on a service that loses data on a simple restart.

Safe Configuration Path

After inventory, you are ready to modify profile definitions. Work on a copy of the compose file, not the live one. Use git diff or a file backup to make rollback trivial. Follow this sequence:

1. Freeze the current compose file as docker-compose.yml.backup in the same directory.
2. Make one change at a time: add a profile, change a profile assignment, or upgrade the Compose file version header.
3. Validate with docker compose config before applying.
4. Apply in a test environment first if possible.

Example of migrating a service into a profile

Suppose your docker-compose.yml has services web, db, and adminer, and you want to move adminer behind a debug profile while keeping web and db always active:

Original snippet:

services:
  web:
    image: nginx:alpine
    ports:
      - "8000:80"
  db:
    image: postgres:16-alpine
    volumes:
      - db_data:/var/lib/postgresql/data
  adminer:
    image: adminer:4.8.1
    ports:
      - "8080:8080"

Modified snippet:

services:
  web:
    image: nginx:alpine
    ports:
      - "8000:80"
  db:
    image: postgres:16-alpine
    volumes:
      - db_data:/var/lib/postgresql/data
  adminer:
    image: adminer:4.8.1
    profiles: ["debug"]
    ports:
      - "8080:8080"

After the edit, run:

docker compose config --profiles
# Expected output:
#   - debug

Now test the intended behavior:

# Without the debug profile, adminer should NOT start
docker compose up -d
docker compose ps
# Expected: only web and db are running.
# adminer-1 should not appear.

# With the debug profile, adminer starts
docker compose --profile debug up -d adminer
docker compose ps
# Expected: adminer-1 is now running and listening on port 8080.

If docker compose up -d still starts adminer without the profile, check for a global COMPOSE_PROFILES environment variable in your shell or .env file. Remove it or unset it for this terminal session to avoid accidental activation.

Example of upgrading Compose file version and profile syntax

Compose files historically used a top-level version: key like version: "3.9". Newer Compose v2 versions no longer require this key; the file is treated as versionless and the CLI infers the schema. If your file still has version: "3.7", profiles work but with older resolution semantics. Migrating to versionless format is optional, but if you do it, remove the version: line and ensure any obsolete service attributes are cleaned up (e.g., links, depends_on with legacy format). Validate immediately:

docker compose config
# The output will be the expanded, normalized configuration. It should not error.

If you see a warning like the attribute 'version' is obsolete, it will be ignored, please remove it to avoid potential confusion, remove the line as part of the migration.

Minimal change principle

For each profile change, ask: "If this fails, can I restore the previous state in under five minutes?" If the answer is no, split the change into smaller steps. For example, first add a profile to a service without changing any runtime options, verify with docker compose config, then change the service image or environment in a second step. This isolates the cause if something breaks.

Verification and Diagnostics

After a profile change, verify not just that the expected services run, but that they run correctly with the right network and data.

Service-level checks

# List all services with their profile status
docker compose config --services
# Example output:
#   web
#   db
#   adminer

docker compose config --profile debug --services
# Expected output:
#   web
#   db
#   adminer  # included because debug profile activates it

# Check which services are actually running
docker compose ps --services
# Example output if debug profile is not active:
#   web
#   db

The mismatch between config --services and ps --services reveals which services are defined but not started. If a service should be started but is absent, the profile was not activated. Use docker compose --profile debug up -d adminer to start it, or set COMPOSE_PROFILES=debug in .env for a project-wide activation.

Network and dependency checks

Check that a profile-activated service can reach its dependencies. For example, if a worker service in the background profile needs the db service, and db is not in any profile (always active), then docker compose --profile background up -d will start both db and worker. But if db is in a different profile, worker may fail to resolve db. To avoid surprises, use depends_on with condition or at least order:

services:
  db:
    image: postgres:16-alpine
    profiles: ["data"]
  worker:
    image: python:3.11-alpine
    command: ["python", "worker.py"]
    profiles: ["background"]
    depends_on:
      - db

Then start with both profiles: docker compose --profile data --profile background up -d. If you forget one, the isolated dependency might fail; check logs:

docker compose logs worker
# Look for connection refused to db:5432 or DNS resolution error.

This is a common failure signal that indicates a profile dependency issue. The fix is to either ensure all needed profiles are activated together, or remove the profile from the dependency so it always runs.

Container-level diagnostics

# Inspect environment variables inside a running container, including COMPOSE_PROFILES if passed
docker compose exec web env | grep -i compose
# Example output if COMPOSE_PROFILES is passed through:
# COMPOSE_PROFILES=debug

# Check a specific container's health status
docker inspect --format='{{.State.Health.Status}}' <container_id>
# Example output: healthy

If a container is unhealthy after the profile change, inspect its logs:

docker logs <container_id> --tail 50
# Example failure line:
# Error: unable to connect to database at db:5432

Use named containers where possible for stable debugging:

docker compose up -d --scale web=2
# Then check both containers
docker compose ps

Remember that scaling a service with a bind mount often fails or causes file permission conflicts; ensure data services are not scaled accidentally.

Functional endpoint test

For a web service, after migration, run a local curl:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000
# Example expected output: 200

If the service is in a profile and you get a connection refused, it probably is not running. Activate the profile and retest.

Failure Modes and Recovery

Even with careful planning, failures happen. Here are the most likely failure modes, how to detect them, and the immediate recovery steps.

Failure mode 1: Profile silently not applied due to environment variable

Symptoms: docker compose up -d starts more or fewer services than expected. docker compose config --profiles shows profiles, but docker compose ps does not match.

Cause: A COMPOSE_PROFILES variable set in .env or the shell is overriding command-line --profile or adding unexpected profiles.

Recovery: Check and unset the variable:

unset COMPOSE_PROFILES
# Or remove the COMPOSE_PROFILES line from .env
docker compose --profile debug up -d

Then verify service list. If the .env file is used for other variables, keep the file but remove only the COMPOSE_PROFILES line.

Failure mode 2: YAML syntax error after editing profile

Symptoms: docker compose up -d fails with a YAML parse error or unexpected key.

Cause: Indentation error or placing profiles key at wrong level.

Recovery: Validate before applying:

docker compose config
# The command may output the error location, e.g.:
# services.adminer.profiles: unexpected element

Fix the YAML and revalidate. If you had a backup, you can restore it instantly:

cp docker-compose.yml.backup docker-compose.yml
docker compose config

Failure mode 3: Data loss after service recreation

Symptoms: After docker compose up -d --force-recreate, a service starts but data files are missing or default.

Cause: The service wrote to the container layer instead of a volume or bind mount.

Recovery: If data still exists in the old container, attempt to extract it before destroying the container:

# If the old container is still present but stopped, find its ID
docker ps -a --filter "name=app_1"
# Copy the data directory out
docker cp app_1:/var/lib/app ./data-backup
# Then recreate with a proper volume

Update the compose file with a volume, then bring the service back and restore data.

Failure mode 4: Network conflict after profile migration

Symptoms: Services in different profiles cannot communicate; logs show DNS errors or connection refused.

Cause: Both services are on the same network but one is not started when the other expects it, or profile isolation was unintentional.

Recovery: Ensure the dependency service is in a profile that is activated at the same time, or remove the dependency from a profile. You can add an external: true network if needed, but avoid network changes during migration unless necessary.

Rollback procedures

For any migration, have a rollback plan. If you use git, commit the known-good compose file before changes. If not, keep a timestamped backup:

cp docker-compose.yml docker-compose.yml.$(date +%Y%m%d%H%M%S)

To rollback a profile change, restore the file and recreate services:

cp docker-compose.yml.backup docker-compose.yml
docker compose down
docker compose up -d
# Verify old behavior
docker compose ps

If you upgraded the Compose CLI version itself and hit a bug, you can downgrade the Docker Compose plugin. On Linux, replace the plugin binary at ~/.docker/cli-plugins/docker-compose with an older version, ensuring it is executable (chmod +x). Restart your terminal or shell after replacing.

Common Pitfalls and How to Avoid Them

1. Using the same .env file for multiple projects without scoping. COMPOSE_PROFILES applies to the project in the current directory. If you run compose from different directories, each project may use the same .env if it is in the home directory. Store project-specific .env in the project directory.
2. Forgetting that profile activation is per command, not persistent. Unless you set COMPOSE_PROFILES, each docker compose command must include --profile. This leads to services stopping when a developer runs docker compose up -d without the flag, creating intermittent outages. Document the required profile in the README or a Makefile target.
3. Not upgrading the Compose CLI before using newer profile features. Some profile behaviors like depends_on with profiles had bugs fixed in later versions. Always run the latest stable Docker Compose v2 when doing a migration.
4. Overlapping profiles that make a service run twice. If a service has multiple profiles, it is activated when any of those profiles is selected. Ensure your profile selections do not cause duplicate service instances unless intended (e.g., using --scale separately).
5. Testing only docker compose up without checking data persistence. A service may start and appear healthy, but a subsequent recreate loses data. The canary file test described earlier is the cheapest way to verify.
6. Ignoring depends_on conditions in profile migrations. If a service uses depends_on: condition: service_healthy, the dependency must be in the same profile set or always active; otherwise the dependent service may fail to start. Prefer profile sets that include all dependencies or avoid profile isolation for health-checked dependencies.

Operations Checklist

Use this checklist before and after the migration. Assign a single owner for each item (for a small team, that might be the same person, but each item must have a named owner).

Revisit the checklist after any failure or incident: assign root cause, update the checklist to prevent recurrence, and re-run affected verification steps.

Conclusion

Docker Compose Profiles are a convenient way to manage optional services, but upgrading or migrating profiles safely requires a disciplined sequence: inventory the current state, make one minimal change at a time, verify the effect with concrete commands, and have a tested rollback path. This article walked through the full lifecycle with practical examples: checking versions and profile activation, moving a service into a profile, validating the configuration, diagnosing common failures, avoiding pitfalls, and using an operations checklist with assigned owners.

The most valuable habits are the persistence canary test and the use of docker compose config as a guardrail. If you adopt only two practices from this guide, adopt those. Before your next production change, choose one low-risk profile modification, run the checks end to end in a development environment, and compare actual output with the expected signals. Document the result and the responsible owner. Then, and only then, apply the same change to production with confidence.