Linux Upgrade and Migration: A Practical, Step-by-Step Playbook

Intro

Upgrading or migrating a Linux system is a routine operation that can go wrong when done without planning. A failed upgrade can lead to downtime, data loss, or a broken system that is hard to restore. This guide provides a practical, step-by-step approach to planning, executing, and verifying Linux upgrades and migrations. It covers version inventory, safe configuration paths, diagnostics, rollback, and an operations checklist. By following these practices, developers, DevOps consultants, and technical startup teams can reduce risk and recover quickly if something fails.

A clear Linux migration reduces rework by separating research, brief, article generation, evaluation, approval, and publishing. For upgrade and migration tasks, this means separating planning, execution, validation, and rollback into distinct phases. The first useful pilot should be narrow, measurable, and easy to inspect locally before deployment. This article provides concrete commands and expected outputs to help you implement a controlled upgrade or migration.

Version and Environment Inventory

Before any upgrade or migration, you must understand the current state of your system. Inventory the operating system version, kernel, installed packages, file systems, and services. This baseline helps you detect changes after the operation and plan rollback if needed.

Gather System Information

Use the following commands to collect essential system details:

# OS release
cat /etc/os-release

# Kernel version
uname -r

# Hostname and architecture
hostnamectl

# Block devices and file systems
lsblk -f

# Mounted file systems
mount | column -t

# Installed packages (Debian/Ubuntu)
dpkg -l > installed-packages-before.txt

# Installed packages (RHEL/CentOS)
rpm -qa > installed-packages-before.txt

Example output for cat /etc/os-release on Ubuntu 20.04:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"

Record this baseline in a secure location outside the system (e.g., a configuration management database or a Git repository).

Identify Services and Dependencies

List enabled services and their states:

systemctl list-unit-files --state=enabled

Check for third-party repositories that might affect the upgrade:

# Debian/Ubuntu
ls /etc/apt/sources.list /etc/apt/sources.list.d/*.list

# RHEL/CentOS
ls /etc/yum.repos.d/*.repo

Also, check disk space because upgrades require free space for downloads and temporary files:

df -h /

Ensure you have at least a few gigabytes free. If not, clean up old kernels or logs.

Prerequisites for Upgrade/Migration

• Backup: Create full system backups or at least critical data and configuration files. Use tools like tar, rsync, or snapshot-capable file systems.
• Rollback plan: For package-based upgrades, use package manager history or snapshots. For migrations, have a replication or copy of the old system.
• Maintenance window: Schedule the operation during low usage. Inform stakeholders.
• Access: Ensure you have out-of-band access (e.g., console, IPMI, cloud console) in case network connectivity is lost.
• Documentation: Keep a log of every command and its output.

Safe Configuration Path

A safe configuration path means making changes in a controlled, reversible manner. Avoid big-bang upgrades; instead, stage changes and test after each step. If possible, test on a non-production clone or VM first.

Pilot First

The first useful pilot should be narrow, measurable, and easy to inspect locally before deployment. Choose a low-risk system or a disposable VM that mirrors production. Perform the upgrade there and document any issues.

Package Upgrade with Verification

For a distribution version upgrade (e.g., Ubuntu 20.04 to 22.04), use the official upgrade tool but with extra safety:

1. Update current packages:

sudo apt update && sudo apt upgrade -y

Expected output ends with something like:

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

1. Install update manager core (Ubuntu):

sudo apt install update-manager-core

1. Ensure prompts are set to LTS in /etc/update-manager/release-upgrades:

Prompt=lts

1. Run the release upgrade (do not use -d unless testing development):

sudo do-release-upgrade

This command is interactive; it will download packages and prompt before making changes. You can simulate first with:

do-release-upgrade -c  # check for available upgrade only

Expected output if an upgrade is available:

Checking for a new Ubuntu release
New release '22.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

1. During the upgrade, the system may reboot. After reboot, verify the new version.

Migration Between Systems

For a migration (moving to new hardware or cloud), consider using rsync to copy the root filesystem while excluding virtual filesystems:

sudo rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /mnt/newroot/

Then update bootloader and fstab on the new system. This is a broad example; always test on a disposable system first.

Verification and Diagnostics

After the upgrade or migration, verify that the system is functional. Check that the OS version is correct, services are running, and applications respond.

Verify OS Version and Kernel

cat /etc/os-release
uname -r

Compare with the baseline.

Check Service Status

systemctl --failed

The output should be empty or show no failed units:

0 loaded units listed.

Check critical services (e.g., sshd):

systemctl status sshd

Expected output includes active (running).

Test Network Connectivity

ip a
ping -c 4 8.8.8.8

Ensure you can reach external networks.

Validate Application Functionality

Run application-specific smoke tests. For web servers, use curl:

curl -I http://localhost

Expected output starts with HTTP/1.1 200 OK.

Check Logs for Errors

Review system logs for any anomalies:

journalctl -p err -b

Investigate any errors that appear.

Failure Modes and Recovery

Despite careful planning, upgrades and migrations can fail. Common failure modes include broken dependencies, kernel panics, service misconfiguration, or data corruption. Here are recovery strategies.

Rollback Using Package Manager

Many package managers keep a history or allow downgrades.

Debian/Ubuntu (APT): Check /var/log/apt/history.log for the transaction. You can downgrade specific packages using apt install <package>=<version> if older versions are still in cache or repository. To revert a do-release-upgrade, there is no official automatic rollback, so it is crucial to have a backup. However, you can try booting an older kernel from GRUB if the system fails to boot.

RHEL/CentOS (YUM/DNF): Use transaction history and undo:

sudo dnf history
sudo dnf history undo <ID>

Example output of dnf history:

ID | Command line              | Date and time       | Action(s) | Altered
-------------------------------------------------------------------------------
 5 | upgrade                   | 2023-10-01 12:00   | Upgrade   | 42
 4 | install httpd             | 2023-09-15 09:00   | Install   |  5

To undo the last upgrade (ID 5):

sudo dnf history undo 5

This will revert package changes, but be aware that it may not revert configuration file changes.

Boot Recovery with GRUB

If the system fails to boot after a kernel upgrade, reboot and select an older kernel from the GRUB menu. If GRUB is corrupted, boot from a live CD/USB, mount the root partition, and reinstall GRUB:

# Example: boot from live USB, mount root at /mnt
sudo mount /dev/sda1 /mnt
sudo grub-install --boot-directory=/mnt/boot /dev/sda

Restore from Backup

If changes are irrecoverable, restore from the backup taken before the operation. For file-level backups, use rsync or tar to restore. For VM snapshots, roll back the snapshot.

Migration Rollback

For a migration, you can switch back to the old system if it still exists. Keep the old system online in read-only mode or have a plan to repoint DNS or load balancer to the old IP.

Common Pitfalls and How to Avoid Them

Many upgrade and migration failures stem from avoidable mistakes. Here are the most common ones, why they happen, and how to prevent or recover from them.

1. Insufficient Disk Space

Why it happens: Upgrade packages and temporary files can consume several gigabytes. If the root partition fills up, the upgrade aborts midway, leaving the system in an inconsistent state.

How to avoid: Before starting, run df -h / and ensure at least 5-10 GB free. Clean up old kernels with sudo apt autoremove (Debian/Ubuntu) or sudo dnf autoremove (RHEL/CentOS), and clear logs or caches.

Recovery: If the upgrade fails due to disk space, boot into recovery mode or live environment, delete unnecessary files, and retry the upgrade. For APT, run sudo dpkg --configure -a to fix broken packages.

2. Ignoring Third-Party Repositories

Why it happens: Many systems have extra repositories (e.g., for PHP, Docker, or custom software). During a distribution upgrade, these repositories may lack packages for the new release, causing dependency conflicts.

How to avoid: List all repositories with ls /etc/apt/sources.list.d/ or ls /etc/yum.repos.d/, and disable non-essential ones before upgrading. Re-enable them after and test compatibility.

Recovery: If the upgrade breaks due to a third-party repo, disable that repo, fix broken packages with apt --fix-broken install or dnf distro-sync, then manually update the problematic software.

3. Skipping the Pilot Test

Why it happens: Time pressure or overconfidence leads to upgrading production directly, which can cause unexpected service failures.

How to avoid: Always test on a clone or VM with similar configuration. Use tools like virt-clone, cloud snapshots, or rsync a copy to a test machine.

Recovery: If production fails and no pilot was done, roll back using backups or snapshots immediately, then perform a pilot before retrying.

4. Not Having a Rollback Plan

Why it happens: Assuming the upgrade will succeed without issues. When it fails, there is no clear path to restore service, prolonging downtime.

How to avoid: Before starting, define a rollback procedure: for VMs, take a snapshot; for physical servers, have a full backup or a secondary boot disk. Document exact commands to revert.

Recovery: If no rollback plan was created and the system fails, user a live environment to access data, reinstall the OS, and restore from backups. It takes longer but is possible.

5. Not Verifying Services After Upgrade

Why it happens: The upgrade completes without errors, but a service fails to start due to configuration changes. Without verification, this is discovered only when users complain.

How to avoid: After upgrade, run a verification checklist: check systemctl --failed, test critical ports with ss -tulpn, and run application smoke tests.

Recovery: If a service fails, check its logs with journalctl -u <service>, compare config files with backups (in /etc or package manager backups), and adjust settings or reinstall the service.

6. Losing Configuration Customizations

Why it happens: Package upgrades may overwrite configuration files, especially if the admin chose to replace them with package maintainer versions during prompts.

How to avoid: Before upgrading, back up /etc directory or at least important config files. Use etckeeper to track changes in /etc with version control.

Recovery: Restore config files from backup, or use .dpkg-old / .rpmnew files left by package managers.

Operations Checklist

Use this checklist for every Linux upgrade or migration to ensure consistency and safety. Assign an accountable owner for each step and review the checklist after each major change.

Checklists should be reviewed by the accountable owner at the frequency stated, and outcomes tracked in your change management system.

Conclusion

Linux upgrades and migrations are manageable when approached methodically. Start with a thorough inventory and backup, test on a pilot, make changes in controlled steps, verify everything, and be prepared to roll back. Use the commands and checklists in this guide to reduce risk and ensure a smooth transition. By separating planning, execution, validation, and recovery, you minimize rework and downtime. Now you can apply these practices to your next Linux system update or move.