sops key migration and rotation

This article uses a general migration from a standalone age key to an SSH host key to explain how to change a secrets file's decryption key, move to another machine, or add a decryption identity. All example paths and recipients are placeholders, not an actual system.

1. Why switch?#

The original setup uses a standalone age key (/persist/secrets/age-key.txt) as the private key for sops decryption:

secrets.yaml（git 内，密文）
    │ 解密
    ▼
/persist/secrets/age-key.txt   ← 独立管理的私钥，要手动备份/迁移

The problems:

• One more key to keep safe
• It is easy to forget the key when moving to another machine, leaving secrets undecryptable
• Without another valid recipient or a backup, losing the private key may make the encrypted data unrecoverable

The new setup reuses the system's existing SSH host key:

secrets.yaml（git 内，密文）
    │ 解密
    ▼
/etc/ssh/ssh_host_ed25519_key   ← 由系统生成或恢复；迁移需明确备份与恢复

The benefit is one fewer standalone key to maintain, but the host key still needs backup and recovery procedures. Reinstalling or regenerating it changes the machine's identity; migration problems do not simply disappear.

The trade-off Compromising the SSH host key expands the impact from “secrets only” to “SSH + secrets.” Whether that is acceptable depends on your threat model, backups, and permission management. A separate decryption key keeps the two capabilities apart.

2. The central concept: two recipients#

sops supports registering multiple recipients in one file (multiple parties able to decrypt it). This is the key migration technique: add the new key first, keep the old key, and remove the old key only after verification succeeds, preserving a fallback throughout:

sops:
  age:
    - enc: <AES 密钥用旧 key 加密的块>
      recipient: AGE_RECIPIENT_PLACEHOLDER_NOT_VALID        # 旧独立 age key（保留兜底）
    - enc: <同一把 AES 密钥用新 key 加密的块>
      recipient: AGE_RECIPIENT_PLACEHOLDER_NOT_VALID      # 新 SSH host key

Each private key can decrypt only "its own copy" of the encrypted block, but each obtains the same AES data key.

3. Prerequisites#

1. Tools and permissions: provide sops, age, and ssh-to-age through the project's declarative tool configuration, and confirm their versions and the limits of administrator access.
2. Recovery material: back up the old key and encrypted data. Confirm that the new host key's public key belongs to the target machine; its private key does not need to leave that machine. An example one-off Nix tool environment is nix-shell -p ssh-to-age age sops.
3. Obtain the new key's age public key:

ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub
# → AGE_RECIPIENT_PLACEHOLDER_NOT_VALID（本例 SSH host key 的 age 公钥）

4. Confirm that the old key can decrypt the file (the pre-migration baseline):

SOPS_AGE_KEY_FILE=/persist/secrets/age-key.txt sops decrypt host/secrets/secrets.yaml > /dev/null

Permissions and verification Only authorized readers should have access to private keys. If root is required, work in a controlled administrator shell with existing authorization; do not use passwordless tool wrappers to bypass the permission design. For the baseline decryption check, inspect the exit code without printing secrets. Record tool versions and the recovery location.

4. Migration steps#

Step 1: update .sops.yaml to declare the new recipient

.sops.yaml contains sops creation rules; define keys with YAML anchors:

keys:
  - &admin-key AGE_RECIPIENT_PLACEHOLDER_NOT_VALID        # 旧独立 age key
  - &server-key AGE_RECIPIENT_PLACEHOLDER_NOT_VALID     # 新 SSH host key
creation_rules:
  - path_regex: host/secrets/secrets\.yaml$
    key_groups:
      - age:
        - *admin-key
        - *server-key

Step 2: use sops updatekeys to synchronize recipients to secrets.yaml

cd /path/to/config-repo  # 替换为目标配置仓库
SOPS_AGE_KEY_FILE=/persist/secrets/age-key.txt \
  sops updatekeys host/secrets/secrets.yaml

The output will show +++ AGE_RECIPIENT_PLACEHOLDER_NOT_VALID (added) and the retained old key. updatekeys updates the recipient wrapping of the data key; it does not rotate the data key. Access to the old key is required.

Step 3: change host/sops.nix to switch the decryption source

# 改之前
sops = {
  age.keyFile = "/persist/secrets/age-key.txt";
};

# 过渡后（原 keyFile 仍可保留到验证完成）
sops = {
  age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
};

A fallback remains At this point, secrets.yaml still has two recipients (old + new). Even if the new setup has a problem, the old key can still decrypt it. Wait until the rebuild has been verified before considering removal of the old key.

Step 4: verify that the new key can decrypt the file

# 在目标机器已有授权的管理员 shell 内；工具已准备
umask 077
task_key_dir=$(mktemp -d /run/sops-migrate.XXXXXXXX)
trap 'rm -rf -- "$task_key_dir"' EXIT
ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key > "$task_key_dir/new-identity.txt" &&
  SOPS_AGE_KEY_FILE="$task_key_dir/new-identity.txt" \
    sops decrypt host/secrets/secrets.yaml > /dev/null

A successful decryption exit code only proves that the CLI can read the encrypted file. To verify the new key specifically, use a controlled environment containing only the new identity and disable additional default keys, KMS, and other sources. You must still verify the actual sops-nix deployment path afterward.

Step 5: verify in stages and apply with authorization

# 在已准备好恢复方案的配置仓库内，用实际 flake target 替换 workstation
nix flake check --no-build
nixos-rebuild dry-build --flake .#workstation
# 经明确授权后，由管理员应用；这会修改运行系统
sudo nixos-rebuild switch --flake .#workstation

5. Verification after rebuilding#

systemctl status sops-install-secrets.service # 仅适用于 systemd 解密路径
systemctl status example-app.service # 检查消费方，不显示 secret 内容
ls -la /run/secrets/app_env # 仅证明产物存在；结合解密日志和应用行为检查

6. Rollback and cleanup#

• Rollback: change sops.nix back to age.keyFile and rebuild. The old key remains at /persist/secrets/age-key.txt, and secrets.yaml retains both recipients. Before rolling back, also confirm that the old identity is still valid, the old encrypted file exists, and the configuration version and system recovery path are available.
• Cleanup: once the new deployment path is confirmed stable:
  1. Remove the old anchor from .sops.yaml and run sops updatekeys to remove the old recipient; use sops rotate --in-place to rotate the data key as needed
  2. Delete /persist/secrets/age-key.txt
  3. Update the project documentation, recovery records, and key-custody records

Cleanup is irreversible Removing a recipient does not revoke the old key's access to git history, backups, or data keys obtained earlier. First verify the new identity and operational deployment path, then rotate the data key according to policy. If the old identity is no longer trusted, rotate the secret values and related credentials as well. Confirm historical recovery requirements before deleting the old private key; deleting a file is not a substitute for actual revocation.

7. Pitfalls and lessons learned#

Related#

• Notes on NixOS update mechanics
• Configuration repository: /path/to/config-repo/host/sops.nix, .sops.yaml, host/secrets/secrets.yaml