Migrate encrypted configuration values to the $8$ cipher format

Starting in Splunk Enterprise version 10.6, you can use the Splunk CLI to migrate existing encrypted configuration values from the $7$ cipher format to the $8$ cipher format.

Migration is optional. The $7$ cipher format remains readable indefinitely, and upgrading to version 10.6 does not require migration. You might want to migrate if you require full FIPS 140-3 compliance for at-rest encrypted values, or if you want to ensure all existing values use the Hash-based Message Authentication Code (HMAC)-based Key Derivation Function (HKDF)-SHA256 key derivation method.

Before you run either command, note the following:

  • Both commands leave the splunk.secret file unchanged.
  • Both commands are safe to run again. Running them again on a fully migrated instance reports Nothing to re-encrypt. (single instance) or reencrypted=0 (search head cluster) without making changes.
  • Both commands require splunkd to be running; they have no offline mode.
  • Both commands prompt for confirmation before making changes. Pass --answer-yes as a CLI argument to skip the prompt in non-interactive contexts such as scripts or automation pipelines. Without a terminal, the command exits without doing the work unless you pass --answer-yes.
  • Both commands leave values unchanged if the commands cannot decrypt them with the current splunk.secret, and report those values in the output. This is expected for any value encrypted under a different splunk.secret, such as values copied from another instance.
  • The splunkd process is running on the instance or cluster you want to migrate.
  • To migrate a single instance, your account holds a role with the edit_server capability.
  • To migrate a search head cluster, your account holds a role with the edit_search_head_clustering capability.

Migrate a standalone instance

Use this procedure on standalone instances, indexers, forwarders, and search heads that are not part of a search head cluster. If search head clustering is turned on for the instance, the command refuses and instructs you to use Migrate a search head cluster instead.

  1. Record the number of $7$ values on disk before you migrate, so you can confirm the result:
    CODE 
    splunk btool passwords list --debug | grep -c '\$7\$'
splunk btool server    list --debug | grep -c '\$7\$'
  2. Run the migration command:
    CODE 
    splunk reencrypt secrets -auth <user>:<password> --answer-yes
    The command rewrites each $7$ value as $8$ in place and reports the number of fields it migrated:
    CODE 
    Re-encrypted 5 field(s) as $8$.
    If there is nothing to migrate, it reports:
    CODE 
    Nothing to re-encrypt.
  3. Verify the migration by running the commands from step 1 again. Both should return 0.
  4. (Optional) Confirm in the audit log:
    CODE 
    index=_audit action="REENCRYPT SECRETS"
    A successful run appears as info=succeeded. A run that left fields un-migrated appears as info=failed even though the command exits 0. Check the output for fields reported as Left N field(s) un-migrated.

Migrate a search head cluster

Use this procedure from any member of a search head cluster. The command signals every member to migrate its own local configuration values. The command returns before the migration is complete; each member migrates on its next heartbeat contact with the captain, which defaults to 5 seconds.

Note: Do not run this command while a rolling restart, rolling upgrade, or splunk.secret rotation is in progress. The captain refuses the request in those conditions.
  1. Confirm cluster health and that no maintenance operation is in progress:
    CODE 
    splunk show shcluster-status  -auth <user>:<password>
splunk list shcluster-members -auth <user>:<password>
    Every member should be Up with a single captain and no rolling restart flag set.
  2. Run the migration command from any cluster member:
    CODE 
    splunk reencrypt shcluster-secrets -auth <user>:<password> --answer-yes
    The command returns immediately with:
    CODE 
    Re-encryption of search head cluster member secrets initiated.
  3. Wait for each member to pick up the signal and complete its migration. Check each member's $SPLUNK_HOME/var/log/splunk/splunkd.log for the completion line:
    CODE 
    SHCSlave - event=SHPMember::reencryptSecrets result=done cipher=$8$ reencrypted=5 unmigrated=0
    A reencrypted=0 value means the member was already fully migrated.
  4. Verify the $7$ count on each member:
    CODE 
    splunk btool passwords list --debug | grep -c '\$7\$'
splunk btool server    list --debug | grep -c '\$7\$'
    Both should return 0.
Note: A member that is down when you run the command still receives the signal when it rejoins the cluster. You do not need to run the command again for a member that was temporarily unavailable. Run the command again only if the cluster captain changed or restarted before all members received the signal.