Remote configuration of OpenTelemetry Collectors

Prerequisites

To use remote configuration for OTel Collectors, make sure that the following conditions are met:

• OTel Collectors management is enabled in Splunk Enterprise.
• At least one OTel Collector agent is connected to agent management.
• The OTel Collector agent reports the OpAMP AcceptsRemoteConfig capability.
• To report remote configuration status, the OTel Collector agent also reports the OpAMP ReportsRemoteConfig capability.
• You have a complete OTel Collector configuration ready to upload as one or more files.

How remote configuration works

Agent management stores configuration sets and their assignments. When an OTel Collector checks in through OpAMP, agent management compares the hash reported by the collector with the hash of the assigned configuration set. If the hashes differ, agent management sends the assigned configuration set to the collector.

Configuration set

A complete desired OTel Collector configuration that contains one or more uploaded configuration files.

Assignment

A relationship between one OTel Collector agent and one configuration set. One configuration set can be assigned to many collectors. One collector can have one configuration set assigned.

Remote configuration status

The current delivery state of the assigned configuration set, such as not assigned, pending, applied, failed, or not deliverable.

Supported workflow

Use configuration sets for the following remote configuration workflows:

• Create a configuration set from one or more files. See Create a configuration set for OpenTelemetry Collectors.
• View configuration set metadata, file information, and deployment status.
• Replace a configuration set when you want to change its name, description, or uploaded configuration files. Replacing creates a new configuration set and moves existing collector assignments to the new set. See Replace a configuration set for OpenTelemetry Collectors.
• Assign a configuration set to one or more OTel Collector agents that report the AcceptsRemoteConfig capability. See Assign a configuration set to OpenTelemetry Collectors.
• Change the configuration set assigned to one or more OTel Collector agents when you want those agents to receive a different desired configuration. See Change the configuration set assigned to OpenTelemetry Collectors.
• Remove a configuration set assignment from one or more collectors when you want those collectors to stop receiving remote configuration from agent management. See Remove a configuration set assignment from OpenTelemetry Collectors.
• Delete a configuration set when no collectors are assigned to it. See Delete a configuration set for OpenTelemetry Collectors.
• Compare assigned remote configuration with the effective configuration that the collector reports.

Limitations

• Remote configuration applies only to OTel Collector agents that report the OpAMP AcceptsRemoteConfig capability.
• One collector can have one assigned configuration set.
• A configuration set must represent a complete desired collector configuration. Agent management does not assign individual configuration files to collectors.
• Configuration sets are not edited in place. To change the desired configuration for assigned collectors, create the configuration set and move assignments to the new set or replace the configuration set directly.
• The total raw body size across all files in one configuration set can't exceed 2 MiB (2,097,152 bytes). This limit applies to the whole configuration set, not to each uploaded file.

• One configuration set can contain up to 1000 files.

• The configuration set name can be up to 2 MiB.

• The configuration set description can be up to 2 MiB.

• Each uploaded file name can be up to 8 KiB (8192 bytes).

• You cannot delete a configuration set while any collectors are assigned to it.
• You cannot view configuration history, compare configuration versions, or roll back to an earlier configuration.
• Collectors without an assigned configuration set receive no remote configuration update from agent management.
• The effective configuration reported by a collector can differ from the assigned configuration set because of local settings, runtime overrides, or changes made outside agent management.

Configuration file contents and storage

Before you upload configuration files to agent management, verify that the files do not contain secrets that require storage protection.

The OpAMP service (OAS) stores configuration set files and assignments. OAS and the Postgres sidecar do not encrypt stored configuration at rest. This includes configuration stored by OAS and copies of that configuration in local Postgres backups or write-ahead log (WAL) archives.

If your organization requires storage encryption for configuration data, enable full-disk or volume encryption in your customer-managed environment.

To keep secrets out of configuration stored by OAS, inject secret values through mechanisms supported by the OpenTelemetry Collector. For example, use one of the following methods:

• Use environment-variable expansion, such as ${env:SECRET}.
• Use a referenced configuration fragment, such as ${file:secrets.yaml}, and distribute that fragment through a separate secure channel.

For more information, see https://opentelemetry.io/docs/collector/configuration.