Index and ingestion separation

Separate ingestion and indexing services in the Splunk Operator for Kubernetes.

You can separate indexing and ingestion logically or physically. Logical separation uses separate processing pipelines that can run on the same indexer instances and does not require Splunk Operator for Kubernetes (SOK). Physical separation uses SOK to manage a dedicated ingestion tier, while a separately managed indexer cluster retrieves processed data through a durable remote queue.

Important: Physical separation is supported only for new separated topologies in Splunk Enterprise 10.6 or higher deployments that use the Bring Your Own License (BYOL) model on a customer-managed platform (CMP) for Kubernetes. Physical separation is currently supported only on Amazon Web Services (AWS). Support for configuring physical separation on an existing SOK-managed deployment has not been established. Existing configuration and data are not automatically migrated or made available through the new configuration.

In physical separation, ingestion runs in a SOK-managed IngestorCluster, while indexing runs in a separate indexer cluster. The ingesting and indexing workloads are separate, but they can run in the same Kubernetes environment.

SOK manages the Queue, ObjectStorage, and IngestorCluster custom resources. The customer provisions and manages the external queue, dead-letter queue, and object store. The separate indexer cluster is configured and managed independently of SOK.

For physical-separation prerequisites and configuration instructions, see Configure physical separation of indexing and ingestion with SOK.

Benefits of physical separation

Physical separation provides the following benefits:

  • Independent scaling: Match resource allocation to ingestion or indexing workload.
  • Data durability: Use a durable remote queue and object store to buffer processed data while indexing capacity is unavailable.
  • Operational clarity: Manage and monitor the ingestion and indexing tiers separately.

To learn more about physical separation, see Physical separation of indexing and ingestion in the Manage Indexers and Indexer Cluster manual.