Physical separation of indexing and ingestion

Understand how physical separation uses a SOK-managed ingestion tier and a separate indexer cluster, including its benefits, management responsibilities, requirements, and configuration sequence.

Physical separation extends the logical-separation model from separate processing pipelines to separate workload tiers for ingestion and indexing. The ingestion and indexing tiers scale independently and transfer data asynchronously through SmartBus.

Important: Physical separation is supported only for new physical-separation deployments using Splunk Enterprise v. 10.6 or higher and the Bring Your Own License (BYOL) model on a customer-managed platform (CMP) for Kubernetes. Physical separation is currently supported only on Amazon Web Services (AWS). Support for configuring physical separation on an existing SOK-managed deployment has not been established. Existing configuration and data are not automatically migrated or made available through the new configuration.

How physical separation works

In a physical-separation deployment, data flows through the ingestion and indexing tiers as follows:

  1. HTTP Event Collector (HEC) and Splunk-to-Splunk (S2S) clients send data to the IngestorCluster.

    Note:

    HTTP Event Collector (HEC) indexer acknowledgment is not supported for physical separation.

  2. The IngestorCluster performs the ingestion and processing work, then publish the processed data through SmartBus.

  3. SmartBus uses a durable remote queue to store messages or retrieval information. Larger messages or ingestion payloads are stored in the configured ingestion object store.

  4. The separate indexer cluster retrieves the processed data asynchronously when indexing capacity is available.

  5. The indexers index the retrieved data and make it available for search.

The IngestorCluster and separate indexer cluster must reference the same Queue and ObjectStorage custom resources. SOK resolves these references and builds the SmartBus configuration for each tier. You can scale the IngestorCluster independently of the indexing tier and configure horizontal pod autoscaling for the ingestion tier.

The ingestion object store used by SmartBus is separate from the remote object store that SmartStore uses for warm buckets.

Benefits of physical separation

Physical separation of indexing and ingestion offers the following benefits:

  • Independent scaling. Scale the ingestion and indexing tiers according to their individual workloads.

  • Data durability. Durable SmartBus buffering helps absorb temporary differences between ingestion demand and indexing capacity.

  • Operational clarity. Manage and monitor the ingestion and indexing tiers separately.

Management responsibilities

In a physical-separation deployment, SOK manages the Queue, ObjectStorage, and IngestorCluster custom resources. The Queue and ObjectStorage include references to the customer-provisioned queue and object store.

SOK does not manage the separate indexer cluster and the connection between the indexer and the IngestorCluster.

System / hardware requirements

Requirements for physical separation include:

  • Splunk Enterprise version 10.6 or higher.

  • A Splunk Enterprise deployment that uses the Bring Your Own License (BYOL) model and runs on a customer managed platform (CMP) for Kubernetes.

  • A compatible version of Splunk Operator for Kubernetes (SOK)

  • A supported Kubernetes environment.

  • A separate indexer cluster and a separate, SOK-managed IngestorCluster.

  • The following customer-provisioned AWS resources:

    • An Amazon Simple Queue Service (SQS) queue, including a dead-letter queue.

    • An Amazon Simple Storage Service (S3) bucket for ingestion data that exceeds the queue message-size limit.

    • An AWS Key Management System (KMS) key if encryption is required for the ingestion object store.

    For multisite deployments, configure the queue and ingestion object-storage location according to the requirements for each site.

  • Service-account permissions for the queue, dead-letter queue, and object store.

  • Network connectivity between the SOK-managed IngestorCluster, the separate indexer cluster, and the AWS services.

Note:

The ingestion object store used by SmartBus is separate from the remote object store that SmartStore uses for warm buckets. The object-store requirements and configuration guidance apply only to the Ingest-Tier Scaling object store.

Configuration sequence

Use the following sequence to configure a new physical-separation topology. This sequence does not provide a migration path from an existing deployment.

  1. Provision the customer-managed message queue, dead-letter queue, and ingestion object store. If encryption is required, provision a key in the applicable cloud key-management service. See Configuring Ingest-Tier Scaling.

  2. Configure the access permissions and network connectivity required by the SOK-managed ingestion tier and the separate indexer cluster.

  3. Create Queue and ObjectStorage custom resources that reference the customer-provisioned queue and object store.

  4. Deploy the SOK-managed IngestorCluster and configure it to reference the Queue and ObjectStorage resources.

  5. Integrate the separate indexer cluster. Configure it to retrieve data through the same SmartBus queue and object store, configure dead-letter queue handling on the indexer side, and maintain the required cluster and replication settings.

  6. Verify the readiness of the SOK-managed resources and the health of the ingestion and indexing tiers.

  7. Validate ingestion, asynchronous data transfer, indexing, and search.

To learn how to configure physical separation using SOK, see Configure physical separation of indexing and ingestion with SOK.