Custom resources for index and ingestion separation

Custom resource reference for index and ingestion separation.

The Splunk Operator provides Queue, ObjectStorage, IngestorCluster, and IndexerCluster custom resources to configure index and ingestion separation.

Note:

The Queue and ObjectStorage resources are immutable after creation, meaning that you cannot change them. However, you can change the queueRef and objectStorageRef references in the IngestorCluster. When a reference changes, SOK regenerates the configuration resources used by the ingestion tier and updates the Kubernetes StatefulSet that manages the IngestorCluster pods.

Queue

Queue stores the remote queue information shared by an IngestorCluster and an indexer cluster. SOK does not create or manage the external queue or dead-letter queue.

The only supported message queue provider is sqs.

Key Type Required Description
provider string Yes Provider of the message queue. Allowed value: sqs.
sqs SQS Yes if provider = sqs SQS message queue inputs.

SQS message queue inputs:

Key Type Required Description
name string Yes Name of the physical queue.
authregion string Optional Region used for authentication and to construct the service endpoint.
endpoint string Optional AWS SQS service endpoint. If omitted, SOK constructs the endpoint based on authRegion.
dlq string Yes Name of the physical dead-letter queue.
secretKeyRef object Optional Per-key selectors for AWS credentials. When not set, IRSA or workload identity mechanism is used. Contains awsAccessKey and awsSecretKey, each with SecretKeySelector with name and key fields.

Example:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Queue
metadata:
  name: queue
spec:
  provider: sqs
  sqs:
    name: sqs-test
    authRegion: us-west-2
    endpoint: https://sqs.us-west-2.amazonaws.com
    dlq: sqs-dlq-test

Include the secretKeyRef field only to use static AWS credentials instead of workload identity. To use static AWS credentials instead of workload identity, add the optional secretKeyRef input under spec.sqs:

CODE 
secretKeyRef:
  awsAccessKey:
    name: s3-secret
    key: s3_access_key
  awsSecretKey:
    name: s3-secret
    key: s3_secret_key

The provider, queue name, authentication region, endpoint, and dead-letter queue cannot be changed after the Queue resource is created. You can update secretKeyRef.

If you omit secretKeyRef, configure the pods to use IRSA or another supported workload-identity mechanism. If you specify secretKeyRef, the referenced Secrets must be stored in the same namespace as the Queue resource.

SOK reads the selected keys and provides the credentials to the referenced tiers through a generated, immutable Secret. SOK monitors the referenced Secrets. When credential data changes, SOK creates a new generated Secret and rolls the affected pods declaratively.

The Queue controller does not validate connectivity to the external queue or dead-letter queue. A Ready status indicates that Queue resource has been reconciled, meaning that SOK has processed the resource. Access and queue health must be verified separately.

ObjectStorage

The object store referenced by ObjectStorage stores messages that exceed the maximum message size. Configure the IngestorCluster and the separate indexer cluster to use the same object-store location. SOK uses the ObjectStorage resource to configure the ingestion tier. The bucket and path must already exist and be accessible to the required pods.

The S3 provider is the only supported object storage provider.

Key Type Required Description
provider String Yes Provider of object storage. The only allowed value is s3.
s3 S3 Yes if provider is s3 S3 object storage inputs.

S3 object storage inputs:

Key Type Required Description
path String Yes Remote storage location for messages that exceed the maximum message size.
endpoint String Optional S3-compatible service endpoint. If omitted, SOK constructs the endpoint based on the authRegion field in the Queue.
encryptionScheme String Optional Remote-storage encryption scheme: sse-s3, sse-c, or none.
kmsEndpoint String Optional KMS endpoint used with encryptionScheme: sse-c. If omitted, SOK constructs the endpoint based on the authRegion field in the Queue.
kmsKeyId String Optional Identifier of the KMS key. Required when encryptionScheme is sse-c.

After creation of ObjectStorage, the inputs are immutable, meaning that they cannot be changed.

The ObjectStorage controller does not create the external bucket or validate connectivity to it. A Ready status means that the ObjectStorage resource has been reconciled, meaning that SOK has processed the ObjectStorage resource and applied the configuration it manages.

Example:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: ObjectStorage
metadata:
  name: os
spec:
  provider: s3
  s3:
    path: ingestion/smartbus-test
    endpoint: https://s3.us-west-2.amazonaws.com

IngestorCluster

IngestorCluster defines the ingestion tier. Its Splunk pods receive and publish events to the SmartBus queue using outputs.conf.

In addition to common spec inputs, the IngestorCluster resource provides the following spec configuration parameters:

Key Type Required Description
replicas Integer Optional The number of ingestion pods (defaults to 1).
queueRef corev1.ObjectReference Yes Message queue reference
objectStorageRef corev1.ObjectReference Yes Object storage reference

Example:

The example uses Amazon Simple Queue Service (SQS) and Amazon Simple Storage Service (S3). It configures the ingestion tier as follows:

  • It creates an IngestorCluster named ingestor in the default namespace with three replicas.

  • The Queue and ObjectStorage references specify the SmartBus queue and object-storage settings.

  • AWS Identity and Access Management (IAM) Roles for Service Accounts (IRSA) provides the pods with access to the required AWS resources.

  • Messages are written to the sqs-test queue in the us-west-2 region.

  • The sqs-dlq-test queue is configured as the dead-letter queue.

  • Messages that exceed the queue's maximum size are written to the ingestion/smartbus-test S3 path.

CODE 
apiVersion: enterprise.splunk.com/v4
kind: IngestorCluster
metadata:
  name: ingestor
spec:
  serviceAccount: ingestor-sa 
  replicas: 3
  queueRef:
    name: queue
  objectStorageRef:
    name: os