Kubernetes Process with Resource Ratio Anomalies

THIS IS A EXPERIMENTAL DETECTION

This detection has been marked experimental by the Splunk Threat Research team. This means we have not been able to test, simulate, or build datasets for this detection. Use at your own risk. This analytic is NOT supported.

Try in Splunk Security Cloud

Description

This analytic detects anomalously changes in the ratio between specific process resources on a Kubernetes node, based on the past behavior for each process running in the Kubernetes scope on that node. This detection leverages process metrics harvested using an OTEL collector and hostmetrics receiver, and is pulled from Splunk Observability cloud using the Splunk Infrastructure Monitoring Add-on. (https://splunkbase.splunk.com/app/5247). This detection also leverages a lookup table that contains average and standard deviation for the cpu:disk operations, cpu:mem, cpu:thread count, disk operations:thread count, and mem:disk operations ratios. This is used to indicate an anomalous change in resource ratios that indicate the workload has changed behavior irrespective of load. Changes in the relationship between utilization of different resources can indicate a change in behavior of the monitored process, which can indicate a potentially compromised application. Deviations in resource ratios, such as memory-to-CPU or CPU-to-disk utilization, may signify compromised processes, malicious activity, or misconfigurations that could pose risks. A change in process behavior could signify a potential security breach within the Kubernetes environment, where an attacker may have compromised a process either on the node or running within a container.

• Type: Anomaly

• Product: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud

• Last Updated: 2023-12-18
• Author: Matthew Moore, Splunk
• ID: 0d42b295-0f1f-4183-b75e-377975f47c65

Annotations

ATT&CK

ATT&CK

ID	Technique	Tactic
T1204	User Execution	Execution

Kill Chain Phase

• Installation

NIST

• DE.AE

CIS20

• CIS 13

CVE

Search

| mstats avg(process.*) as process.* where `kubernetes_metrics` by host.name k8s.cluster.name k8s.node.name process.executable.name span=10s 
| eval cpu:mem = 'process.cpu.utilization'/'process.memory.utilization' 
| eval cpu:disk = 'process.cpu.utilization'/'process.disk.operations' 
| eval mem:disk = 'process.memory.utilization'/'process.disk.operations' 
| eval cpu:threads = 'process.cpu.utilization'/'process.threads' 
| eval disk:threads = 'process.disk.operations'/'process.threads' 
| eval key = 'k8s.cluster.name' + ":" + 'host.name' + ":" + 'process.executable.name' 
| lookup k8s_process_resource_ratio_baseline key 
| fillnull 
| eval anomalies = "" 
| foreach stdev_* [ eval anomalies =if( '<<MATCHSTR>>' > ('avg_<<MATCHSTR>>' + 4 * 'stdev_<<MATCHSTR>>'), anomalies + "<<MATCHSTR>> ratio higher than average by " + tostring(round(('<<MATCHSTR>>' - 'avg_<<MATCHSTR>>')/'stdev_<<MATCHSTR>>' ,2)) + " Standard Deviations. <<MATCHSTR>>=" + tostring('<<MATCHSTR>>') + " avg_<<MATCHSTR>>=" + tostring('avg_<<MATCHSTR>>') + " 'stdev_<<MATCHSTR>>'=" + tostring('stdev_<<MATCHSTR>>') + ", " , anomalies) ] 
| eval anomalies = replace(anomalies, ",\s$", "") 
| where anomalies!="" 
| stats count values(anomalies) as anomalies by host.name k8s.cluster.name k8s.node.name process.executable.name 
| where count > 5 
| rename host.name as host 
| `kubernetes_process_with_resource_ratio_anomalies_filter`

Macros

The SPL above uses the following Macros:

• kubernetes_metrics

kubernetes_process_with_resource_ratio_anomalies_filter is a empty macro by default. It allows the user to filter out any results (false positives) without editing the SPL.

Lookups

The SPL above uses the following Lookups:

• k8s_process_resource_ratio_baseline with data

Required fields

List of fields required to use this analytic.

• process.*
• host.name
• k8s.cluster.name
• k8s.node.name
• process.executable.name

How To Implement

To implement this detection, follow these steps: \

• Deploy the OpenTelemetry Collector (OTEL) to your Kubernetes cluster.\
• Enable the hostmetrics/process receiver in the OTEL configuration.\
• Ensure that the process metrics, specifically Process.cpu.utilization and process.memory.utilization, are enabled.\
• Install the Splunk Infrastructure Monitoring (SIM) add-on. (ref: https://splunkbase.splunk.com/app/5247)\
• Configure the SIM add-on with your Observability Cloud Organization ID and Access Token.\
• Set up the SIM modular input to ingest Process Metrics. Name this input "sim_process_metrics_to_metrics_index".\
• In the SIM configuration, set the Organization ID to your Observability Cloud Organization ID.\
• Set the Signal Flow Program to the following: data('process.threads').publish(label='A'); data('process.cpu.utilization').publish(label='B'); data('process.cpu.time').publish(label='C'); data('process.disk.io').publish(label='D'); data('process.memory.usage').publish(label='E'); data('process.memory.virtual').publish(label='F'); data('process.memory.utilization').publish(label='G'); data('process.cpu.utilization').publish(label='H'); data('process.disk.operations').publish(label='I'); data('process.handles').publish(label='J'); data('process.threads').publish(label='K')\
• Set the Metric Resolution to 10000.\
• Leave all other settings at their default values.\
• Run the Search Baseline Of Kubernetes Container Network IO Ratio

  Known False Positives

  unknown

Associated Analytic Story

• Abnormal Kubernetes Behavior using Splunk Infrastructure Monitoring

RBA

Risk Score	Impact	Confidence	Message
25.0	50	50	Kubernetes Process with Resource Ratio Anomalies on host $host$

The Risk Score is calculated by the following formula: Risk Score = (Impact * Confidence/100). Initial Confidence and Impact is set by the analytic author.

Reference

• https://github.com/signalfx/splunk-otel-collector-chart

Test Dataset

Replay any dataset to Splunk Enterprise by using our replay.py tool or the UI. Alternatively you can replay a dataset into a Splunk Attack Range

source | version: 1