BPF Trace


BPF Trace

High level tracing language

eBPF based

static and dynamic tracing

kernel and user namespace

Tracing

- Ability to interrupt flow of running program. at configurable location. 

- execution of code to obtain some data

- For visibility 

It affects performance. 

Perf sample all data and then you filter. it is different. 

Static Tracing

hard coded instrumentation points in code

Dynamic Tracing

Can instrument any software event. E.g. function call, return

live modification of instruction set

it needs symbol table

E.g. gdb

Probe Providers

They provides probe point. At probe point we can execute some code. 

1. BEGIN/END : built-in events. BEGIN to initialize data structure END to aggregate counters. 

2. hardware: processor level events

3. interval: built-in timers

4. kprobe / kretprobe : Kernel function tracing

5. profile: timed sampling

6. rawtracepoint: Kernel static trace point with raw argument

7. software: kernel software event

8. tracepoint: kernel static tracepoint 

9. uprobe / uretprobe : user-level function start / return

sudo bpftrace -l | wc -l

153620

Most of them are kernel level. static trace points. 

e.g. 1 openat syscall using tracepoint:syscall:sys_enter_openat

Here 

(1) "comm" is built-in variable having process name

(2) "args" is struct containing all tracepoint arguments

for more details about "args" struct, use below command

bpftrace -vl tracepoint:syscall:sys_enter_openat

To execute

sudo bpftrace -e 'tracepoint:syscall:sys_enter_openat {printf();}

sudo bpftrace <file name> 

It has buil-in histagram

in the command we can specify filter

/pid == 32184/

/comm == "<name of process>"/


Reference

github.com/bpftrace


 https://media.ccc.de/v/osmodevcon2024-204-introduction-to-xdp-ebpf-and-afxdp

eBPF, XDP and AF_XDP


1. NIC -> Kernel
2. Kernel process it layer by layer
3. Sub-systems tunnel, traffic shaping, IPSec, NAT, Packet filter
4. enqueued to socket receive queue. 
5. wake up user space process
6. copy data to user space process

eBPF, XDP and AF_XDP are Kernel Feature for high speed packet processing


 * [eBPF](https://ebpf.io/what-is-ebpf/) (extended Berkeley Packet Filter) is a kind of virtual machine that runs sandboxed programs inside the Linux kernel.

- TCP Dump uses BPF. 

- Not tied specifically to networking

- in kernel verifiers. No endless loop. no access to invalid memory. 

Instructions

- RFC by IETF defines instruction set

- VM + limited instruction set

- JIT

Program Types

1. XDP

2. socket filtering, kprobe, tc schedular (action, classifiers), performance events

3. tracing: function entry, function exit, modify return value

4. cgroups, light weight tunnels

5. TCP stream parsing ( kernel assisted TLS and KCM = kernel connection machine. IP multiplex)

$ bpftool prog

$ clang target bpf

eBPF maps

1. K-v store

2. hash, array, radix-tree, bloom filter

3. _per_cpu variants for multi-processor efficiency

Use cases

1. device driver for DPDK, VPP and related user plane frameworks

2. load balancer (e.g. dnsdist)

3. software switch (e.g. vSwitch)

4. DoS protection filter

5. Intrusion Detection Systems (e.g. Suricata)

6. eupf = eBPF + AF_XDP for accelerated 5G UPF

* [XDP](https://docs.cilium.io/en/latest/bpf/progtypes/#xdp) (eXpress Data Path) is a framework for eBPF that enables high-performance programmable packet processing in the Linux kernel

Kernel 4.18
XDP_DROP : DoS. Drop at line rate. similar to disconnect cable. 
XDP_PASS : Slow Path OR other NIC. use case like netfilter. 
XDP_TX : 
XDP_REDIRECT

Register eBPF program to XDP hook

* [AF_XDP](https://www.kernel.org/doc/html/next/networking/af_xdp.html) is an *address family* that is optimized for high-performance packet processing. It allows in-kernel XDP eBPF programs to efficiently pass packets to userspace via memory-mapped ring buffers.

XDP Socket
Operates on raw packet
Memory mapped Rx/Tx Ring. so zero copy. It is at network interface card interrupt receive level. receive queue of network card. Morden NIC has many receive queues. one for each CPU. based on connection.  it can edit the packet.no read/write. as no copy between user space and kernel space. 
Single socket for RX+Tx. Not possible
descriptor points to memory 

eBPF program will decide go to AF_XDP or not. 

AF_XDP can also generate pcap. 

Ref:
github.com/xdp-project/xdp-tools
github.com/xdp-project/xdp-tutorial
qmonnet.github.io/whirl-offload/2021/09/23/bpftool-features-thread
https://media.ccc.de/v/osmodevcon2024-204-introduction-to-xdp-ebpf-and-afxdp

XDP is close to DPDK. But DPDK is faster. 

Connect. Build. Contain. Agents in the AI Governance Era


Reference https://www.meetup.com/collabnix/events/316301263/

1. Contain, Forecast, Adjudicate: Three Agent Controls No Protocol Gives You

https://www.nasiko.com/

2. Securing Your Agentic Stack (Workshop) 

https://agentic.dockerworkshop.com/

https://agentic.dockerworkshop.com/#/labs/securing-the-agentic-stack-slides

https://agentic.dockerworkshop.com/#/labs/securing-the-agentic-stack

1. What it contains? What is software Artifact

SBOM

docker scout sbom --format spdx --output baseline.spdx.json catalog-service:baseline

2. From where it comes from?

Provenience

3. CI pipeline. Who approve image with vulnerabilities? Can I. verify attestation source

https://docs.docker.com/scout/deep-dive/advisory-db-sources/

match with SBOM

Now AIBOM term is emerging. https://www.ajeetraina.com/ai-bom-explained-why-your-sbom-stops-where-your-ai-system-starts

VEX: Vulnerability Exploitability eXchange 

In Docker Desktop, we can filter vulnerabilities based on fixable or not

SLSA

Level 1 to 3

FIPS 140 For US

4. Can it be restricted? Sandbox  

We need base image with near zero vulnerabilities.

All images shall be signed

Allow coding agent only what it needs.

"/rc" in Claude. Remote control. You will keep getting notifications in your mobile. 

AI Agent

local sandbox and cloud sandbox. SBX is running on microVM. Agent can change kernel also. So SBX runs on microVM instead of container.

Inside mircoVM also we run Docker engine and docker daemon 

Docker Hub have MCP policy, AI policy

Now we have SBX compose file with policy

MCP Toolkit

Hardened MCP servers

https://agentic.dockerworkshop.com/#/labs/securing-the-agentic-stack-slides/workshop-75

Docker hub

DHI Docker Hardening Images

Images and AI models also on 

3. Docker sbx kits: you explorations to contain AI Agents begins here

AI Agent

https://docs.docker.com/ai/sandboxes/customize/

https://docs.docker.com/ai/sandboxes/customize/kits/

https://hub.docker.com/search?type=sbx_kit

Tools

1. MIXIN kit

It has enhanced capabilities

build agent from scratch

2. Sandbox Kit

-------

Kit has spec.yaml file

Files are payload. it can have docker compose file. certificate file etc. Some will go to sandbox and other files remain on laptop

start from Mixin kit, as Sandbox kit has many definition

Now let's have customize AI agent. 

https://floci.io/ is like localstack. Cloud emulators 

floci CLI is inside sandbox

1. create shell sendbox

2. run docker compose

same can be done with spec.yaml file

We have DHI for langchain also. It can be inside sandbox

4. Beyond the Agent: Building AI Systems You Can Trust

If it hallucinate then workflow has problem or model has problem? 

5. The New Primitives of AI: YAML, OCI, and Agent Infrastructure

you write agent in your Jupiter Notebook

"It works on my notebook"

"docker agent"

oci artifacts

API Days India 2026 - Part 1


I did not attend this event in-person. I gone through YouTube Playlist https://www.youtube.com/playlist?list=PLcWDDGrTp5AU It has 41 videos

In part 1, let me cover few of them

-------------------

1. Made in India. the founders behind API tools

  • specmatic
  • Beeceptor
  • keploy.io
  • postman
  • bruno
  • karate labs

------------------------

2. Restoring Trust in AI-Native development

Earlier we used to have Pre-commit hook before agent started coding. 

Now, SDD = spec driven development

Vibe coding was based on prompt

now spec is new source code

Harness engineering

1. Guides gives feedback to agent

2. Sensors for self-correcting loop

3. Executable intent

4. Executable architecture

5. Continuous Governance

------------------------

3. Death of API Management

The speaker is founder of "bunny and cloud" a collaborative development tool for humans and AI

https://bunnyandcloud.com/

He explain reasons. 

1. A New practices is emerging: Context engineering

A right context at right time for agent, so it can reason and act reliably. 

2. Agents are taking all management attention

AI GWs and context contracts extend API Management into agent governance. 

3. API Management does not get intent

But agent needs intent

We need to hard code and orchestrate the agents for different workflows. Agent does not think about workflow. 

4. APIs are relegated to the execution layer

Reasoning layer by LLM (Probabilistic) 

context layer by MCP and RAG

Execution layer (deterministic) 

5. The GW shift

Kong is decoupling API GW and AI GW

Portkey AI GW pioneer is acquired by Palo Alto Networks

AI GW enforce policy to every call to LLM, MCP, RAG

6. The tokenomics is replacing APInomics

7. Context Management includes managing APIs

Microcontext. Not DB dump. Agent shall receive smallest truthful slice of bounded context. 

8. From API endpoint Management to capabilities management

Agent Registries

Instead of DX, now we need AX (Agent eXperience) 

Context Management is new API Management

pillar 1: Identity and intent context

pillar 2: Business and domain context

pillar 3: Knowledge and evidence context (RAG) 

pillar 4: Execution and feedback context (MCP)

AI GW examples

1 Portkey AI GW

2 Kong

3 Truefoundry (someone added from the audience) 

The attack surface is different for AI GW. 

------------------

36. Skills and MCP

https://www.skills.sh/

API related classes / certifications

https://apimasters.io/


Kyverno


 Kyverno Policy

- 2 types: Policy and ClusterPolicy

- Multiple Rules

- Match / Exclude

-- Match resources

kind is mandatory. names, namespaces, operations, selector are optional. 

Wildcards * supported in kinds, names, namespaces

All are AND condition. 

Any means OR condition

We can mention based on who created

exclude:

any:
- clusterRoles:
- cluster-admin
- subjects:
- kind: User
name: John

match AND exclude

exclude must be a subset of match

- Action

1. Validate ( allow / deny )

2. Mutate

3. Generate new K8s object

4. Verify image (Cosign/Sigstore)

- Enforce / Audit

* It uses JAMESPath

- Filter JSON

# Returns a list of container objects that match the condition
{{ request.object.spec.containers[?starts_with(image, 'nginx')] }}

validate:

message: "Nginx images are not allowed!"
deny:
conditions:
all:
# Filter the list. Use length() to count.
# If count > 0, it means a violation exists -> Block.
- key: "{{ request.object.spec.containers[?starts_with(image, 'nginx')] | length(@) }}"
operator: GreaterThan
value: 0

- pipe

Used to extract a specific field from a complex object into a flat list.

# Input: List of container objects
# Output: ["nginx:latest", "redis:alpine", "busybox"]
key: "{{ request.object.spec.containers[].image }}"

kyverno jp query -i object.json 'spec.containers[].name'

Combo (Filter + Pipe): “Get the containerPort of the container named ‘app’”

 request.object.spec.containers[?name == 'app'].ports[].containerPort

You want to find the names of all volumes that are of type emptyDir.

spec.volumes[?emptyDir != null].name

kyverno jp query -i object.json "spec.volumes[?emptyDir != null]"

- null handling

Risk: {{ request.object.metadata.labels.team }} (If null -> Error).

Safe: {{ request.object.metadata.labels.team || '' }} (If null -> treat as empty string).

- The length function

Example: “A Pod must not have more than 3 containers.”

validate:
deny:
conditions:
all:
- key: "{{ request.object.spec.containers | length(@) }}"
operator: GreaterThan
value: 3

- if / else 

- sum

- contain means exist : contains(request.object.metadata.labels, 'production')

advance mutate foreach

mutate:

foreach:
- list: "request.object.spec.containers"
patchStrategicMerge:
spec:
containers:
- name: "{{ element.name }}"
securityContext:
readOnlyRootFilesystem: true

* Aut-Gen: If rule for pod then automatically generate rules for Deployment, StatefulSet, DaemonSet, etc.

https://release-1-8-0.kyverno.io/docs/writing-policies/autogen/

Generate rule has synchronize flag

synchronize: true: Kyverno complete managed lifecycle

synchronize: false: Kyverno created for the first time then user can edit it manually without getting revert back like synchronize: true

* request.object is the incoming resource configuration (the new state) that is being submitted to Kubernetes API server

Documentation: https://kyverno.io/docs/policy-types/cluster-policy/variables/ 

  # -------------------------------------------------------------

  # ACTION: If Kyverno is dead, just let the request go through.

  # -------------------------------------------------------------

  failurePolicy: Ignore # or Fail


PolicyReport

PolicyReport stores the results of those rules.

The PolicyReport is essentially a “Health Check Report Card” for your Kubernetes resources.

Its main goal is Observability & Auditing.

It provides summary and then detail about all failures. For Example

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
name: polr-ns-default
namespace: default # It lives next to the Pod, not at the cluster level
labels:
app.kubernetes.io/managed-by: kyverno
summary:
pass: 0
fail: 1
warn: 0
error: 0
skip: 0
results:
- policy: require-labels # The name of the ClusterPolicy responsible
rule: check-for-team-label # The specific rule name
category: Best Practices
severity: medium
result: fail # The outcome (fail, pass, warn, error, skip)
message: "Validation error: label 'team' is required"
source: kyverno
resources: # The specific object that failed
- apiVersion: v1
kind: Pod
name: nginx
namespace: default
uid: a1b2c3d4-e5f6...

* Background scan never delete resource even with enforce mode

* For background scan following variable are not relevant

request.userInfo.*

request.operation

request.dryRun

serviceAccountName in admission context

Document: https://kyverno.io/docs/policy-reports/background/

* Cleanup policy delete pods

https://kyverno.io/docs/policy-types/cleanup-policy/

Kyverno CLI

* for given resource, policy is pass or fail

kyverno apply policy.yaml --resource pod.yaml

* test JAMESPath expression against JSON

kyverno jp query -i object.json 'metadata.labels'

* Check policy YAML is written correctly or not

kyverno validate policy.yaml

External Data Source

Purpose: It allows you to load data from outside into a variable before the rule logic (validate/mutate) runs.

In order to consume data from a ConfigMap in a rule, a context is required... The context data can then be referenced in the policy rule using JMESPath notation.

Kyverno supports 3 main data sources in context:

1. Kubernetes Resources (via API Call): Look up existing data in the cluster (e.g., ConfigMaps, Secrets, Services).

2. External APIs: Make an HTTP call to a service outside the cluster.

3. Image Registry: Fetch metadata about a container image (e.g., image size, architecture).

verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

https://main.kyverno.io/docs/policy-types/cluster-policy/external-data-sources/

Kyverno Mutate — JSON Patch

the standard patchStrategicMerge merges YAMLs together.

patchesJson6902 is a mutation method specific operations (RFC 6902 standard) to tell Kyverno exactly how to change the data.

Usage: 

* Removing a field (impossible with standard merge).

* Adding an item to a specific position in a list (arrays).

* Replacing a value entirely without merging.

It follows the JSON Patch format:

* op: The action (add, remove, replace).

* path: The location of the field (e.g., /metadata/labels/mytag).

* value: The data to put there.

Example: “Add a sidecar container”

https://main.kyverno.io/docs/policy-types/cluster-policy/mutate/#rfc-6902-jsonpatch

TTL Default of Certification in Kyverno:

https://pkg.go.dev/github.com/kyverno/kyverno/pkg/tls

const (

    CAValidityDuration = 365 * 24 * time.Hour      // 365 days

    TLSValidityDuration = 150 * 24 * time.Hour     // 150 days

    CertRenewalInterval = 12 * time.Hour           // 12 hours

)

https://www.udemy.com/course/complete-certified-kyverno-associate-kca-exam-prep/

https://medium.com/@kienlt.qn/prepare-for-the-kyverno-certified-associate-kca-exam-c144906f9bc2

Concurrent Policies Generation Number Default!

PolicyException CRD

CEL