ML Interfacing pipeline
NIM is off the self interfacing framwork
NIM is about decide which model to use, based on number of GPU? Which GPU? Performance criteria (throughput v/s latency)? floating point library. NIM can autodetect hardware
GenAI with RAG has many services.
NIM operator to deploy RAG application CR: 1. NIM Cache (PVC) 2. NIM service 3. NIM pipeline all service can increase together.
NIM monitoring and autoscaling: Prometheus
1. Utilization of hardware
2. inter token latency
3. first token time generation.
4. request per second
Monitoring of NIM
2 seconds, 15 chat user etc are input for SLA.
NIM monitoring operator choose metrics from many metrics exposed by NIM
Autoscaling
In the sample chat application : milvus DB is needed. RAG is frontend service
ML Deployment
Canary has mode deployment time, because more iteration. 10% increase means 10 iteration.
Blue Green has less.
Fine tune Max Surge, Max unavailable etc.
Traffic Mirroring is for ML load also.
image compression, local registry etc needed because model size is big.
Service Continuity in 5G
There is no inter cluster redundancy by K8s. We need to use proprietary solution OR cloud.
In telecom a component is connected with multiple. E.g vCU with vDU, EMS, 5GC
Sync Driver
GRAF framework with AI, Management Data analytics function(MDAF), policy driven
A1 interface is better than MDAF. As MDAF is at core network. it will add latency.
Nephio : open source project : LF + Google
Automate LCM of cloud infra and NF. Intent based declarative approach
LinkedIN : saurabhswaraj
GitOps based approach
1. N/w driver is GR aware
2. DB drive sync 2 MySQL
YAML PV and DB table synch
Each cluster has GRAPH controller
We can have several other use cases also
GRAPH is framework. We can develop our own driver.
GRAPH f/w = Redundancy Manager at Orchestrator + GRAPH controller at each cluster
Based on CRD, different driver will be deployed at each K8s cluster.
Still GRAPH is not open source. It is in process for open source. At present it is in R&D stage.
GRAPH can work with many orchestrator including Nephio
All DB has Replication Manager. Why do we need DB drive? DB drive is not novelty. Our novelty is framework.
Policy based, when failed comes up again, what will happen.
Keptn
Kept has Two Operators: App LCM and for Metrics
Day1 and Day2 operation
Integrated with Argo
Metrics based scaling: HPA and Scala
works for outside K8s application
It works with DORA metrics
Multistage application delivery.
1.5 million+ TPS
Each pod will have local bucket and then implement leaky bucket algorithm.
Start bucket size with big number
Learning from rate limiting journey
Service with smaller threshold has higher precision
higher threshold has upto 10% error rate
pod has its own swimlane
-------------------------------------------------------------------------
For inter+intra rate limit, need to synch API-GW and serviceMesh.
Unified config management
Naavik knows, where service resides (Kubecon Paris)
Canary release pattern : First canary call, the service registry will read from DB and put it in cache. For remaining, no need to access DB. Response from cache. Canary pod can put value in S3 bucket, for further optimization, and notify all pods to read form s3 bucket
