AI
Ray Is Flat by Default, and That Is a Problem
Ray abstracts distributed computing through high-level APIs and translates model requirements to Kubernetes via KubeRay’s three custom resources. But Ray ships with no security mechanisms enabled inside the cluster. Every node can reach every other node. Ray also runs arbitrary code, which means anyone who gets a foothold can abuse GPUs and send cloud bills to places nobody wants them. The head node doubles as a worker node, so overloading it can take down the entire cluster.
Namespace Isolation and Virtual Clusters
KubeRay can install into a dedicated namespace, dropping cluster-level permissions down to namespace-level. Multi-team setups can share one KubeRay instance across namespaces. For stricter isolation, vcluster creates virtual Kubernetes clusters inside a real GKE cluster. Each team gets its own control plane, custom resource definitions, and policies, while the host cluster still enforces shared rules. Kyverno sits at the admission layer and blocks or mutates resources that violate those rules, including the policy preventing worker processes from running on the head node.
Inbound Traffic: Private Ingress, IAP, and Model Armor
Three patterns cover inbound traffic. Cloud Functions and similar internal triggers stay inside the VPC using a private API behind an internal load balancer. External users, including developers hitting the Ray dashboard (which ships with no authorization), go through Identity-Aware Proxy, which gates access behind Google sign-in and IAM role checks. The inference gateway layer adds model armor, blocking prompt injection attempts such as “ignore all previous instructions” before they reach Ray jobs. Load-aware routing and LoRA adapter switching also live at this layer.
Outbound Identity and Mutual TLS
Outbound traffic from virtual clusters to data sources maps Google accounts to Kubernetes service accounts through Workload Identity, keeping IAM policies from overlapping across teams. For worker-to-worker encryption, ambient mesh beats sidecar Istio for smaller models because the HBONE tunnel removes the double-proxy overhead. Proxyless gRPC would be cleaner still, but Ray abstracts all gRPC internals, so the library cannot be patched. The practical recommendation is ambient mesh where possible, or GKE-managed Cloud Service Mesh with sidecar for teams that want a fully managed option.
Ray Tokens Are Not Ready for Production
Ray’s built-in token feature transmits tokens over unprotected HTTP headers. The tokens are not granular, they cover the whole cluster or nothing, and they must be regenerated on every cluster start. Enabling them on any part of the system forces tokens on all communication, including head-node-to-worker traffic. With mutual TLS already in place, adding Ray tokens delivers no meaningful security gain. The verdict: skip Ray tokens until the feature matures, and rely on the service mesh and proxy layers instead.
Compliance Monitoring with Prowler
Security posture drifts. Prowler, an open-source scanner, runs as a Kubernetes CronJob and reports on infrastructure misconfigurations, permission issues, and missing authorization on dashboards. It ships pre-packaged with role bindings and job definitions, so installing it takes minutes. Reports land in a dashboard and flag problems like Ray’s missing dashboard auth. Custom checks can be added on top of the built-in rule set. Running it continuously keeps the gap between current state and desired state visible before an attacker finds it first.
Notable Quotes
ray is flat. But what it means ray is flat in terms of security. And it means that by default there is no security mechanism enabled within the cluster. Everything has access to everything. Kateryna Hrytsaienko · ▶ 03:07
ray tokens are not granular. They just a passcards saying that you can move from one node to another. Kateryna Hrytsaienko · ▶ 21:41
if you use ray tokens they are transmitted via HTTP nonprotected headers. Kateryna Hrytsaienko · ▶ 22:00
security unfortunately is not constant and you need to put a lot of effort to just keep it running on the same level as it is. Kateryna Hrytsaienko · ▶ 23:16
Key Takeaways
- Ray ships with no internal security; every node reaches every other node by default.
- vcluster gives each ML team an isolated control plane without duplicating GPU resources.
- Ray tokens transmit over plain HTTP headers and should not be used in production today.
About the Speaker(s)
Kateryna Hrytsaienko is a Software Engineering Consultant at Valtech, co-lead and founder of GDG Kyiv, Women Techmakers Ambassador, and lecturer at KSE and Kyiv Polytechnic Institute. She works primarily on MLOps topics with a security focus and runs workshops on building first ML pipelines.