VALIDATOR-TEST-KIT-1005.md
Rama main · réplica de lectura del repositorio original.
# Test kit — #1005 on a quota-controlled cluster
For the validator running this on a real quota-enforcing namespace (OpenShift dev cluster included).
Branch `1005-runtime-resources`, based on `616778fe`.
**What you are proving:** the two Pods the runtime creates *dynamically* — the meeting bot and the
agent worker — now declare CPU and memory requests **and** limits, so your namespace admits them
instead of rejecting them. Nothing else about the release changes.
**What has already been proven, and where** — so you can skip re-doing it:
| Already green | Where |
|---|---|
| Both profiles admitted under a `ResourceQuota` requiring all four values, sized independently | k3d v1.35.5, namespace `vexa-quota` — see `OBSERVATION-BUNDLE-1005.md` A1/A2 |
| Image, command, env, labels, tolerations, nodeSelector, workspace mounts all survive | same, A3 |
| Removing one dimension reds admission; restoring it greens the same spawn | same, A4 |
| Offline runtime suite, Helm render, Compose stack, all 34 repo gates | same, A- |
**What only your cluster can prove:** OpenShift SCC behaviour on the spawned Pods, your real quota
headroom, and whether the real bot image lives inside the memory ceiling you set. Read
[the honest caveats](#honest-caveats) before you start — one of them is likely to bite.
---
## 1 · Install
Two values blocks matter. The first is the new one.
```yaml
# values-quota.yaml
runtime:
backend: k8s
# Sizing for the Pods the runtime SPAWNS. NOT runtime.resources (which sizes the runtime
# Deployment itself). One value per dimension sets BOTH the request and the limit.
workloadResources:
meetingBot:
cpu: 2 # → requests/limits cpu: 2000m
memoryMb: 4096 # → requests/limits memory: 4096Mi
agentWorker:
cpu: 0.5 # → requests/limits cpu: 500m
memoryMb: 1024 # → requests/limits memory: 1024Mi
```
```bash
helm upgrade --install vexa deploy/helm/charts/vexa \
-n <your-namespace> --create-namespace \
-f values-quota.yaml \
--set global.imageTag=<TAG> \
--set secrets.adminApiToken=$ADMIN_TOKEN \
--set secrets.internalApiSecret=$INTERNAL_API_SECRET
```
Confirm the sizing reached the runtime before you spawn anything:
```bash
kubectl -n <your-namespace> get deploy vexa-runtime -o json \
| jq '.spec.template.spec.containers[0].env[]
| select(.name|test("^RUNTIME_(BOT|AGENT_WORKER)_"))'
```
Expected — four entries, non-empty:
```json
{"name":"RUNTIME_BOT_CPU","value":"2"}
{"name":"RUNTIME_BOT_MEMORY_MB","value":"4096"}
{"name":"RUNTIME_AGENT_WORKER_CPU","value":"0.5"}
{"name":"RUNTIME_AGENT_WORKER_MEMORY_MB","value":"1024"}
```
If they are empty strings, the chart values did not land — the spawned Pods will declare nothing and
your namespace will reject them exactly as before. Fix this before going further.
> **Sizing note, not a formality.** `memoryMb` is a **hard ceiling**: a workload exceeding it is
> OOM-killed. Before this change nothing was enforced, so bots ran unbounded and we have no measured
> figure to hand you. `4096` above is deliberately more generous than the chart default (`2048`).
> Start high, watch actual usage, then tighten — do not start at the default and discover the ceiling
> as a dropped meeting.
---
## 2 · Spawn both profiles and watch
Trigger one meeting bot (your normal `POST /bots` path) and one agent dispatch. Then, for each:
```bash
kubectl -n <your-namespace> get pods -l runtime.managed=true
kubectl -n <your-namespace> get pod <pod> -o json | jq '{
qos: .status.qosClass,
phase: .status.phase,
resources: .spec.containers[0].resources,
image: .spec.containers[0].image,
command: .spec.containers[0].command,
envCount: (.spec.containers[0].env|length),
labels: .metadata.labels
}'
```
### Expected observations
| Pod | Name shape | Expect |
|---|---|---|
| meeting bot | `vexa-<meeting-workload-id>` | `resources.requests` **and** `.limits` = `{cpu: 2, memory: 4096Mi}`; `qosClass: Guaranteed`; `phase: Running`; **no** `command` (the bot image's own ENTRYPOINT boots it); `labels` carry `runtime.managed=true` + `runtime.workload_id` |
| agent worker | `vexa-agent-…` | `resources` = `{cpu: 500m, memory: 1Gi}` — **different from the bot**; `qosClass: Guaranteed`; `command: ["python","-m","worker"]` |
`2000m`/`500m` display as `2`/`500m` — the API server normalizes; either spelling is the same value.
Quota consumption should move:
```bash
kubectl -n <your-namespace> describe resourcequota
```
Both classes should appear in `requests.cpu` / `limits.memory` usage while running, and drop back when
the workloads finish.
### The one-line red control (optional, 30 seconds)
Proves the quota is really enforcing and that the values came from Vexa, not from a LimitRange:
```bash
kubectl -n <your-namespace> run quota-probe --image=busybox:1.36 --restart=Never -- sleep 30
```
Expected: **rejected** — `must specify limits.cpu … requests.memory …`. If this *succeeds*, a
LimitRange is defaulting your namespace, and per the issue's closing rule a green run there does not
prove Vexa emitted the resources. Re-check with the `jq` above: the values must equal what you set in
`values-quota.yaml`, not the LimitRange defaults.
---
## 3 · Capture this on failure
Please attach all of it — partial reports cost a round trip.
```bash
NS=<your-namespace>; POD=<the failing pod>
# 1. Did admission reject the spawn? The runtime logs the API server's own reason verbatim.
kubectl -n $NS logs deploy/vexa-runtime --tail=200 | grep -iE "kubectl create|forbidden|StartFailed"
# 2. Namespace events — quota denials, SCC denials, scheduling failures all land here
kubectl -n $NS get events --sort-by=.lastTimestamp | tail -40
# 3. The Pod as the API server accepted it (or the whole object if it never got created)
kubectl -n $NS get pod $POD -o yaml
# 4. Why it is not running
kubectl -n $NS describe pod $POD
# 5. What the workload itself said
kubectl -n $NS logs $POD --previous --tail=200 || kubectl -n $NS logs $POD --tail=200
# 6. Quota state at the time
kubectl -n $NS describe resourcequota
kubectl -n $NS get limitrange -o yaml # if any — see the LimitRange note above
```
Reading the three failure shapes:
| Symptom | Almost certainly |
|---|---|
| No Pod at all; runtime log says `forbidden: failed quota: … must specify …` | the sizing env did not reach the runtime, or one dimension is empty — recheck §1 |
| No Pod; message mentions `unable to validate against any security context constraint` | **SCC**, not quota — see caveats below |
| Pod exists, `Pending`, events say `Insufficient cpu/memory` | your quota or node headroom is smaller than the size you set — lower `cpu`/`memoryMb` or raise the quota |
| Pod runs then dies with `OOMKilled` (exit 137) | `memoryMb` is too low for the real workload — raise it |
---
## Honest caveats
**1 · Spawned-Pod SCC on OpenShift is untested — this is the most likely surprise.**
You proved the *control plane* satisfies your SCC. The bot and agent Pods are **different objects**:
bare Pods created by the runtime at spawn time, not children of any Deployment, so they inherit
nothing from the control-plane pod templates. A `restricted-v2` SCC may additionally require
`runAsNonRoot`, a dropped capability set, or a `seccompProfile` on those Pods — **this change adds
none of those**, and no OpenShift cluster was available to test it. If §2 fails with an SCC message
rather than a quota message, that is a *separate, known-open* gap: capture the event text and the SCC
name (`kubectl -n $NS get pod $POD -o jsonpath='{.metadata.annotations.openshift\.io/scc}'`) and send
it — it is a follow-up issue, not a defect in this one.
**2 · The memory ceiling is a guess until you measure it.** No live meeting has run under an enforced
limit. Set generously, observe (`kubectl top pod`), then tighten.
**3 · Enforcement is Kubernetes-only.** The docker and process backends accept the same resource
intent and do not act on it. Compose and Lite deployments are unchanged; no parity is claimed.
**4 · Guaranteed QoS changes scheduling density.** Request equals limit by design (runtime.v1 carries
one value per dimension). Reserved capacity is now real — a node that used to fit N unbounded bots
fits `floor(allocatable / size)` of them.
**5 · What the k3d validation substituted.** The live legs in the bundle ran `busybox:1.36` in place
of the bot and worker images, so the *declaration and admission* path is proven against a real API
server but the real images were never run inside the limits. Your run is the first that does that —
which is exactly why it is the one that closes the issue.
---
## Appendix · the exact scripts used for the recorded evidence
Re-runnable against any cluster; drop them in a scratch dir.
<details><summary><code>quota-ns.yaml</code></summary>
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: vexa-quota
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: require-requests-and-limits
namespace: vexa-quota
spec:
hard:
requests.cpu: "4"
requests.memory: 4Gi
limits.cpu: "4"
limits.memory: 4Gi
pods: "10"
```
</details>
<details><summary><code>quota-validate.py</code> — spawns both profiles through the real kernel</summary>
Run from `core/runtime` as
`PYTHONPATH=src uv run python quota-validate.py {green|red-unsized|red-no-memory}`.
```python
import json, os, subprocess, sys, time
NS = "vexa-quota"
def kubectl(*args, stdin=None, check=True):
r = subprocess.run(["kubectl", *args], capture_output=True, text=True, input=stdin)
if check and r.returncode != 0:
raise SystemExit(f"kubectl {' '.join(args)} FAILED:\n{r.stderr}")
return r
def pod_json(name):
return json.loads(kubectl("get", "pod", name, "-n", NS, "-o", "json").stdout)
def spawn(profile_name, workload_id):
from runtime_kernel import Runtime
from runtime_kernel.k8s_backend import K8sBackend
from runtime_kernel.models import WorkloadSpec
from runtime_kernel.profiles import apply_command_overrides, default_registry
rt = Runtime(backend=K8sBackend(namespace=NS),
profiles=apply_command_overrides(default_registry()), grace_sec=5.0)
return rt, rt.create(WorkloadSpec(workloadId=workload_id, profile=profile_name,
env={"VEXA_X": "y"}))
def main():
os.environ["BROWSER_IMAGE"] = os.environ["AGENT_WORKER_IMAGE"] = "busybox:1.36"
os.environ["AGENT_IMAGE"] = "busybox:1.36"
os.environ["AGENT_WORKER_COMMAND"] = os.environ["BOT_COMMAND"] = "sleep 120"
mode = sys.argv[1] if len(sys.argv) > 1 else "green"
if mode == "green":
os.environ["RUNTIME_BOT_CPU"] = "1"; os.environ["RUNTIME_BOT_MEMORY_MB"] = "512"
os.environ["RUNTIME_AGENT_WORKER_CPU"] = "0.25"; os.environ["RUNTIME_AGENT_WORKER_MEMORY_MB"] = "256"
elif mode == "red-no-memory":
os.environ["RUNTIME_BOT_CPU"] = "1"; os.environ.pop("RUNTIME_BOT_MEMORY_MB", None)
os.environ["RUNTIME_AGENT_WORKER_CPU"] = "0.25"; os.environ.pop("RUNTIME_AGENT_WORKER_MEMORY_MB", None)
elif mode == "red-unsized":
for k in ("RUNTIME_BOT_CPU", "RUNTIME_BOT_MEMORY_MB",
"RUNTIME_AGENT_WORKER_CPU", "RUNTIME_AGENT_WORKER_MEMORY_MB"):
os.environ.pop(k, None)
print(f"=== MODE: {mode} ===")
for profile, wid in (("meeting-bot", f"mtg-{mode}"), ("agent", f"agent-{mode}")):
kubectl("delete", "pod", f"vexa-{wid}", "-n", NS, "--ignore-not-found",
"--grace-period=0", "--force", check=False)
print(f"\n--- profile={profile} workloadId={wid} ---")
try:
rt, status = spawn(profile, wid)
except Exception as exc:
print(f"SPAWN REJECTED: {type(exc).__name__}: {str(exc)[:600]}")
continue
print(f"kernel state: {status.state.value}")
c = pod_json(f"vexa-{wid}")["spec"]["containers"][0]
print("ADMITTED. container.resources =", json.dumps(c.get("resources", {})))
print("image =", c["image"], "| command =", c.get("command"))
deadline, phase = time.time() + 60, ""
while time.time() < deadline:
phase = pod_json(f"vexa-{wid}")["status"]["phase"]
if phase == "Running":
break
time.sleep(2)
print("phase =", phase, "| qosClass =", pod_json(f"vexa-{wid}")["status"].get("qosClass"))
rt.stop(wid); rt.destroy(wid)
print("terminal:", rt.get(wid).state.value)
if __name__ == "__main__":
main()
```
</details>
<details><summary><code>a3-live.py</code> — proves no generated field is erased</summary>
Creates a small PVC, spawns one Pod carrying resources + tolerations + nodeSelector + a two-mount
workspace set, and prints what the API server accepted. Full source in the session scratch; the
essential call is:
```python
K8sBackend(namespace=NS).start(
wid, Runnable(image="busybox:1.36", command=["sleep", "120"]), env,
Resources(cpu=0.5, memoryMb=256),
)
```
with `RUNTIME_K8S_TOLERATIONS` / `RUNTIME_K8S_NODE_SELECTOR` set in the process env and
`VEXA_WORKSPACE_MOUNT_SOURCE` / `_TARGET` / `VEXA_MOUNTS` in the workload env. Then read the Pod back
and assert `image`, `command`, `env`, `labels`, `restartPolicy`, `tolerations`, `nodeSelector`,
`volumes`, `volumeMounts` and `resources` are all present on the accepted object.
</details>