AI
Why a single VM breaks under AI workloads
Putting everything on one large VM solves the deployment problem and creates five others. Ollama recommends 8 GB of RAM for a 7B parameter model and 32 GB for a 33B, and some models also need GPUs. Running the chat interface on the same oversized node fixes performance but blows your cost budget.
Adding a second model means manually updating host and port references. User accounts and chat history need to survive a UI restart, which host-local storage cannot guarantee. Credentials baked into a job spec become a liability every time a key rotates.
Nomad’s answers: node pools, service discovery, and autoscaling
Node pools bundle hardware constraints, GPU requirements, and placement rules into a single label. Set node_pool: large in a job spec and the scheduler handles the rest. Inference lands on the large private pool; Open Web UI lands on the small public one.
Nomad’s native service discovery registers each task under a name like ollama-backend and routes only to healthy instances, so adding a model needs no manual port updates. CSI volumes attach AWS EBS or similar cloud storage to whichever node runs the job. Nomad Variables deliver credentials at runtime under ACL policies, never baked into the spec.
Provisioning the cluster with Terraform
Terraform provisions the VPC, security groups, one Nomad server, and two EC2 clients in a single apply. Nomad exposes port 4646 for the UI with TLS on. A nomad_join tag auto-peers servers, so scaling to a three-node cluster later requires no manual configuration.
AWS credentials are exported as Terraform variables and written into Nomad Variables in the same apply. The Open Web UI job reads them at runtime to authenticate to S3, with no secrets baked into the spec. Each EC2 instance sets its own node pool membership and public/private metadata through a user-data script.
Deploying and connecting the AI services
The granite 3.3 job spec targets node_pool: large and runs two tasks: one starts the Ollama container, the other is a one-shot exec that curls IBM Granite 3.3 2B weights into Ollama. The task registers as ollama-backend, so any additional Ollama instance joins the same service group automatically.
Open Web UI targets the small public pool. Its OLLAMA_BASE_URLS variable uses a Nomad template stanza to enumerate every healthy ollama-backend endpoint. He walked through the Nomad jobs run granite live deploy (40:02). Admin creation runs through Nomad Actions, logged and ACL-gated, skipping SSH entirely.
Custom models and live cluster expansion
Nguyen used pirate model creation via nomad exec (52:33): entered the Ollama container, wrote a Modelfile with FROM granite3.3:2b and a pirate system prompt, then ran ollama create pirate-granite-3.3-2b. Open Web UI picked it up through existing service discovery.
Adding IBM Granite 4.0 required one Terraform variable change (aws_medium_private_client_count = 1) and a re-apply. The new EC2 client joined the cluster within two minutes. Granite 4.0 targets node_pool: medium and registers under the same ollama-backend service name, responding as fast as 3.3 on a large pool while using less memory.
Q&A
How much does it cost to run this compared to something like AWS Bedrock? Nomad adds no extra cost on top of the EC2 instances; you pay only for the AWS resources, so the small public node is priced as a T2 medium. ▶ 47:54
When you exec, is it better to target the task or the allocation? Allocation is more specific; because a task can have multiple allocations, targeting the allocation lets you reach the exact instance with the issue. ▶ 57:45
Why was the first model response slow? Ollama itself was initializing for the first time on that node; subsequent requests on the same instance were noticeably faster. ▶ 58:12
Notable Quotes
you shouldn’t bake uh credentials into Tu Nguyen · ▶ 7:21
this workshop has been my love letter Tu Nguyen · ▶ 1:05:23
the jobs are running on the right hardware Tu Nguyen · ▶ 1:05:43
Key Takeaways
- Node pools let you pin resource-heavy inference jobs to large private nodes and lightweight UIs to small public ones.
- Nomad native service discovery removes manual host/port updates when you add or replace model backends.
- Nomad variables inject secrets at runtime so credentials never appear in job specs or container images.