The Vicious Loop of Growing Context

▶ Watch (0:16)

Alex runs on Arize traces. Each trace contains input, prompts, metadata, and user interactions. Spans grow. Context limits hit. Alex fails. The system analyzing data was constrained by the data. That loop repeated. Context engineering is choosing what the model sees, not just fitting tokens. Bad context gives bad answers. The problem became a product and UX issue, not just an engineering one.

Smart Truncation with Memory

▶ Watch (5:17)

Naive truncation took the first 100 characters and dropped the rest. Follow-ups looked like new conversations. Over truncation broke reasoning. Summarization seemed obvious but was too inconsistent. The LLM decided what to keep with no control. The working solution: keep the first 100 characters and the last 100, store the middle in memory. The agent can retrieve any important tool call or message. Alex has not needed a change to this strategy in months.

Offloading Heavy Tasks to Sub-Agents

▶ Watch (9:23)

Long sessions cause failures that appear late. Users stay in one chat. Conversations grow. The solution: sub-agents. The main conversation keeps only chat and light context. Heavy data lives in sub-agents. The main agent delegates heavy data tasks. After the sub-agent returns a result, the main conversation continues. This pattern keeps the main context small and prevents context overload.

Ongoing Challenges: Long-Term Memory and Context Metrics

▶ Watch (11:19)

Huge context still breaks things. Provider limits get hit. Conversations grew from under 10 turns to 20 plus. Long-term memory does not exist yet. Alex cannot reference previous chats. Context selection is still a heuristic: first 100, last 100. There is no principled context budget or clear quality metrics. Evals measure whether context was right, but a more sophisticated approach is needed.

Q&A

How are you handling cache invalidation like Claude Code? Focusing on long-term memory, not cache. Alex uses database IDs and a tool to access earlier messages. ▶ Watch (14:59)

Notable Quotes

Agents don’t fail because of prompts, they fail because of context. Sally-Ann Delucia · ▶ Watch (14:13)

Over truncation had broke the reasoning. Sally-Ann Delucia · ▶ Watch (6:09)

Summarization was too inconsistent. Sally-Ann Delucia · ▶ Watch (6:28)

Context management is iterative. Sally-Ann Delucia · ▶ Watch (13:47)

Key Takeaways

  • Context engineering matters more than prompt engineering for agent success.
  • Smart truncation with memory outperforms naive truncation and summarization.
  • Sub-agents keep the main conversation light by offloading heavy data tasks.