AI
Three Principles Behind TypeAgent
TypeAgent rests on three principles: distill models into logical structures for determinism and no hallucination, use structure to control information density so attention heads focus correctly, and build explicit search graphs for multi-agent collaboration.
Two of those principles run through every demo. In the Kevin Scott podcast memory search demo (04:39), the agent queries all past conversations to list books from a specific episode, then books a calendar meeting with details pulled from that history. The entity-to-action demo (07:01) retrieves a concert and dinner restaurant from conversation memory, makes a live reservation, and adds the event to a calendar.
Structured RAG Beats Classic RAG on Recall
Classic RAG embeds text chunks, matches them against a query by semantic similarity, and stuffs results into the prompt. Structured RAG extracts dense, structured information first, then retrieves by direct lookup rather than vector distance.
The difference shows in the numbers. Lucco loaded 25 episodes of Kevin Scott’s “Behind the Tech” podcast into both systems and asked each to list every book discussed.
“We are trying to create human like memory with superhuman precision and recall for agent conversations.”
Structured RAG returned all 63 books. Classic RAG returned 15 of them and one author.
Applying Structured RAG to Email and Images
Structured RAG generalizes to any text-bearing data source. Madan indexed two years of non-confidential Outlook email on a local laptop and queried it for summaries of specific technical threads. No embeddings, no cloud sync: key information extracted and returned in dense packages for the LLM to answer.
The photo library query demo (12:36) pushed further: 181 December phone photos indexed by joining EXIF metadata (location, date) with structured descriptions extracted from each image. The agent answered questions about countries visited, states represented, specific weeks, county names, and hiking trail names, including trails identified from a photo of a trailhead sign.
Agents That Remember Unfinished Work
When Lucco reopened TypeAgent after closing it, the agent sent an unprompted message: the Arizona trip montage was still pending review. No explicit reminder was set. The system indexed project state the same way it indexes conversations, so unfinished work surfaces automatically on the next session.
Asking “what’s the status of the Arizona trip montage?” returned the current state: created, but images not yet reviewed. One more command opened the montage canvas with the photos loaded. The same Structured RAG mechanism that recalls books from a podcast or streets from an email thread here tracks task state between sessions.
Notable Quotes
We are trying to create human like memory with superhuman precision and recall for agent conversations. Steven Lucco · ▶ 9:15
Structured Rag did substantially better because it works with dense structured information instead of relying only on semantic similarity. Steven Lucco · ▶ 10:08
distill models into logical structures Steven Lucco · ▶ 1:02
Key Takeaways
- Structured RAG recalled all 63 books from 25 podcast episodes; classic RAG recalled only 15.
- TypeAgent extracts dense structured representations from conversations, emails, and images without embedding-based retrieval.
- Agents built on logical structures can persist project state and surface unfinished work on re-open.