The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Key Concepts

• Coherency: A system property of LLMs characterized by relevance, consistency, stability, and emergence.
• AI Decision Loop/Nudge and Iterate Framework: A methodology for interacting with LLMs involving framing, generating, judging, and iterating.
• Latent Space: The high-dimensional space where AI models operate, with concepts represented as patterns.
• Superposition: The ability of neural networks to represent multiple concepts with fewer parameters.
• Force Vector: The directional force exerted by a prompt in the latent space, guiding pattern alignment.
• Hallucinations: An indicator of coherence, where the model fills in gaps based on internal logic.
• Retrieval Augmented Generation (RAG): A technique to ground LLM outputs with factual anchors.
• Three-Layer Model: A model consisting of the latent space, execution layer (tools and APIs), and conversational interface.

Initial Disappointment and the Shift with GPT-4

The speaker, Travis Fry Singinger, initially felt disappointed with GPT-3.5 due to its brittleness, surface-level fluency collapsing at edge cases, prompt sensitivity, and context limits. However, GPT-4 brought a significant shift, creating an uncanny feeling of effortless alignment and surpassing previous AI experiences. This led to the question of what fundamentally changed between these models.

Experiments and Projects

Live Streamed Paired Programming (Vibe Coding)

Travis started experimenting with GPT models through live-streamed paired programming, initially finding it difficult to extract usable code. This evolved into prototypes for AI pair programming, which he still uses in tools like Cursor and Windsurf.

WebCAT Utility

To address the lack of internet access in early GPT-4, Travis developed WebCAT, a Python Azure function that scrapes web pages and provides content to the model. This enabled deeper collaborative experiments.

AIBuddy.software Blog

Travis built a blog, AIBuddy.software, in collaboration with AI, using Ghost as the platform. The AI assisted in template creation and content generation, demonstrating the potential for collaborative content creation.

Mr. Fluff's Reign of Tiny Terror (Feline Metal Album)

Travis created a concept album, "Mr. Fluff's Reign of Tiny Terror," using AI for lyrics, music composition, and image editing. Despite being a niche project (cat metal, AI-generated), it garnered over 3,000 views and positive feedback, indicating the ability to create valuable content with AI assistance.

AI Decision Loop and Nudge and Iterate Framework

Travis explored decision intelligence and pairing behaviors to understand reliable AI interactions. He developed an analysis tool to examine his chat history, leading to the formulation of the AI decision loop, later simplified into the "Nudge and Iterate" framework.

AI Decision Loop (Academic Version)

1. Frame: Define the problem and context (prompt engineering).
2. Generate: Produce outputs based on the prompt.
3. Judge: Evaluate the quality and fit of the outputs.
4. Validate: Verify outputs against external requirements.
5. Iterate: Refine the prompt and repeat the process.

Nudge and Iterate Framework (Simplified Version)

Frame the problem, generate responses, evaluate the responses, and continuously prompt and nudge the model for more reliable outputs.

Coherency Theory

Travis proposes that LLMs operate based on "coherency" rather than intelligence. Coherency is a system property, not a cognitive one, and is the infrastructure that thought navigates.

Properties of Coherency

1. Relevance: Output feels topical, connected, and purposeful.
2. Consistency: Maintains a singular tone, terminology, and structure.
3. Stability: Withstands pressure and interrogation without collapsing.
4. Emergence: Exhibits unexpected capabilities through coherent pattern alignment (e.g., diagnosing swine disease in GPT-4o).

Coherency in the Machine

Superposition and Latent Space

LLMs use superposition to represent complex ideas with fewer parameters. Concepts are not stored in single neurons but distributed across multiple neurons. The latent space is a high-dimensional space where concepts are represented as patterns.

Prompts as Force Vectors

Prompts act as force vectors in the latent space, guiding the AI to align patterns. The concept of a cat or a dog wears grooves in that latent space, and the AI follows that space.

Execution and Inference

When prompted, the model activates relevant conceptual clouds in the latent space (e.g., storytelling, freedom). These specialized patterns emerge during training and activate when the context calls for them. The model merges these concepts to create new ideas based on the activated subnetworks.

Essence Reconstruction

The value of LLMs lies in their ability to recreate the essence of an idea and create something new. Even when they hallucinate, the output feels correct because they are creating a compelling pattern.

Building for Coherence, Not Intelligence

Hallucinations as an Indicator

Hallucinations are an indicator of coherence, showing the model's attempt to complete the pattern. Retrieval Augmented Generation (RAG) can be used to provide factual anchors and steer the generation.

Three-Layer Model

1. Latent Space: Internal model structure, concepts, weights, activations.
2. Execution Layer: Tools, APIs, and retrieval mechanisms to bring extra context.
3. Conversational Interface: Where intent passes from the user to the machine.

Design Principles

• Treat prompting as interfaces, not one-off interactions.
• Use RAG to ground coherency with factual anchors.
• Design for emergence, not control (avoid deterministic expectations).
• Use the frame, generate, judge, iterate loop.
• Avoid fragile chains and reinforce context at each point.
• Watch for breakdowns in tone, structure, and flow.

Conclusion

LLMs are high-dimensional mirrors that resonate with users through structure, not thought. They are coherent, not intelligent, and their superpower lies in the collaborative dance between human intent and machine resonance. The key is to design for structured resonance rather than chasing artificial intelligence.