The lab notebook
Field notes on AI engineering
824 articles · page 17 of 28
The Economics of AI Agents: When Automation Actually Pays Off
A grounded look at what AI agents actually cost to build and run, and the honest math for deciding when automating a workflow pays off.
Embedding Models Compared: OpenAI vs Cohere vs Open-Source in 2026
A practical, no-fluff comparison of OpenAI, Cohere, and open-source embedding models for real RAG systems in 2026.
Enterprise MCP Adoption: Rolling Out Tool Servers Across Teams
A practical playbook for rolling out Model Context Protocol servers across engineering teams without creating a governance mess.
Eval-Driven Development: Writing Evals Before You Write Prompts
Why the best prompt engineers write test cases before they write a single prompt, and how to build that habit into your workflow.
Evaluating Agent Task Completion: Did It Actually Finish the Job?
Your agent says "done" — but a green checkmark isn't proof of work, and here's how to actually verify task completion before it ships.
Evaluating Bias and Fairness in LLM Outputs
A practical guide to detecting, measuring, and mitigating bias in LLM outputs using counterfactual testing, statistical parity checks, and automated evals.
Evaluating Chatbot Quality: Helpfulness, Tone and Safety
A practical framework for scoring chatbot helpfulness, tone, and safety with rubrics, code, and real failure examples.
Evaluating Code Generation: Beyond Pass@k
Pass@k tells you if code runs once, not if it's safe, maintainable, or correct across edge cases — here's what to measure instead.
Evaluating Fine-Tuned Models Against Base Models
A practical framework for proving your fine-tuned model actually beats the base model, with metrics, code, and pitfalls to avoid.
Evaluating Hallucination Rate: Detection Techniques Compared
A practical comparison of hallucination detection methods, from NLI-based scoring to LLM-as-a-Judge, with code you can run today.
Evaluating Multi-Agent Systems: Attributing Failure to the Right Agent
Multi-agent pipelines fail in shared, hard-to-trace ways — here's how to pinpoint which agent actually broke the chain.
Evaluating Multi-Turn Conversations: Beyond Single-Response Metrics
Single-turn scores miss the failures that matter in real chat products; here's how to evaluate multi-turn conversations properly.
Evaluating RAG vs Evaluating Agents: Different Metrics, Different Rules
RAG and agent evaluation look similar on the surface but break in completely different places, and using one playbook for both will hide your real bugs.
Evaluating Retrieval-Free vs RAG Answers Side by Side
A practical framework for building side-by-side evals that tell you when RAG actually beats a retrieval-free LLM, and when it's dead weight.
Evaluating Structured Output Reliability: JSON Mode Failure Rates
JSON mode doesn't mean correct output — here's how to actually measure schema conformance, field accuracy, and failure rates in production.
Evaluating Summarization Quality: Metrics That Actually Work
A practical guide to evaluating LLM summarization, from ROUGE's blind spots to LLM-as-a-Judge rubrics you can actually trust.
Evaluating Tool-Calling Accuracy: Did the Agent Pick the Right Tool?
A practical guide to measuring whether your LLM agent picked the right tool, with the right arguments, at the right moment.
Evaluating Voice and Audio AI Agents: Unique Challenges
Voice agents fail in ways text agents never do—latency, ASR errors, and prosody all need their own evaluation harness.
The Future of AI Coding Agents: Where Codex and Claude Code Are Headed
A grounded look at where AI coding agents like Codex and Claude Code are actually headed, past the hype.
Graph RAG Explained: When Knowledge Graphs Beat Vector Search
Graph RAG fixes the multi-hop reasoning gaps in vector search by letting your retrieval layer traverse relationships, not just embeddings.
The Hermes Agent Checkpoint System: Learning by Milestones
Why checkpoints, not chapters, are the real unit of progress when you're learning to build production agents.
What Students Build in 30 Days of Hermes Agent
A tour of the real, working AI agents students ship during 30 Days of Hermes Agent, from research bots to inbox triagers to deployment pipelines.
Hermes Agent Week 1 Deep Dive: Building the Walking Skeleton
A concrete walkthrough of Hermes Agent Week 1 — the agent loop, one tool call, a minimal prompt, and the test harness that keeps you honest.
Hermes Agent Week 2 Deep Dive: Planners, Executors and Critics
Inside week 2 of 30 Days of Hermes Agent, where students split a single agent into planner, executor, and critic roles.
Hermes Agent Week 3 Deep Dive: Memory and Context Engineering
Week 3 of Hermes Agent covers short and long-term memory, context window budgeting, and retrieval — the systems that make agents feel coherent.
Hermes Agent Week 4 Deep Dive: Shipping to Production
Inside Week 4 of 30 Days of Hermes Agent, where students deploy, monitor, guard, and demo a real production-grade AI agent.
The Hermes Naming Story: Messenger Gods and Agent Architecture
Why we named our flagship agent course after the Greek messenger god, and what Hermes mythology actually teaches about routing, orchestration, and message passing.
How Long Does It Really Take to Learn to Build AI Agents?
A realistic, no-fluff breakdown of how long it takes different backgrounds to learn AI agent development, from beginner to production-ready.
Human Evaluation vs LLM-as-a-Judge: When You Still Need People
A practical guide to knowing when LLM-as-a-Judge is enough and when your eval pipeline still needs human reviewers in the loop.
Human-in-the-Loop Agent Design Patterns
A practical guide to human-in-the-loop agent patterns, including approval gates, confidence routing, and escalation queues you can ship today.