Don't Just 'Trust Me, Bro' with Your AI

Speakers

• Sebastian Nilsson

About this session

Remember back when we had to convince developers that unit tests, observability and automated testing were important? We are now in the same situation with AI solutions being built. In today's world of fast moving LLMs and generative AI, "trust me" isn't a strategy, it's a liability. AI outputs are inherently non‑deterministic, especially given the factors of different models, temperature settings, provider switches and more. So, “it worked on my AI” is not a sustainable approach. This is why you need AI-aware observability, not just classic logging. What really defines a working AI solution? At least, it has to be measurable, monitorable, and reliable. We need robust evaluation frameworks. We'll need things like automated evals, human-in-the-loop assessment, user-feedback, or even LLM-driven judging, to confirm expected behavior and detect regressions. You wouldn't ship a classic app without telemetry, so why treat AI differently? This talk equips developers with the mindset and tools to build AI systems that are intelligent and production-ready: traceable, testable, monitored, and trustworthy.