The Most Dangerous Number in Your Dashboard Is the One You Trust
Most organizations are not short on data. They are drowning in it. Dashboards refresh by the hour, completion rates climb toward one hundred percent, and satisfaction scores glow a reassuring green. And that, as psychometrician Dr. Anna Lissitz explains in this episode of the Kirkpatrick Podcast, is exactly the problem. The confidence leaders feel in their data is often the very thing standing between them and the truth.
Vanessa opens the conversation with a suspicion many of us share but rarely say out loud: that organizational confidence in data is frequently misplaced. Leaders are confident they have data — there is plenty of it — but far less certain whether it is the right data, or whether it tells an accurate story. Anna, who wrote the chapter on triangulation and validation in Building a Culture of Evaluation, spends her working life answering exactly that question. Her message is direct: a single metric, measured one way, at one moment, from one perspective, cannot carry the weight organizations put on it.
Valid Versus Available
The distinction Anna draws — and the one we return to constantly in our work — is between data that is valid and data that is merely available. Valid data measures what you actually intend to measure. Available data is whatever is easiest to collect. Completion rates and end-of-course satisfaction surveys dominate reporting not because they tell us the most, but because they ask the least of us. They are convenient, and conveniently, they tend to confirm what we already hoped was true: everyone finished, everyone smiled, so the program must have worked.
This is where the Kirkpatrick Model does its sharpest work. The Four Levels® — Level 1: Reaction, Level 2: Learning, Level 3: Behavior, and Level 4: Results — exist precisely so that no one mistakes a high satisfaction score for impact. A glowing Level 1 reaction tells you people enjoyed the experience. It says nothing about whether behavior changed on the job, or whether the business outcome the program was built to move actually moved. As Anna puts it, you have to take that extra step and ask: did anything actually change?
When Confidence Becomes Confirmation Bias
Anna offers a reframe worth sitting with. The data an organization feels most confident about, she suggests, is sometimes confident precisely because it is telling people what they wanted to hear. Truly valuable data does the opposite — it surfaces something you did not know to look for. Vanessa names the trap plainly: you can make data say almost anything depending on how you slice it, what you collect, and how you word the question. Confidence, in other words, is not the same as accuracy.
Vanessa tells a story from her work that makes the point concrete. Across countless projects, teams that pushed to skip the analysis and evaluation steps rarely came out ahead. When organizations build their evaluation plan before they finalize the program — the approach taught in Kirkpatrick programs — measurement is designed into the work, not bolted on at the end.
Triangulation Is Not Academic — It Is How You Stop Guessing
Triangulation can sound like a word built to intimidate. At its heart, it simply means gathering evidence from multiple sources, methods, points in time, and perspectives — and then asking whether those views agree. Data, she reminds us, is not only numbers; qualitative insight is data too.
She makes it vivid with her own world. As a professor, Anna hires and re-staffs the adjuncts who teach her classes. Her first impressions — polished, confident, great with people — are often contradicted once she triangulates: dropping into the virtual classroom, watching how quickly assignments get graded, reading student evaluations against what she observes directly. Course evaluations alone, she notes, are notoriously unreliable — a student who earned an A tends to rate the class highly. Only by combining sources does a trustworthy picture emerge.
Low Response Rates, Leading Indicators, and the Performance Environment
One of the episode’s most useful reframes concerns the data that looks like no data at all. When managers do not complete a Level 3 survey, that silence is itself a finding — a leading indicator. Perhaps they lack the time, cannot observe the behavior closely enough, or never trusted the process to begin with. Each of those points back to the Performance Environment: the conditions, systems, and psychological safety that determine whether learning ever becomes performance. A missing response is not a gap in your spreadsheet — it is the spreadsheet trying to tell you something.
Where AI Helps — And Where Judgment Still Wins
Anna is genuinely excited about AI’s role in evaluation, particularly for analyzing the open-ended, qualitative responses teams used to dread coding by hand. But she is equally clear-eyed: an AI tool is only as good as what you put into it. A biased prompt produces a biased answer, just as a poorly worded survey question produces misleading results. The same foundational principles — careful instruments, sound judgment, and triangulation — apply to AI outputs as to any other source. The tools are advancing quickly. The need for human judgment is not going anywhere.
Better Measurement Is a Leadership Decision
Underneath all of it sits culture. A culture of evaluation is built on courage and trust: the courage to collect data that might challenge what leaders believe, and the trust that the findings will be used to improve rather than to punish. When people watch their feedback disappear into a void, they stop offering it. When they see evaluation drive real change, they lean in. That shift — from reporting data to actually using it — starts at the top, and it is ultimately about Return on Expectations: delivering the outcomes stakeholders care about most, and being able to prove it.
This is why we keep returning to a single conviction: evaluation is the unlock for the future of organizational health and performance. Not the dashboard. Not the volume of data. The discipline of measuring the right things, in the right ways, from enough angles to be sure. Anna leaves us with the reassurance that statistics and triangulation only sound scary — done well, they take guesswork off your plate rather than adding to it.
Put the Tools in Your Own Hands
Knowing the difference between valid and available data is the first step. Building the instruments that produce it is the next — and it is exactly what the Kirkpatrick® Builder Certification teaches. In this hands-on, live-online program, participants design a full set of evaluation instruments across all four levels and assemble them into a single defensible chain of evidence for one of their own initiatives. As the program puts it: you don’t watch someone evaluate, you build.
Go deeper. Learn more and join the founding Builder cohort at kirkpatrickpartners.com/event/kirkpatrick-builder-certification, and read Anna’s chapter in Building a Culture of Evaluation: The New Kirkpatrick Model for Performance and Innovation.
