A Demo Is Engineered To Work. Your Workflow Is Not.
Seven signals that a vendor has not deployed at your scale
I have been pitched by hundreds of AI vendors. I have also been the vendor in the room, and I have rebuilt the platform that 22,000 life insurance agents run their business on. I know what the pitch looks like from behind the laptop.
So let me be fair before I am critical: most AI vendors are selling something real, under enormous pressure to show traction. The problem is structural rather than moral. The demo is the artifact they have perfected, and it is not the thing you are buying.
The seven signals
The dataset is theirs. Ask to run it on yours, live, in the room. A vendor who has deployed at scale expects this request. One who has not will find a reason to defer it.
The metric has no denominator. Ninety-five percent accurate on what, measured against whom, on whose data, at what volume? Until those are answered the number is decoration.
No failure cases. Ask what it does badly. A vendor with real deployments has a ready, specific answer and is not embarrassed by it. A vendor with none has not met production.
The integration is "straightforward." It never is, and it is frequently not in the quoted price. Ask who does it and whether that is a separate statement of work.
The pilot has no end date. Open-ended pilots are structured to be difficult to cancel. That structure is not accidental.
You cannot meet an existing customer. Reference calls are standard practice. A refusal is information.
The team in the demo is the founders. Ask who will actually be assigned to your account, and whether you will meet them before signing.
The one question
If you remember nothing else from this: ask to see it fail.
How a vendor responds to that question tells you more than the whole presentation. The good ones brighten up. They have been waiting for someone to ask something real, and they will tell you precisely where their product struggles, because knowing that is what makes them useful rather than impressive.
What this is not
This is not an argument that AI software is a scam. I sell AI software. It is an argument that the questions most buyers have been taught to ask are the wrong ones, and that better questions are available for free.
The goal is not to be cynical about AI. It is to be expensive to fool.
These seven signals are the first section of my keynote Tools, Tricks And Traps.