How to tell if your AI is actually selling anything

Every tool in this category will show you a number that went up.

Conversion lift. Revenue influenced. Assisted sales. AI-attributed GMV. The dashboards are confident, the arrows point up, and the number is almost always large enough to justify the subscription and small enough to seem credible.

In most of these tools, that number is unfalsifiable. Not wrong, necessarily, but impossible to check. There is no operation you can perform, as the merchant paying for it, that would tell you whether it's real.

That should bother you more than it usually does, because you're being asked to make a recurring spending decision on evidence you cannot inspect. Five questions get you most of the way out of that. They work on any vendor in this category, and we've answered them for ourselves below, including where our answers are weaker than we'd like.

1. What exactly counts as a sale you made?

Ask for the rule. Not the marketing description, the rule.

The dangerous version is proximity-based: if a shopper interacted with our widget and then bought something within N days, we count it. A lot of attribution works this way, and it will always produce an impressive number, because shoppers who engage with anything on your site are the shoppers most likely to buy. The tool is measuring your best customers and taking credit for them.

The version that means something is deterministic. Either the tool's own action put the item in the cart, in which case it knows with certainty because it did it, or it didn't. Anything softer than that belongs in a separate, clearly labelled category, with a stated reason for each individual case.

We use three tiers and only three. Direct means our add-to-cart action put the item there, with no inference involved. AI Influenced means the shopper added it themselves after we surfaced, recommended, compared or substantively discussed that exact product, judged per item, with a written rationale you can read. Organic is everything else.

2. Which direction does it round when it's unsure?

This is the single most revealing question you can ask, and almost nobody asks it.

Every attribution system has ambiguous cases. What matters is which way the ambiguity falls, because that choice tells you what the system was built to optimise. If uncertain cases resolve in the vendor's favour, the reported number is a ceiling dressed up as a measurement.

Ours rounds against us, in three specific ways. Attribution that hasn't been graded yet counts as organic, not as ours. Orders we can't confidently link to a shopper session count against the baseline rather than being excluded. Removed items are netted out of cart value, so a shopper who adds and then removes doesn't inflate anything.

The consequence is that our reported figure is a floor, and the true contribution is higher than what we show. That costs us on every invoice. We consider it the most important design decision in the product, because it's the one that makes the number worth reading at all.

3. Can you open the sale and read it?

No amount of methodology substitutes for this one. Take any number in the dashboard. Click it. Do you get to the actual conversation, what the shopper typed, what the tool replied, where it went right or wrong? Or do you get a slightly more detailed chart?

If you can't drill into a number you can't check it, and a number nobody can check isn't evidence of anything. Being able to open it matters beyond verification, too. Reading the conversations behind a good week is how you learn what's working on your store, and reading the ones behind a bad week is how you find out what's broken.

Every figure in our dashboard links through. "From 34 conversations" is a link, and it opens the 34.

4. Is it revenue, or is it cart value? Is it placed, or is it paid?

These get conflated constantly, sometimes carelessly and sometimes not.

An item added to a cart is influence. It is not money. A meaningful share of carts never become orders, so cart value and revenue are different quantities, and anyone reporting the first while saying the second is inflating by whatever their cart abandonment rate happens to be.

Separately, an order placed is not an order paid. The two diverge, and if you sell cash on delivery they diverge enormously.

Our headline figure is value added to cart, and we say so explicitly rather than calling it revenue. Order-level revenue attribution is computed in our pipeline but isn't surfaced in the dashboard yet, so we don't market it. When we report orders, they're placed orders, and we say that too. We would rather show you a smaller honest number now than a larger one we'd have to walk back.

5. What's the baseline, and over what window?

A lift is a ratio, and a ratio means nothing without knowing what's underneath it.

Ask what the comparison group actually is. "Versus site visitors" is a much weaker claim than "versus comparable shoppers who didn't engage," and the two produce very different numbers from the same data. Ask over what window, because short windows are volatile and a vendor can shop for a flattering fortnight. Ask what the sample size was, because a lift computed on nine converters tells you nothing at all.

We publish lift as a range with both ends: an upper figure, and a deliberately pessimistic floor computed by assuming every order we couldn't link belongs to a shopper who never engaged. Both, always, with the dates attached. Below roughly fifty engaged shoppers or ten converters in a window, we show "not enough data" instead of a rate, because at that size the number would be meaningless and publishing it anyway would be a small lie.

There's one more caveat we volunteer rather than wait to be caught on. Shoppers who talk to a sales agent self-select. Engaged shoppers both chat more and buy more. So we say "shoppers who talked to it converted at X% versus Y% for those who didn't," and we never say we caused the difference. That framing is less impressive, and it's the one that survives a smart person reading it.

Why we're handing you the questions

Because we lose the sale if you don't ask them.

If you evaluate this category on who reports the biggest number, we come last on purpose. We round against ourselves, we publish floors, we call cart value cart value, and we volunteer the self-selection caveat. Every one of those choices makes our figure smaller than a competitor's on identical underlying performance.

The only world where that's a good strategy is one where you check. So we'd rather arm you with the questions, take our chances, and be the tool whose number you still believe in six months.

Run the five questions on whatever you're using now. If it passes all of them, keep it, because you're in better hands than most merchants. If it can't answer question two, or you can't complete question three, then nobody has measured whether the thing works, and that includes the vendor charging you for it.