The bots don't use the words on your list

Every “request a demo” form gets three kinds of traffic. People who want the product. Agencies selling SEO, design, developers, leads. And scripts.

The usual defence takes an afternoon. Two hidden fields a person never sees and a bot fills in. A timestamp, so a submit two seconds after page load scores against you. A word list of the things agencies say: backlinks, digital marketing, guest post, lead generation. Three points and you’re dropped, the visitor still sees “thanks”, and nobody gets paged.

It works on the noisy ones. It fails in both directions on the quiet ones.

The pitch that walks straight in reads like this: “Hi there, love what you’re building. I run a small studio that helps SaaS founders fill their pipeline with done-for-you outreach. We got an invoicing startup in Austin 14 booked demos last month. Open to a 15 minute chat?” No trigger words. Sent after 38 seconds. Zero points. It lands in the sales channel and someone loses ten minutes to it.

The customer who gets flagged reads like this: “I run a lead generation shop for realtors. We use QuickBooks today. What does onboarding look like for a team of six?” The words “lead generation”, because that’s what the business does. One point. Delivered, but with a warning line, which is a small insult to the one person the form exists for.

A word list scores words. The thing you want scored is a sentence.

One question instead of a list

Jev is TypeSafe’s decision model. You send it state and typed questions and it returns probabilities, not prose. So I asked it one question per submission:

Is this a genuine request from a person who wants the product described, written about their own business or need?

with a yes/no answer (TypeSafe calls that a noul) and criteria that say what a no looks like: selling something to the site owner, a template, gibberish, hidden fields filled in. The routing rule lives in code, not in the model:

P(real) ≥ 0.80   deliver
P(real) < 0.20   drop
otherwise        hold: deliver with a flag, a person decides

Two more questions ride in the same request, since Jev reads the state once and answers everything in parallel: is this a pitch, is this automated. They don’t route. They label the drop.

And one more, because “the line that gave it away” is worth more than the probability. Jev doesn’t generate text, so I split the message into sentences, key them s0 to sN, and ask a choice question: which sentence most shows this is not a genuine request. Code maps the key back. That’s a pattern from TypeSafe’s docs for searching text line by line, and it means every drop in the channel carries a quote.

For a checkout the state is different (card country, IP country, attempts before success, orders from the device in the last hour, account age) but the shape is the same: one noul that routes, one that labels, one choice over the facts for the evidence.

The two that broke the word list

The polished pitch:

DROP  P(real)=0.13  reason=pitch
evidence: "We got an invoicing startup in Austin 14 booked demos last month."
word list: deliver (0 pts)

The customer with the trigger words:

DELIVER  P(real)=0.89
evidence: "What does onboarding look like for a team of six?"
word list: flag (1 pt: pitch:lead generation)

Same question, same thresholds, opposite results from the list, and each one carries the sentence that decided it.

The run

I built a fictional small invoicing SaaS and generated 312 submissions to its demo form and its checkout: a bit over half real, the rest pitches, scripts, card testers, and a grey zone written to be genuinely hard: “how much”, “is this legit?”, a message in Spanish, a university group wanting an interview, an agency asking to resell. Eight workers, one request each.

Demo form and checkout
submissions	312
wall time	9.3 s
throughput	34 per second
median latency	226 ms
p95 latency	332 ms
cost	$0.0106
delivered / dropped / held	185 / 109 / 18

Nothing landed on the wrong side of the line. No real request was dropped. No pitch, script or card-testing order was delivered.

Against the labels I wrote the data with, 297 of 312 matched. The 15 that didn’t are worth reading, because most of them are on me:

• The university group wanting an interview was dropped at 0.05. Correct: it isn’t a request for a demo. I’d labelled it grey because it’s a polite email, which is not the question.
• The agency asking to resell to thirty bookkeeping clients was delivered at 0.86. That’s a lead.
• A second prepaid-credit order in a week from a known account was delivered at 0.88. That’s a customer.
• Four “real” messages went to hold, and three of them are my generator’s fault. It composed “I run Bluebird Daycare, a lead generation shop for realtors”, a sentence that contradicts itself, and Jev held those at 0.56 to 0.75. The model noticed something wrong with a sentence I hadn’t read carefully. The fourth was a brand-new account buying prepaid credit, held at 0.77, which is fair.

The word list, on the same 183 demo requests: 32 pitches or scripts let through, 14 real requests flagged. Forty-six wrong calls from the list against zero on the wrong side from one question.

Against labels I didn’t write

The invented set proves the argument. It doesn’t prove the model, because I wrote the labels. So I ran the same questions, with the site context swapped for a comment section, over the UCI YouTube Spam Collection: 1,956 real comments under five music videos, hand-labelled spam or ham by the dataset’s authors in 2015. The spam is people promoting channels, links and money schemes to an audience that didn’t ask, which is the agency category in a different costume. Jev has never seen this data and I didn’t tune the questions after the first run.

YouTube comments
comments	1,956, of which 1,005 spam
decided	1,783
precision / recall on decided	0.992 / 0.980
held for a human	173 (8.8%): 95 spam, 78 ham
forced yes/no at 0.5, no hold lane	96.0% accuracy
cost	$0.053

The misses are instructive. Most of the eighteen spam comments it delivered are about view counts and subscriber numbers, which the dataset’s authors labelled spam and which read as ordinary comments to me too. The seven ham comments it dropped are bare video links and “like me”. The hold lane caught “Check me out!”, emoji walls, and the view-count chatter, which is exactly what I’d want a person to glance at.

The word list caught nothing on this set. Not one comment. That is what a word list is: it knows one site’s vocabulary and nothing else.

What I’d be careful about

• The invented labels are mine. Half the disagreements turned out to be my labels or my generator being wrong. The YouTube run is the one with labels I didn’t write, and a real inbox is the test after that.
• Jev reads literally. The criteria spell out what a “no” means, because the docs say to, and the drop on the university group is that literalness working as intended.
• The hold lane is doing real work. “how much”, “my friend runs a shop”, a German card on a Turkish IP, a prepaid card on a day-old account: those went there, which is where I want them.
• Non-English is weaker by the docs’ own account. The Spanish message is in the set on purpose and came out right, but one message is not a test.
• It’s a form, so the attacker controls the state. TypeSafe’s docs list adversarial text as a known weak spot. The honeypot and the timer stay; they’re signals in the state now, not the verdict.

The part that generalises

Every risky form has a version of this. Checkout. Signup. “Contact sales”. Any inbox where the cost of a missed real one is a lost customer and the cost of a false one is someone’s attention. The rule list is cheap and catches the loud half. One typed question at two hundred milliseconds and three thousandths of a cent catches the quiet half and tells you why.

The code, the generated submissions and both runs are on GitHub: 0x4rch/doorman.