I built a tool because I was the questionnaire guy

Somewhere in the middle of a long compliance push, I became the questionnaire guy.

If you’ve worked at a company that sells software to other companies, you know the genre. A prospect’s security team sends over a spreadsheet with two hundred questions. Do you encrypt data at rest? Describe your incident response process. How do you handle key rotation? Do you perform background checks? Someone has to answer all of them, and the honest answer to most is “we already answered this exact question last month, in slightly different words, for a different customer.”

So you go find that spreadsheet. Except it’s in someone’s email, or a shared drive folder named Security Docs (FINAL) v3, and the answer you wrote then is subtly different from the answer someone else wrote in a questionnaire two months before that. Now you have a consistency problem on top of a throughput problem, and the consistency problem is the one that actually matters — because two customers comparing notes on contradictory answers is a much worse afternoon than a slow turnaround.

I wrote Questionnaire Copilot to fix that for myself.

The shape of the problem

The naive framing is “answering questionnaires is slow, make it fast.” That’s wrong, or at least incomplete. The real problem has three parts:

Retrieval. The answer almost certainly exists. Finding it is the work.
Consistency. The same question should get the same answer every time, and when the answer changes, it should change everywhere.
Provenance. When someone asks why we answered that way, you want to know where the answer came from and when it was last reviewed.

Notice that only the first one is about speed. The other two are why a folder of past spreadsheets isn’t a solution even though it technically contains all the information.

So the tool isn’t really a questionnaire filler. It’s a vault of canonical, approved answers, with a matching layer bolted on top. Each entry is a question, its approved answer, tags, and a source — the audit or customer it was written for. That’s the artifact that has value. The answering workflow is just the interface to it.

Matching without reaching for an LLM

Here’s the part where you’d expect me to say I used embeddings.

I didn’t. The matching runs on Postgres trigram similarity — pg_trgm, one migration, an index, and a similarity threshold. When a question comes in, it gets compared against every question in the vault and the closest matches come back ranked.

This is dumber than semantic search, and I chose it on purpose:

It’s deterministic. The same question produces the same suggestions today and next quarter. For something that feeds into a compliance artifact, being able to explain exactly why a suggestion appeared is worth more than marginally better recall.

It has no external dependency. No API key, no per-token cost, no vendor to add to your own vendor risk assessment. There is something deeply funny about a tool for answering security questionnaires that itself introduces a third-party data processor — you’d have to disclose it on the next questionnaire.

It’s good enough. Security questionnaires are unusually well-suited to lexical matching. The vocabulary is constrained and repetitive. “Do you encrypt data at rest?” and “Is data encrypted at rest?” share almost every trigram. The questions that trigram matching misses tend to be the genuinely novel ones, which need a human anyway.

Embeddings are on the roadmap via pgvector, and they’ll help with the long tail. But shipping the simple version first meant I was using it within a week instead of tuning a retrieval pipeline for a month. The vault filled up with real answers, and a full vault is what makes the tool useful — not the cleverness of the matcher.

LiveView, and no JavaScript

The app is Phoenix LiveView, Postgres, and zero custom JavaScript. Not “a little bit of JavaScript.” None.

That includes the parts you’d assume need it. Drag-and-drop CSV upload is a LiveView upload. Keyboard navigation through the answering flow — j and k to move, s to skip — is a phx-window-keydown binding. Tag filtering, live search over the vault, copy-to-clipboard: all server-rendered, all over the websocket.

This is the thing about LiveView that’s hard to convey until you’ve built something real with it. The answering interface is genuinely stateful — you’re moving through a list, some items are answered, some skipped, a search panel is open over here, there’s a modal asking whether to save a new answer to the vault. In a conventional SPA that’s a client-side state machine, a set of API endpoints, and a serialization format between them. In LiveView it’s a struct in a process, and the process is on the server where the data already is.

The failure mode of LiveView is latency-sensitive interactions — if you need sub-frame response to mouse movement, it’s the wrong tool. Filling out a form is not that.

What I’d tell someone building the same thing

The vault is the product. I spent early effort on the matching and the answering UI, and the thing that actually made it valuable was the boring CRUD around approved answers with tags and sources. If you build one of these, build the library first and the workflow second.

Track where answers came from. The source field seems like metadata until the first time someone asks “is this still true?” An answer written for an audit two years ago and an answer written last month are different kinds of claim, and you want to be able to tell them apart at a glance.

Prompt to save new answers. When you write an answer that isn’t in the vault, the app asks if you want to add it. Without that, the vault goes stale — you answer the novel questions in the tool and they evaporate. This one-line-of-UX decision is most of why the vault stays current.

Keep the humans in it. The tool suggests; it never submits. Every answer is accepted, edited, or written by a person. The failure mode of automating this completely isn’t a wrong answer — it’s a confidently wrong answer attached to a contract, and there is no undo for that.

It’s MIT licensed and self-hosted, because a tool whose entire database is your organization’s security posture is not one you want to hand to a SaaS vendor.

Follow-up, a week later: I added semantic search after all, and it changed the tool completely. The reasoning above turned out to be aimed at the wrong target — here’s what changed my mind.