Snitch: AI social listening that knows when not to act

The problem

Monitoring a keyword is easy. Monitoring it usefully is not. A brand called Arc, Linear or Wonderful gets thousands of posts that aren’t about it. Meanwhile the posts that matter (a bug report, a complaint about a competitor, someone asking for exactly what you sell) arrive in the same stream as everything else.

Then there’s acting on them. A reply is public and speaks for the brand. That’s where an AI system most needs to know its limits.

Three questions, not one

Snitch answers three different questions, and they are not variations on a theme:

The mode isn’t a label. It is the question the classifier is asked, and asking the wrong one doesn’t just mislabel a mention, it deletes it. Ask “is this post about the customer?” of a thread about a rival and the honest answer is no, so the post is dropped before it’s stored. Each mode therefore names its own subject and says explicitly what is not required. A problem-mode match ideally names no product at all.

This is a requirements decision more than a modelling one. Getting the question right matters more than which model answers it.

Cheapest filter first

Only one layer costs money per mention. Running it on posts about pasta would spend almost the whole budget on noise, so the layers run in cost order.

Fig. 3.1 · Four layers, in cost order

1. 01Platform queryFree

   Required and excluded terms are pushed into each platform’s own search syntax, for X, Reddit and YouTube.

   Noise rejected here costs nothing at all.

2. 02Local pre-filterFree

   Before any model call, it rejects only what is certain: the excluded terms you named. Optional strict matching discards posts with no local sign of the brand.

   Strict matching is off by default, because a silent drop is the worst failure this app has.

3. 03Classifier, told the subjectPaid per mention

   An LLM classifier gets the brand, competitor or problem context in the customer’s own words, the keyword’s group and its ambiguity flag. It is explicitly allowed to be uncertain.

   Without permission to be uncertain, models round everything up.

4. 04Confidence bandRouting

   The score is split three ways: confident matches reach the feed, hopeless ones are dropped, and the middle goes to Needs review.

   Needs review: no alert, no auto-draft, and a visible count.

A confidence band, not a coin flip

Fig. 3.2 · Where a mention landsInteractive · illustrative thresholds

Classifier relevance score

0.50

DroppedNeeds reviewFeed

0floorthreshold1

Needs review

• No alert sent
• No reply drafted
• Counted in the Needs review headline, so the queue can’t grow unnoticed

Default with no AI key configured every mention scores 0.5, below the default threshold, so an unconfigured deployment fills Needs review instead of claiming judgements it can’t make.

Nothing is discarded invisibly. Each keyword’s Refine panel lists recent posts that matched but never reached the feed, whether rejected by rules, judged not relevant or collapsed as a duplicate. Over-tight rules can be seen and loosened rather than quietly losing mentions.

Reviewing teaches it, without training. “About us” and “Not about us” decisions update the mention’s status, and the last few decisions per keyword go back to the classifier as labelled examples on the next scan. It adapts per brand with no extra infrastructure.

Approval is not publishing

Snitch drafts replies when a mention is worth answering: a bug report, complaint, question, misinformation, purchase intent or testimonial, or anything urgent. Drafting is the easy part. The design is built so two things can’t happen: publishing something nobody approved, and publishing it twice.

Fig. 3.3 · From draft to public reply

1. 01

   Draft

   Editable; approve, rewrite or discard it. Auto-drafting is off for competitor matches.

2. 02

   Approve

   Records who signed off and when.

   Guard Approval is consent, not publication.

3. 03

   Publish, explicitly

   A separate request, made by a person.

   Guard Needs the replies:publish scope, which the default read-write API key leaves out. Each workspace has its own posting switch.

4. 04

   Queued once

   The body is snapshotted at queue time, so what goes out is exactly what was approved.

   Guard One post per draft (unique constraint); a conditional QUEUED → SENDING claim means only one worker proceeds.

5. 05

   Posted or retracted

   Retried up to three times; an expired grant fails immediately. Events are emitted on post and retract.

The Slack workflow follows the same rule. Someone can ask @heysnitch in a mention’s thread to reply or to create a Linear ticket. Snitch creates an idempotent, audited action proposal, and a person approves or rejects it. An approved Linear action runs. An approved reply becomes an approved draft, and publishing it is still a separate, explicit step. Approving in Slack never puts anything public on its own.

One capability layer for people and agents

The dashboard, the versioned REST API and the MCP endpoint are driven by one Zod capability registry, so an AI client gets exactly the same rules as a person:

• Scoped API keys are enforced on every route and MCP tool call.
• Rate limits per principal. API keys get tighter budgets than dashboard sessions, because a script in a loop and a person clicking are different risks. If Redis is unreachable the limiter fails open and logs. That trade-off is chosen deliberately, so the product doesn’t go down with its counter store.
• Idempotency keys on writes. A repeated key returns the first response, and the same key with different arguments is refused.
• An audit trail of every state change, recording actor, target, surface and success. Reads aren’t audited, because logging every list call would bury the entries that matter.
• A transactional outbox. Domain events commit with the change that caused them, so a crash can delay a webhook but can’t lose it. Endpoints that keep failing are disabled.

Why this belongs in an applied-AI portfolio

Strip away the social-listening specifics and these are the questions CS-02 asks of support automation. What may the system do on its own? What must a person approve? How do you make uncertainty visible instead of guessing? How do you make every action attributable and safe to retry? Snitch is where I’ve had to answer them in working code.