Case study · Trust & Safety
Safety by design: building the abuse controls for a community platform
I review reports, appeals and enforcement escalations for a living. When I built my own community, Back in the Day, I designed its safety rules first and wrote them into the database, where nobody can click around them.
- My role
- Policy design, build and testing (solo)
- Built with
- Supabase (Postgres), Next.js, Claude Code
- Rule tests
- 216 automated checks, all passing
The problem
Back in the Day is a nostalgia community for people who grew up between 1990 and 2018. Members post memories, chat live during watch parties, trade collectibles and build profiles. That's every surface where abuse shows up: posts, live chat, usernames, profiles and anything involving trades.
The constraint that shaped everything: there is no moderation team. I'm the only admin, and I'm not watching a queue around the clock. So the rules had to work on their own, resist being gamed, and leave room for a human to fix mistakes later.
How I approached it
- Enforce rules where they can't be skipped. Every rule lives in the database. Hiding a button stops honest users; a rule in the database stops everyone, including someone calling the API directly.
- Assume the report button will be used as a weapon. Mass-reporting is an abuse pattern too. Reports have to be hard to fake, hard to stack and hard to point at the wrong person.
- Act fast, but keep it reversible. Stop the harm quickly, and make it easy for a reviewer to undo a wrong call without it immediately happening again.
- Treat a false positive as a real failure. A filter that blocks innocent people pushes good members away. I tested for wrongful blocks as hard as for missed abuse.
The rules, and why
Reports that can't be weaponized
An account is paused automatically when 3 or more different members report it within 7 days. Only reporters whose accounts are over 24 hours old and in good standing count.
Why: one person with a grudge can't take someone down, and a batch of brand-new accounts made to pile on doesn't count. Three established members acting independently is a strong enough signal to act before I can look.
Tradeoff: three real members could still coordinate against someone. That's why the pause is reversible, and why reinstating an account also clears the reports that were still pending, so the same reports can't trip the pause again a second later.
The server decides who is being reported and saves a copy of the reported content (up to 500 characters). Each member can report a given item only once, and nobody can report themselves.
Why: a reporter can't attach a report to the wrong account, can't stack duplicate reports, and the evidence survives even if the author edits or deletes the post afterward. A reviewer always sees what was actually reported.
A paused account is signed out on every device, can't post, and its content disappears for everyone except the owner and admins.
Why: the harm stops immediately, but nothing is deleted. If the pause was wrong, restoring the account brings everything back.
De-escalation built into the product
Live chat allows 200 characters, one message every 3 seconds, and turns ALL-CAPS messages into normal sentence case. Chat only opens from 30 minutes before a showing until 30 minutes after.
Why: most chat problems are flooding and shouting. Slowing the pace and lowering the volume prevents a lot of them without punishing anyone, and a chat that isn't open can't be raided at 3 AM.
Friend requests are capped at 30 a day. Replies have a short cooldown.
Why: spam accounts work at volume. Normal members never hit these limits.
Shrinking the scam surface
The Trading Post is link-out only: no money moves through the site. Sale listings require a secure (https) link, and trades require contact details.
Why: this comes from my years on fraud and payment-dispute work. Most marketplace scams depend on the platform holding the money or on a buyer clicking an unsafe link. Taking payments off the platform removes the first, and requiring https links narrows the second.
Permissions members can't give themselves
Members can edit their own profile, but never the fields that control enforcement: suspended status, admin status, or their own counts. Only the server can pause an account.
Why: if a client can write a field, someone will eventually try. Those fields are locked at the database level, not just hidden in the app.
Testing for misses and for false alarms
The content filter screens posts, chat, listings, usernames and profiles. It catches slurs and adult terms, including look-alike spellings with numbers and symbols swapped in, and numeric hate codes.
The harder problem is the opposite one. Plenty of ordinary words contain blocked words inside them, and a filter that just searches for letters blocks innocent names. So the tests include a list of legitimate usernames that must get through, such as scunthorpe, essex_girl, dickens_fan, assassin_creed and uranus_rocks. Then the tests run all of the site's own capsule text through the filter and fail if any of it is blocked.
Every rule on this page has automated tests behind it: reports, the auto-pause, chat limits, permissions and the filter in both directions. 216 checks pass against a real Postgres engine.
The same thinking in a second product
Guidance (opens in a new tab) is a mentor-matching app, which means strangers messaging each other one on one. That's a higher-risk setting, so the controls are built around giving each person a way out:
- Blocking works both ways. Once either person blocks the other, neither appears in the other's matches, new contact is refused, and any open connection between them ends.
- Report and block in one step, so a member who feels unsafe doesn't have to choose between them.
- Sign-in is rate-limited to 10 attempts per minute per IP to slow password guessing, and passwords are stored only as salted hashes.
- Email verification before contact: once email sending is switched on, members must confirm their address before they can send or accept a mentorship request. Reset links work once and expire within an hour.
What's not built yet
I'd rather say this plainly than have someone find it:
- Guidance has no admin review screen yet. Reports are saved, but I read them directly from the database.
- Back in the Day has no appeal form. A paused member can't ask for a review from inside the site yet. That's the next thing I'd add: a short appeal with the reported content shown alongside it.
- No moderation metrics yet. I'd want to track how often reports lead to action and how often a pause gets reversed. A rising reversal rate would tell me the pause threshold is too low.
Why this matters for the job
At work I sit on the reviewer side: reading reports, judging appeals, and documenting decisions so the next person can act on them. Building these systems myself showed me the other side: how a threshold gets chosen, how a rule gets gamed, and why a reviewer needs to see the evidence exactly as it was reported. I bring both sides to the work.