Case study 01
ApplyTron
A job-hunt pipeline that ingests listings, scores them against your real resume with an LLM, writes tailored ATS-safe documents — and then stops and asks you before anything is sent anywhere.
The problem I was actually solving
Applying to jobs at volume is a data pipeline wearing a trench coat. The listings are an unstructured feed. The resume is a document that needs to be re-derived per target. The submission is a form fill. Every part of it is automatable, and there are a hundred tools that automate it.
The reason most of them are useless is that they will happily send a wrong document to a real employer, and you find out later. That failure mode is not hypothetical to me — I've spent months reconciling records that systems had confidently written wrong. So the design constraint came first, before the stack:
The system may do anything reversible without asking. It may not do anything irreversible — sending a document to an employer — without an explicit human approval recorded against that specific document.
Architecture
The state machine underneath it
The LLM call is the least interesting part of this system. The engineering is the guarded transition table around it — a document can only move where the machine allows, which is what makes "it cannot skip the approval step" a property rather than a promise. Try to break it: the only buttons offered are the legal transitions from wherever the document currently is.
The unglamorous problem: aggregator links go nowhere
A job feed gives you a redirect URL, and that URL usually lands on the aggregator's own interstitial rather than the employer's form. A browser automation pointed at it fills in nothing useful. So at ingestion the system tries to find the real posting: it derives a likely company slug and asks the public job-board APIs that Greenhouse, Lever and Ashby expose, in parallel, matching on the role. A hit is stored as the listing's apply URL and preferred over the feed's link from then on; a miss just leaves the original.
It is the least interesting code in the repository and it moved the success rate more than any prompt I have written. A resolved URL is also not a trusted one — the same field can be pasted in by hand on a listing that needs manual attention, which is why the host allowlist is checked again at submission time rather than inherited from here.
Three closed loops
A single model call is an open loop: whatever comes back is what ships. Three places in this system close that loop, and they are unrelated mechanisms — the temptation to fold them into one “retry helper” is the mistake I expect to be tempted by later.
| Loop | What closes it | Where it stops |
|---|---|---|
| Generate → verify → regenerate | Output with a contract — a JSON shape, a 250-word ceiling on the cover letter, a list of banned phrases, an employer name that has to appear verbatim in my own resume before the model is allowed to claim it — is checked by pure functions, never by a second model call grading the first. A failure is appended to the conversation as the assistant's turn plus a critique, which repairs far better than rebuilding the prompt. | Bounded by attempts and cumulative output tokens, so a constraint the model can never satisfy cannot bill forever. |
| Employer questions | ATS forms reveal follow-up questions as you fill them, so reading once is not enough: read what is still required, answer, fill, read again. | Only facts I have actually stated. An answer it cannot ground stays blank and blocks submission — an invented work-authorisation answer is a misrepresentation on a real application, not a UX papercut. |
| Queue retry | A navigation timeout is not the same failure as a form the worker cannot read. Transient ones go back in the queue with exponential backoff and jitter. | Four attempts, and classification defaults to permanent. Jitter matters even with one worker: a restart requeues every in-flight row at once, and without spread they all re-hit the same ATS in the same second. |
Decisions and what they cost
| Decision | Why | What it costs |
|---|---|---|
| Mandatory approval gate | An unattended agent sending wrong documents to real employers is unrecoverable. Reputation damage has no rollback. | It is no longer "fire and forget". Throughput is capped by how often the user checks email. |
| Weighted sub-scores, not one holistic number | Asked for a single 0–100 fit score, the model clusters on attractor values — 70, 72, 78 — and stops discriminating. So it grades five narrow criteria instead (required skills, title, experience level, domain, logistics), each with evidence, and the percentage is computed in code from fixed weights. | More output tokens per listing, and the weights are my judgement rather than anything fitted to data. |
| Claude Haiku rather than a larger model | Scoring fans out across every listing in a run. The cheap model is accurate enough for a ranking decision once the rubric does the structuring, and it keeps a run affordable. | Weaker on nuanced role fit. Mitigated by only using the score to rank, never to reject outright. |
| Auto-submit gated on a host allowlist | The document that reaches a submit button may have come from a URL the user pasted. Recognising the destination as Greenhouse, Lever or Ashby is what separates “a form I understand” from “an arbitrary page on the internet”, and it is checked at the moment of submission, not at ingestion. | Most employers are on neither, so most applications stop at a screenshot and get finished by hand. |
| Pure-code checks, not a model grading a model | An LLM judge is another open loop — it fails in the same ways as the thing it is checking, and it costs a second call on every generation. A word count, a banned-phrase scan and a grounding check against the source resume are all deterministic, so they are ordinary functions with ordinary unit tests. | Only catches what I thought to encode. It cannot tell me the writing is bad, only that it broke a rule. |
| Row-level security in Postgres | Multi-user from day one. Authorization enforced in the database rather than in application code, so a route that forgets a check still cannot leak. | Policies are harder to debug than a plain conditional, and every new table needs its policy written deliberately. |
| Playwright worker on a separate host | Headless Chromium needs long-running processes and real memory. That does not belong in a serverless request. | A second deployment target, a second set of secrets, and a queue between the two to keep them decoupled. |
| react-pdf instead of an HTML-to-PDF service | ATS parsers read the text layer. Generating the document structure directly keeps it real text rather than a rendered image. | Layout control is more limited than a browser engine, so the templates are deliberately plain. |
- Auth (email and Google), ingest, scoring, document generation and the approval flow are live and multi-user today.
- Submission is off unless the operator sets
AUTO_SUBMIT=true, and even then it only clicks submit when the landing page is a recognised Greenhouse, Lever or Ashby host. Everything else — including Workday — stops at a filled-in form and a screenshot for me to finish by hand. - Employer-specific required questions — work authorisation, visa status, education — are answered by a loop that reads the form, answers from my own stated facts, fills, and reads again. A field it cannot ground in something I actually said stays blank, blocks auto-submit, and gets named in the failure reason.
- The worker still only fills what a generic selector heuristic finds. Per-ATS field adapters — Greenhouse, Lever and Workday field IDs — are the next piece of work if hit rate starts to matter.
- Transient failures back off and retry (four attempts, one minute to thirty). Anything not recognised as transient is treated as permanent and handed to me, because a wrong “retry” verdict costs forty-five minutes of silence before I'm told to apply by hand.
- Scoring quality has not been measured against a labelled set. I can tell you it ranks sensibly; I cannot yet tell you its precision, and I would rather say that than quote a number I made up.
What I'd tell you in an interview
The interesting engineering here isn't the LLM call — it's the state machine around it. A document moves through ingested → scored → generated → pending_approval → approved → applying → applied, and every transition is a conditional update that names the status it expects to be moving from. If the row is not in that state the write affects nothing and the caller backs off, so two browser tabs and the worker can all reach for the same application without one of them winning twice.
That matters most at the approved → applying step, which is how the worker
claims a row. The cost of getting it wrong is not a confusing UI — it is the same
application submitted to the same employer twice. For the same reason the submit click
is the last thing attemptApply does: the queue cannot distinguish a click
that failed from one that succeeded and got lost, so nothing that could throw is allowed
to run after it. When the worker restarts mid-flight it requeues anything stranded in
applying back to approved rather than assuming either outcome.
Stack
- Next.js
- TypeScript
- Supabase
- PostgreSQL
- Claude API
- react-pdf
- Resend
- Playwright
- Railway
- Docker
- Vitest