Building reliable AI workflows with Inngest: lessons from PaidInsight

This first-hand build article describes PaidInsight's code and commit history as of September 17, 2026, before launch, and makes no claims about customer outcomes.¹⁷

On July 24, a production sync activated PaidInsight's full scheduled-function fleet. We had specified how often each job should run, but the combined schedule consumed more of the execution budget than we had allowed for.

The July 25 fix records eight one-minute pollers and ten more jobs running every five to fifteen minutes. Together, they were consuming about 12,800 executions a day against the then-recorded allowance of 50,000 a month. That is roughly a quarter of the allowance each day: enough to exhaust it in under four days. The same record identifies the operational risk: reaching the plan's cap would pause execution, including the event-driven functions handling emails, interview analysis, and participant rewards. This is the rate and risk recorded at the time, not a claim that customers experienced an outage.⁸

That is the part of building an AI product this article is about: the work between "the model returned something" and "the customer received a usable result". Retries, recovery, and idle operating cost became substantial engineering questions for us. If you are vibe coding an application, these are useful questions to put beside the next feature request.

We build PaidInsight with coding agents working from written plans, so everything below is also a list of things we now ask those agents to check.¹⁶

What PaidInsight actually runs

PaidInsight runs AI phone interviews and turns them into a research report. Simplified, the pipeline is:

1. A voice provider calls a participant and, when the call ends, sends us a webhook.
2. The single-interview enrichment path has three model stages: analyse the interview's structure and answer depth, extract practices and evidence, and normalise labels against the study's running registry. Batch enrichment arranges that work differently, as we describe below.
3. When the customer starts analysis, synthesis builds a qualified interview corpus, assembles evidence, distils findings, reviews their support, and prepares the report. The report path validates the result and freezes the document used for export.
4. A renderer turns that frozen document into the PDF the customer downloads.

Those stages cross several boundaries: provider callbacks, model requests, database writes, and document rendering. A failure between them can leave useful work completed but the next stage uncertain. A model stream that stops producing tokens without settling its result was one such problem in our report tooling.

The agents discussed here have narrow jobs and structured outputs. For example, the evidence judge assesses the findings presented to it, rates their support as strong, adequate, or thin, and records risks and revision notes. Its schema-checked assessment feeds subsequent report work.⁹

Here are seven patterns from that implementation. The design questions also apply when you use another workflow system; the execution guarantees depend on the system you choose.

1. Keep long-running AI pipelines out of webhook handlers

In the Retell completion path, the webhook handler verifies the signature, processes the call event, and queues eligible enrichment work using interview/enrich.requested. The model stages run in the function that consumes that event. The provider does not have to wait for the entire enrichment chain before receiving an acknowledgement.

This is a boundary for background work. An interactive setup request can still call a model directly; PaidInsight's setup endpoint does. Choose the boundary around how long the work takes, who is waiting, and what must survive an interruption.¹⁰

Inngest's durable execution uses named steps. Once a successful step result has been recorded, later executions of the same run reuse it. The function can execute again from the top while the SDK supplies those stored results and resumes unfinished work. Put side-effecting work inside the appropriate step boundary.¹

Our provisioning workflow separates claiming work, preparing the interviewer prompt, deploying the provider resources, marking the study live, and delivering pending invitations. The detailed prompt-generation path has additional steps. When a later step retries within the same run, an already-recorded prompt result can be reused. A newly dispatched run also needs the database's claim and version rules; it does not inherit another run's memoized steps automatically.¹⁰

Synthesis uses the same separation for evidence preparation, findings, review, narrative generation, validation, and persistence. This avoids repeating recorded work just because a later stage needs another attempt. It does not guarantee that an external call happens only once: a provider can accept a request before the step result is saved. That gap needs its own recovery policy.

2. Give each logical operation an identity that survives retries

For enrichment, we derive the event ID from the interview and prompt version: interview:{interviewId}:enrich:{promptVersionId}. Provisioning uses campaign:{campaignId}:provision:{reason}. These names make the intended operation visible during debugging.

A random UUID is also a valid idempotency key if you generate it once for the logical operation, persist it, and reuse it on retries. Stripe explicitly suggests UUIDs or another sufficiently random string. The mistake is assigning a new identity every time the same request is retried.³

Some jobs also need a deliberate redispatch after the earlier attempt has been settled. Our batch-enrichment helpers add a generation suffix such as :retry-2 for that purpose. A new dispatch identity is an explicit decision to admit another attempt; the database still decides whether the underlying work may proceed.¹⁰

Inngest's event-ID deduplication window is 24 hours, with documented exceptions for features such as batching, debouncing, and function pausing.² Our enrichment and provisioning workflows therefore also claim work in Postgres. The simplified enrichment rules are:

• A completed run already exists for this interview and prompt version: skip.
• A run is running and was claimed less than 30 minutes ago: skip, someone else has it.
• A run is running and older than that: reclaim it. The previous worker is presumed dead.
• A run is failed: retry it.
• Otherwise: create it.

A comment in our provisioning function reads "duplicate events are expected and must be handled by the claim service". The event ID helps limit duplicate delivery; the claim checks the persisted state of the work.¹¹

3. Postgres owns persisted state. The workflow engine coordinates steps and concurrency.

For provisioning and enrichment, workflow code coordinates steps and concurrency while the database records the run's state and ownership. Provisioning's service layer also decides whether to schedule a new retry. This keeps the product's state transitions explicit rather than treating an orchestration status as the whole business outcome.

For enrichment, a claim returns an owner token. The result-persistence step updates the run only if that token still matches. If another worker has reclaimed the run, the stale worker cannot replace its result; it returns stale-owner.

That protects the fenced database writes. It cannot undo a model request the stale worker already made, or prevent a provider from charging for it. External effects need provider idempotency where supported, attempt records, or reconciliation before repeating the operation.

The single-interview enrichment function uses an Inngest concurrency key with a limit of one per study. Inngest applies that limit to actively executing steps, not whole function runs; runs can overlap between steps. Other studies can progress within the overall limits.⁴ Database protection still matters: the run checks the registry version it started from before persisting. A version conflict produces registry_stale instead of overwriting the newer registry.

For batch work, we use two phases: parallel provisional extraction per interview, then finalisation that normalises labels across the batch. This separates work that can proceed independently from work that shares a registry. It is a useful design question before simply raising a concurrency limit.¹¹

4. Classify a failure before you retry it. "Unknown" is a class of its own.

We use distinctions like these when deciding whether another attempt is appropriate. The concrete policy belongs to each workflow.

Transient. A rate limit or temporary provider failure may justify retrying with backoff. A timeout also needs a side-effect check: did the provider finish work before the response was lost? Inngest defaults to four retries after the initial attempt, and lets applications configure that policy or throw a non-retriable error.⁵

Permanent or intervention-required. Missing configuration and invalid input need correction. A report rejected by validation needs the appropriate repair or review path. Repeating the unchanged operation should not be the only response.

Unknown. We cannot establish whether the provider completed the operation. PaidInsight's prompt-generation and Retell deployment services use provider_outcome_unknown to quarantine certain ambiguous attempts instead of automatically repeating them. Repeating a request could create a duplicate resource or incur another charge.

Recovery must match the evidence available. For example, our deployment reconciliation command accepts one exact recorded rejection before resource creation. It is not a general resolver for any unanswered provider request. Other workflows make different trade-offs, including the narrowly defined report-tool retry below. Write that decision down for each operation.¹²

Retries also need a time boundary. Our synthesis recovery service considers eligible work within a 36-hour window from run creation. An operator-attention mechanism identifies failed runs beyond that window and uses a deduplicated notification outbox. Its error categories help the operator distinguish provider timeouts, configuration problems, invalid input, and report validation failures. A recovery window is not a promise to retry every kind of failure throughout that period.¹²

5. Give streams a deadline and choose the fallback deliberately

A stream can fail while the SDK's result promises remain unsettled. In our report tooling, that left the caller waiting for a result that would not arrive. Handling a returned HTTP error did not cover that failure mode.

The shared runtime's streaming-result settlement code races the result against reported failure, cancellation, and a deadline. A reported stream error can settle immediately; the deadline catches a result that otherwise remains pending.

Our internal report-rewrite tool also has a narrow stall-retry wrapper. Its default allows one separately recorded extra attempt when an execution failure has no output hash, provider status, or finish reason. Schema failures, truncation, and HTTP rejections do not enter that retry path. This is a policy for that tool, not a promise that every customer report or unknown provider outcome is retried the same way.

Fallbacks need the same precision. The ordinary synthesis path can retain a compact report when optional narrative generation produces no narrative, subject to the remaining validation. The governed revision path requires a complete narrative result and otherwise stops. Neither path promises automatic delivery of improved prose later.¹³

Decide which reduced output is still useful and valid for the workflow. When the missing stage is essential to the promised result, stop and explain what needs attention.

6. Check how each webhook-driven workflow recovers

Several core paths start with a webhook: a callback arrives, a handler claims it, and downstream work is queued. If receipt or processing is interrupted, your stored state can lag behind what happened at the provider. A callback that worked once does not establish how that state will recover.

In the code snapshot reviewed for this article, 31 of 44 registered functions are scheduled. They include recovery jobs, retention cleanup, reminders, and showcase maintenance. The count is not a measure of recovery coverage. Concrete recovery examples include:

• Retell webhook recovery identifies failed or stale processing rows, with a 15-minute stale threshold, bounded backoff, and an attempt cap before operator attention.
• Retell call-start reconciliation uses the recorded correlation token to look for a matching provider call. Depending on the result, it can attach the call, settle it, reopen the invitation, or flag it for attention. Multiple matching calls require attention rather than an arbitrary selection.
• Enrichment batches, provisioning retries, synthesis retries, and queued emails each have a sweep that re-queues due work.

The schedule tooling records a usefulWork count to help distinguish recovery activity from idle polling. Webhook responses also matter: our email-provider handler returns HTTP 503 for an unfinished claim, preserving eligibility for the provider's retry policy. It returns 200 for processed or duplicate results. Resend documents a bounded retry schedule and a replay option. That response preserves a recovery opportunity; it does not guarantee eventual completion.⁷

Coverage still needs a workflow-by-workflow check. The current operator recovery matrix explicitly records a scheduled-owner gap for Deepgram call-start reconciliation. Keep gaps visible instead of inferring complete recovery from a large function fleet.¹⁴

7. Your scheduled fleet has a budget. Write the test before the outage.

Back to July. The immediate change moved the affected frequent pollers to a 30-minute cadence and four workspace/cosmetic reconcilers to hourly. Staggered offsets reduced simultaneous starts. The commit records that primary event-driven paths were unchanged, while the polling delay before recovering a missed event could grow from about a minute to about thirty. That delay is one part of recovery time; it excludes queueing and processing.⁸

In August we added more explicit budget accounting. Inngest defines an execution as a function run plus each step inside it, so a job with one executed step consumes two executions even when it finds no work.⁶ The reviewed source now contains:

• One schedule catalog listing every cron function with two cadences, a default free-tier profile and an opt-in standard profile, plus the number of steps it executes when there is no work to do.
• A pure calculator that accounts for the peak occurrence count of each schedule across rolling 30-day windows, including calendar boundaries, and combines those counts into a conservative idle budget.
• A budget test with a ceiling of 10,000 idle executions per 30 days. At the code snapshot used here, the free-tier projection is 9,220 and the standard projection is 59,380. These are code-based idle projections, not observed usage or an estimate of total active-work cost.
• A test that reads the source files named in the catalog and checks that each cataloged schedule is registered exactly once. Together with the budget test, this catches registration mismatches and excessive cataloged schedules. Detecting a new job that bypasses the catalog still requires checking the function inventory.

We also have an opt-in local QA mode that registers a single no-op probe instead of the production fleet. It predates the July budget fix. Merely pointing a development server at a local database does not enable it; the mode must be selected explicitly.¹⁵

What to ask your coding agent

Start with one workflow you already have. Ask your coding agent to show the evidence before proposing a change:

1. "Find the long-running model or provider calls in our webhook handlers. Which need a durable job, and which interactive requests should remain synchronous? Explain the boundary."
2. "For this background job, list the database writes and external calls a duplicate attempt could repeat. Show the protection for each and identify any gap."
3. "List every webhook we depend on. For each one, name the scheduled sweep that recovers a missed delivery, or say that there is none."
4. "For each provider operation, distinguish a retryable failure, a failure needing intervention, and an unknown outcome. Show the retry policy and the risk of repeating it."
5. "Which model calls can leave results unsettled? Show their deadlines and cancellation behaviour. Where is a reduced result acceptable, and where must the workflow stop?"
6. "Project our scheduled job executions over 30 days against the plan cap, and add a test that fails when a new schedule breaks the budget."

Choose the smallest change that closes a demonstrated gap. That might be a persisted operation ID, a guarded state update, a deadline, or a recovery job. Keep the reproduction and expected behaviour with the change so the next agent can check the same boundary.