Retry policies are business rules, not infrastructure settings
Generic retries hide the only question that matters
Most workflow builders start with a retry count and a delay. Retry three times, then fail. That setting is useful for a network timeout. It is dangerous for a payment capture, a customer notification, or an inventory reservation where the remote system may have completed the action before the response disappeared.
The workflow needs to know the effect boundary. Before the boundary, retrying is usually safe. After the boundary, the system needs an idempotency key, a reconciliation check, or a compensation path. Infrastructure can provide delivery guarantees; only the business workflow can define what delivery means.
Typed outcomes make failure visible to the operator
A typed outcome separates retryable, compensated, rejected, and awaiting-review states. A carrier status update that times out can retry. A duplicate invoice match should move to review. A rejected approval should follow the rejection branch. The operator sees the reason and the next action instead of a red failed run with no business context.
- Retryable failures
- 62% transient or rate limited
- Compensation paths
- 24% external effect completed
- Human review
- 14% ambiguous business state
- Duplicate actions
- 0.03% after idempotency controls
The audit trail should explain the decision, not just the attempt
A useful run history records the input, the external effect, the policy that selected the next branch, and the person who approved an exception. That history lets an operations lead answer why a workflow retried, why it stopped, and whether the customer or ledger was affected. Timestamps without reasoning are logs, not an audit trail.
Frequently asked
When is it safe to retry a workflow step?
When the step has not produced an external effect, or when the destination supports idempotency and the same key can safely be reused. A transport timeout is not proof that the remote action failed. The workflow needs a reconciliation or idempotency strategy before repeating a side effect.
What is compensation in a workflow?
Compensation is a deliberate business action that reverses or neutralizes an effect that cannot simply be retried. Examples include releasing a reservation, voiding a draft invoice, or notifying an operator that a payment requires review. It is not the same as deleting a log entry or marking a run failed.
When should a workflow stop for human review?
When the system cannot establish whether the external effect occurred, when policy requires approval, or when the available data conflicts. The review task should include the evidence, the attempted action, the possible outcomes, and a clear decision to resume, compensate, or close.
More from Field Notes
All essays
Workflow Workflows that survive contact with reality: durable execution, explained
Real workflows fail. Networks blip, downstream systems quota out, humans go on vacation. Durable execution is the architecture that keeps a multi-hour workflow honest.
Workflow Human-in-the-loop done right: when to pause, when to auto-resume
Designing workflow human-in-the-loop steps that respect human time, audit cleanly, and don't stall the workflow — patterns and anti-patterns.
Workflow Retry, backoff, and saga compensation: anti-patterns and the real shape
How to design retry, exponential backoff, and saga compensation in workflow automation — the anti-patterns that bite production and the patterns that survive.