Process webhooks with retries and failure cases

Redelivery should not create another business action

A webhook informs an application about an event elsewhere by sending an HTTP request. That simple connection contains an important uncertainty: the receiver may already have saved a change while its response is lost on the return path. The sender then lacks clear confirmation and may deliver the same information again. Treating every arrival as a new instruction would make transport uncertainty change business outcomes.

The question is therefore not only whether a request arrives, but what effect a repeated event may have. Our complete example processes an explicitly fictional counter. An event adds a small value and records a processing receipt. Repetition returns that receipt without adding again. No payment, order or genuine customer record is changed. The experiment isolates the relationship between event identity, stored effect and acknowledgement.

Real local HTTP execution includes parallel deliveries, a controlled failure before commit and a deliberately lost return path after commit. These cases expose the key distinction: a missing response does not establish whether the business effect occurred. Persisted state must let the next attempt answer that question without guessing from the transport result or creating a second effect merely because confirmation was unavailable.

Separate event identity from a delivery attempt

This teaching contract has an event identifier inside signed content and an additional identifier for the transmission attempt. Several attempts may carry one event. The event identifier governs business deduplication. If every new attempt automatically became a new instruction, legitimate redelivery could repeat the effect despite a valid signature. Authentication and stable business identity solve different problems and need separate representation.

An actual provider may define identifiers and redelivery differently. GitHub, for example, documents that requested redelivery retains its delivery identifier. Our freely chosen demo fields do not reproduce that interface. Before integration, establish which value has stable event meaning and which describes transport. Field names alone do not prove those semantics or allow a generic receiver to claim compatibility with every sender.

The example binds source identity to the configured credential rather than accepting a tenant field from event JSON. Receipts have a composite unique key of source and event identifier. Production with several providers or customer areas needs an authoritative mapping of its own. An identifier supplied in an arbitrary request must not create additional permission to choose another tenant, account or processing target.

GitHub: documented delivery identifiers

Authenticate the unchanged payload first

The receiver computes an HMAC over a fixed purpose marker, delivery identifier, time value and actual incoming body bytes. JSON parsing follows successful comparison. Reformatting content before checking would alter the agreed signature material. Sender and receiver need the same explicit representation, and the implementation uses Python’s existing cryptographic library instead of implementing a hash algorithm or inventing a hidden transformation.

This is a bounded original teaching format, not a compatible GitHub endpoint. GitHub’s documentation illustrates why real integrations must preserve original bytes and follow the provider’s specified verification method. Another provider may require different headers and rules. A self-selected demo field should not be presented under another service’s name as though an operational integration had already been completed and accepted.

The experiment generates its own short-lived 32-byte secret. It is neither published nor saved in the test results. Production needs managed secret delivery and protected transport. This local loop uses no live provider credentials and assesses our own acceptance and processing core, not the reachability or behaviour of an external webhook service. Those operational connections require their own evidence.

Bound bytes, time and the JSON contract

Before processing, the example limits the body to 8192 bytes and applies a deliberately chosen five-minute clock tolerance. Invalid authentication or unsuitable time values are rejected. That time window is not durable replay prevention: a correctly signed new attempt may still carry the same event. Stable event identity remains necessary even after authentication succeeds and the request appears sufficiently recent.

The decoder explicitly rejects duplicate keys and non-finite numeric constants. Content must contain exactly event identifier, kind and value. Only increment is supported. Value must be an integer from one to one hundred; a boolean does not count as an integer here, and a numeric string is not silently converted. Counterexamples actually exercise these distinctions instead of assuming ordinary JSON parsing enforces them.

These narrow rules describe the fictional counter. A real event interface can require more properties and kinds, but every supported kind needs its business contract. Unknown actions should not execute merely because their payload is syntactically valid JSON. The receiver must know the meaning it supports and the requests it visibly rejects, keeping structural acceptance separate from permission to perform a particular operation.

Commit effect and receipt in the same transaction

The core begins an immediate SQLite transaction before writing. Within it, the receiver checks for an existing receipt. If none exists, it updates the counter and inserts a receipt containing the body fingerprint and resulting value. Only then does it commit both together. Failure before completion rolls back the pair, leaving no counter change without its corresponding completed processing record.

The immediate write transaction serialises competing changes in this SQLite arrangement. Two concurrent requests therefore cannot independently observe an absent receipt and both apply the effect. The second observes the first completed state. A unique database key additionally expresses event identity. Actual parallel HTTP delivery checks the combination rather than relying on a sequential example that would never expose that race.

This boundary covers only the demonstrated local database effect. Sending mail or issuing a request to another service is not automatically part of the same transaction. Such additional work needs its own dependable handover, such as a deliberately developed outbox. A SQLite commit cannot guarantee that every independent remote system performs an operation once, nor turn a local receipt into universal exactly-once delivery.

SQLite: Transactions

Distinguish repetition from contradictory reuse

An identical repetition finds the same stored body fingerprint and returns the original outcome without modifying the counter. Tests deliver one event again under another transmission identifier. The existing receipt still governs its business effect. The number of transport attempts is not converted into a number of new instructions, preserving the intended meaning even when a retry is legitimate and authenticated.

Reusing an event identifier with different bytes produces a conflict. This intentionally strict teaching contract also treats semantically similar JSON with different whitespace as different bytes. It preserves the connection to the original payload. Real provider integration must establish whether retries are byte-identical or whether another explicitly defined comparison form is needed. Neither an ad hoc rewrite nor an undocumented normalisation should decide that silently.

Conflict differs from temporary processing failure. Blindly repeating the same contradictory content will not repair it. It requires investigation of identity and contract expectations. A real integration should make that case understandable and should not silently overwrite previously confirmed content with later input. The existing receipt remains authoritative in this example, providing a stable basis for resolving the discrepancy rather than hiding it.

Follow the complete processing core

The following Python module contains authentication, strict decoding, the event contract, transaction and processing receipts. Storage is a real database file, and every invocation opens its own connection. An outer HTTP receiver supplies validated headers and complete body bytes. The local fixture uses a small threaded server for that role but publishes no production endpoint or claim that general server deployment concerns are solved.

An optional callback after the counter update exists solely as a controlled test failure point. It comes from trusted fixture setup, not a request field, and deliberately fails before joint completion. A production integration does not need to expose that test mechanism publicly. Keeping test control separate from incoming event data ensures the experiment can exercise failure without granting arbitrary senders a new operational capability.

import hashlib
import hmac
import json
import re
import sqlite3

def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("Doppelter Schlüssel")
        result[key] = value
    return result

def reject_constant(value):
    raise ValueError("Ungültiger Zahlenwert")

class WebhookReceiver:
    def __init__(self, database, secret, after_update=None):
        if not isinstance(secret, bytes) or len(secret) != 32:
            raise ValueError("Demo-Schlüssel muss 32 Bytes lang sein.")
        self.database, self.secret = database, secret
        self.after_update = after_update
        with sqlite3.connect(database) as con:
            con.execute("CREATE TABLE IF NOT EXISTS counter(id INTEGER PRIMARY KEY CHECK(id=1), total INTEGER NOT NULL)")
            con.execute("INSERT OR IGNORE INTO counter VALUES(1,0)")
            con.execute("CREATE TABLE IF NOT EXISTS receipts(source TEXT NOT NULL, event_id TEXT NOT NULL, body_hash TEXT NOT NULL, result INTEGER NOT NULL, PRIMARY KEY(source,event_id))")

    def receive(self, raw, delivery_id, timestamp, signature, now):
        if not isinstance(raw, bytes) or len(raw) > 8192:
            return 413, {"error": "body_size"}
        if (not isinstance(delivery_id,str) or not re.fullmatch(r"[a-z0-9-]{1,64}",delivery_id)
            or not isinstance(timestamp,str) or not re.fullmatch(r"[0-9]{10}",timestamp)
            or not isinstance(signature,str) or not re.fullmatch(r"[0-9a-f]{64}",signature)
            or type(now) is not int or abs(now-int(timestamp)) > 300):
            return 401, {"error": "authentication"}
        material = ("demo-webhook-v1\n"+delivery_id+"\n"+timestamp+"\n").encode()+raw
        expected = hmac.new(self.secret, material, hashlib.sha256).hexdigest()
        if not hmac.compare_digest(expected, signature):
            return 401, {"error": "authentication"}
        try:
            event = json.loads(raw.decode("utf-8"), object_pairs_hook=unique_object, parse_constant=reject_constant)
        except (ValueError, UnicodeDecodeError, RecursionError):
            return 400, {"error": "invalid_json"}
        if (not isinstance(event,dict) or set(event) != {"event_id","kind","value"}
            or not isinstance(event["event_id"],str) or not re.fullmatch(r"evt-[a-z0-9-]{1,40}",event["event_id"])
            or event["kind"] != "increment" or type(event["value"]) is not int
            or not 1 <= event["value"] <= 100):
            return 422, {"error": "event_contract"}
        digest = hashlib.sha256(raw).hexdigest()
        source = "demo-source"  # Bound to this configured secret, never chosen by event JSON.
        con = sqlite3.connect(self.database, timeout=5, isolation_level=None)
        try:
            con.execute("BEGIN IMMEDIATE")
            old = con.execute("SELECT body_hash,result FROM receipts WHERE source=? AND event_id=?",(source,event["event_id"])).fetchone()
            if old:
                con.rollback()
                if old[0] != digest:
                    return 409, {"error": "event_conflict"}
                return 200, {"status": "duplicate", "result": old[1]}
            con.execute("UPDATE counter SET total=total+? WHERE id=1",(event["value"],))
            result = con.execute("SELECT total FROM counter WHERE id=1").fetchone()[0]
            if self.after_update:
                self.after_update(event["event_id"])
            con.execute("INSERT INTO receipts VALUES(?,?,?,?)",(source,event["event_id"],digest,result))
            con.commit()
            return 200, {"status": "applied", "result": result}
        except Exception:
            con.rollback()
            return 503, {"error": "temporary_processing_failure"}
        finally:
            con.close()

A stored outcome is not the latest aggregate state

The first event raises the counter to three. Another later raises it to seven. Repeating the first event still returns its original outcome of three. The receipt describes that event’s completed processing, not the latest aggregate value. Substituting the current counter would weaken the relationship between event and result and could make a legitimate retry appear to report a different historical outcome.

A real process may need another suitable receipt, such as a record identifier or a durable result status. Its meaning should remain stable. The receiver should not invent a fresh successful operation for every repetition. It confirms existing processing through its identity; a separate current-status read can provide later changes when needed. These are distinct questions that deserve distinct response meanings.

Receipt retention matters too. Deleting a receipt too early may let an old transmission appear new later. This example removes no receipts automatically. Production needs coordinated retention, redelivery windows and recovery. Backing up only the counter without its receipts would not preserve a complete consistent processing state for this contract. Restore procedures need to respect the relationship between both sets of information.

Actually roll back failure before commit

A controlled case fails after updating the counter but before inserting the receipt and committing. The HTTP receiver returns temporary failure. Independent database inspection finds the previous counter unchanged and no receipt for that event. The update step has not survived as a half-completed success. This is observed persisted state, not merely an assertion that a transaction API was invoked.

A subsequent attempt can process that event normally, creating the intended change and receipt. This differs from the conflict case where another payload was already confirmed under the same identifier. Temporary failure may justify another attempt; contradictory established identity requires investigation instead of an endless automated retry loop. Distinguishing those responses helps both processing logic and operators understand what can usefully happen next.

Actual response codes and retry strategy depend on the provider contract. This test fixture uses status 503 with Retry-After for its controlled temporary failure. It does not claim every external sender automatically follows that hint. Before release, review the chosen counterpart’s documented deadlines and redelivery rules, then test them through the real integration rather than transferring assumptions from the independent local sender.

Reproduce response loss after a successful commit

Another controlled case completes the transaction and only then closes the HTTP connection without sending a response. The caller observes a lost return path, but independent database inspection already finds the counter effect and receipt. Describing the entire request merely as failed would hide this distinction and invite unsafe assumptions about whether a new attempt should perform the business change again.

A real subsequent HTTP delivery of the same event finds the receipt, returns duplicate and makes no further counter change. The crucial retry case is therefore executed directly. Loss is simulated deliberately at a known point; this is neither a measured random Internet failure rate nor a general guarantee of flawless delivery across multiple third-party systems with independent states and retry policies.

The receiving process is then stopped and restarted with the same database file and the same test secret retained by the parent experiment. An earlier event remains a duplicate with its original receipt. The result consequently does not depend on a transient process-local set. Actual operational recovery must also preserve the relevant database and credential relationships through its intended restore mechanism and access controls.

Review 55 conditions across 35 HTTP requests

Execution uses Python 3.12.14 and SQLite 3.53.1. Fifty-five conditions cover first processing, identical repetition, changed content under the same identity, old receipts after later effects and concurrent delivery. Precommit failure, lost response after commit and process restart are included. Each relevant effect is independently read from the database rather than inferred solely from a successful-looking response returned by the receiver.

Additional requests contain incorrect signatures, unsuitable timestamps, malformed JSON, duplicate keys, invalid UTF-8, special numeric constants and incompatible field types. Unsupported kinds, additional properties, missing fields and oversized content are rejected. Those requests neither alter the business counter nor create a corresponding receipt. The controlled sequence ends at twenty-five, preserving the expected sum of the five distinct applied events.

The illustration uses actual results and a stored snapshot taken after deliberate response loss. No production provider was configured, no load test performed and no guarantee established for every infrastructure failure. Evidence concerns this small receiver and atomic effects within its local SQLite file. Keeping that boundary visible makes the demonstration useful without promoting it into an unsupported claim of universal delivery assurance.

Separate slow work from dependable acceptance

For longer tasks, a receiver may durably accept validated input and process it later. Its response must then mean exactly that durable acceptance. Keeping the event only in volatile memory while claiming successful takeover creates another contract when the process exits unexpectedly. The state actually promised to the sender needs a real persistence boundary, not merely a queued-looking interface or an asynchronous function call.

This small counter instead completes synchronously through database commit and needs no long-running background work. Production may require an inbox or queue, but its states, retries and handovers need separate review. Changing architecture does not automatically retain every property demonstrated here. Remote side effects deserve particular attention because they cannot simply be placed inside the local transaction shown in the example.

An operational view should distinguish completed, repeated, conflicting and temporarily failed events. Identifiers and bounded reasons are often more useful than uncontrolled full payload copies. What records are needed and appropriate depends on the actual task. This experiment stores only its fictional fixtures and outcomes, not third-party personal data, and does not provide a generic logging approval for an unrelated service.

Accept the actual provider integration against the same contract

Before making a real connection, obtain event names, verification rules, identifiers, deadlines and redelivery behaviour from current original documentation. Bind target authority to the correctly configured credential. Exercise accepted, invalid and repeated requests through the actual integration route. A functioning independent sender is a foundation, not evidence of compatibility with an external service whose headers and semantics may differ materially.

Also review changes and recovery. A receiver version requiring different fields may suddenly reject older retries. Restoring an earlier database may forget events already completed elsewhere. Such transitions belong in the operational contract rather than remaining surprises discovered during the next outage. Stored event identity connects the technical receipt to the business effect and helps make those transitions concrete and reviewable.

A dependable webhook flow separates arrival, checking, durable effect and acknowledgement. It expects repetition and exposes contradictory identities. The complete module and executed counterexamples provide a concrete way to understand those rules. The real application should then accept responsibility only for boundaries it has actually implemented and verified, including the provider connection, additional side effects and its recovery process.

Distinguish the tested failure locations

Retry-safe webhook checklist

• Derive stable identity from the actual provider contract.
• Verify signatures over the agreed original bytes.
• Bind source and target authority to authoritative context.
• Bound size, time, field types and supported event kinds.
• Commit business effect and receipt atomically.
• Distinguish identical retries from contradictory reuse.
• Preserve receipts as original outcomes.
• Actually test pre/postcommit failure and concurrency.
• Plan restart, restore and receipt retention together.
• Give additional remote effects their own dependable handovers.

Sources and review scope

Primary sources reviewed on 9 November 2026. Original fictional counter, 55 executed conditions and 35 HTTP requests with Python 3.12.14 and SQLite 3.53.1. Original demo signature format, no production provider endpoint or universal exactly-once guarantee for remote effects.

• Python: HMAC
• Python: sqlite3
• SQLite: Transactions
• GitHub: Validate Webhook Deliveries
• GitHub: Webhook Best Practices
• RFC 9110: HTTP Semantics