The retry, step by step

How the checkout's send() grew from one attempt to a retry that waits, spreads out and gives up on the right errors, one change at a time.

Jonas BergPayments team6 November 20263 min read

The function that sends an order to the payment provider is twelve lines long today. It took four changes to get there, and each one fixed a night we remember.

retry.go

One attempt

func send(order Order) error {
	resp, err := client.Post(url, order)
	if err != nil {
		return err
	}
	return check(resp)
}

One attempt: a dropped connection loses the order.

Retry five times

func send(order Order) error {
	var last error
	for try := 1; try <= 5; try++ {
		resp, err := client.Post(url, order)
		if err == nil {
			return check(resp)
		}
		last = err
	}
	return last
}

Five tries, back to back.

Back off

func send(order Order) error {
	var last error
	for try := 1; try <= 5; try++ {
		resp, err := client.Post(url, order)
		if err == nil {
			return check(resp)
		}
		last = err
		time.Sleep(backoff(try))
	}
	return last
}

Wait 0.2 s, 0.4 s, 0.8 s… before the next try.

Only what can recover

func send(order Order) error {
	var last error
	for try := 1; try <= 5; try++ {
		resp, err := client.Post(url, order)
		if err == nil {
			return check(resp)
		}
		if !isNetwork(err) {
			return err // a decline is final
		}
		last = err
		time.Sleep(backoff(try) + jitter())
	}
	return last
}

Declines are final: never retry a refused card.
Jitter spreads the retries of a thousand orders.

The same rule in the .NET gateway#

The courier app talks to payments through a small gateway written in C#. It uses the same rule, through a retry policy instead of a loop.

PaymentClient.cs - Gateway

public sealed class PaymentClient
{
    private static readonly ResiliencePipeline Retry = new ResiliencePipelineBuilder()
        .AddRetry(new() { MaxRetryAttempts = 4, BackoffType = DelayBackoffType.Exponential, UseJitter = true })
        .Build();

    public Task<Receipt> SendAsync(Order order) =>
        Retry.ExecuteAsync(async ct => await _http.PostAsJsonAsync<Receipt>("/charge", order, ct)).AsTask();
}

Four retries, growing waits, with jitter.

Gateway.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Polly.Core" Version="8.4.1" />
  </ItemGroup>
</Project>

Proving it before Thursday#

The test that guards the rule runs a hundred times before every release.

checkout

go test ./payments -run TestRetryGivesUpOnDeclines -count=100
ok  	northwind/payments	3.214s