Adding retries to your API is a must to make your system more resilient. You can add them in database operations, communication with external apis and every other operation that might depend on a response of a third party.
Here's an example of a retry implementation for database transient errors:
1. First, let's code the retry function
The function receives the number of attempts, delay (interval between attempts) and the function which represents the operation you want to do.
func Retry(attempts int, delay time.Duration, fn func() error) error {
err := fn()
if err == nil {
return nil
}
if !IsTransientError(err) {
// log.Println("--- retry not needed")
return err
}
if attempts--; attempts > 0 {
time.Sleep(delay)
// log.Println("--- trying again...")
return Retry(attempts, delay*2, fn) // exponential backoff
}
return err
}
2. IsTransientError function
In this example I created a function that check if the operation is worth trying again based on the error.
Let's say the repository returns "sql.ErrNoRows", which is returned when the query indicates no rows were found for the select command. This error definitely does not need to be attempted again. That's why we're looking for errors that are transient, that might happen for a few milliseconds/seconds for network reasons for example.
func IsTransientError(err error) bool {
if err == nil {
return false
}
// example : deadlock postgres
if strings.Contains(err.Error(), "deadlock detected") {
return true
}
// closed connection
if strings.Contains(err.Error(), "connection reset by peer") {
return true
}
// network/postgres timeout
if strings.Contains(err.Error(), "timeout") {
return true
}
// Max connections exceeded
if strings.Contains(err.Error(), "too many connections") {
return true
}
return false
}
3. Using the function
Here's an example of retrying a database operation called in a login method in service layer:
func (s *UserService) Login(ctx context.Context, email, password string) (*UserOutput, error) {
var user *domain.User
err := retry.Retry(3, 200*time.Millisecond, func() error {
var repoError error
user, repoError = s.repo.GetUserByEmail(ctx, email)
if repoError != nil {
if repoError == sql.ErrNoRows {
return ErrUserNotFound
}
return repoError
}
return nil
})
if err != nil {
return nil, err
}
//... ... ...
Top comments (0)