Terraform consistently failing to release locks after upgrade to 1.10.0

[kush-openai]

Terraform Version

Terraform v1.10.0
on darwin_amd64

Terraform Configuration Files

terraform {
  required_version = "1.10.0"
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "xxxxxx"

    workspaces {
      name = "ankush-test-0"
    }
  }
  required_providers {
    azuread = "~> 2.23.0"
    azurerm = "~> 3.109.0"
  }
}

# PROVIDER SETUP

provider "azurerm" {
  subscription_id = "xxxxx"
  features {}
}

Debug Output

Error: Error releasing the state lock

Error message: unable to unlock workspace while state version upload is
still pending
Lock Info:
  ID:        xxxxxxxxxxxxxxxxxx
  Path:      
  Operation: OperationTypeApply
  Who:       root@github-runner-xxx
  Version:   1.10.0
  Created:   2024-12-03 03:08:52.868010616 +0000 UTC
  Info:      

Expected Behavior

Release lock operation should be retries till the state version upload is complete

Actual Behavior

Error releasing the state lock

Steps to Reproduce

We were able to reproduce it consistently with

terraform plan-lock=false -out plan.out

terraform apply plan.out

This was also happening when plan was empty

Additional Context

We believe this is happening because of this change in 1.10+. This is from the release log of terraform enterprise version https://developer.hashicorp.com/terraform/enterprise/releases/2024/v202410-1

"""
Workspaces API unlock action will now return a 400 status instead of 503 when the latest state version is still pending, but only for Terraform CLI 1.10+ clients.
"""

Because a 400 is returned, the terraform client does not do any retry and fails immediately. This is the code where retry logic is configured :

go-tfe/tfe.go

Line 605 in f9d7888

func (c *Client) retryHTTPCheck(ctx context.Context, resp *http.Response, err error) (bool, error) {

. It only retries on 500+ error codes.

Cross posted here : hashicorp/terraform#36155