Fastly plugin crashes when backend variables change

[sethyates]

Terraform Version

Terraform v1.9.6
on linux_amd64
+ provider registry.terraform.io/carlpett/sops v1.1.1
+ provider registry.terraform.io/fastly/fastly v5.13.0
+ provider registry.terraform.io/hashicorp/archive v2.6.0
+ provider registry.terraform.io/hashicorp/google v5.43.1
+ provider registry.terraform.io/hashicorp/google-beta v5.44.0
+ provider registry.terraform.io/hashicorp/null v3.2.3
+ provider registry.terraform.io/hashicorp/random v3.6.3
+ provider registry.terraform.io/pagerduty/pagerduty v3.15.6

Affected Fastly Terraform Resource(s)

• fastly_service_vcl

Terraform Configuration Files

module "app" {
  source = "../../../modules/cdn_run_backend"

  name       = google_cloud_run_v2_service.tokens.name
  project_id = module.integration.project.project_id
  region     = local.region
  prefix     = "oauth"

  depends_on = [google_cloud_run_v2_service.tokens]
}

resource "fastly_service_vcl" "integration" {
  name               = local.connect_domain
  default_ttl        = 3600
  http3              = true
  stale_if_error     = false
  stale_if_error_ttl = 43200

  domain {
    name = local.connect_domain
  }

  dynamic "backend" {
    for_each = [module.app]   # <- have to remove the module.app when the provider crashes

    content {
      name                  = backend.value.name
      address               = backend.value.address
      override_host         = backend.value.override_host
      between_bytes_timeout = backend.value.between_bytes_timeout
      connect_timeout       = backend.value.connect_timeout
      first_byte_timeout    = backend.value.first_byte_timeout
      max_conn              = backend.value.max_conn
      port                  = backend.value.port
      use_ssl               = backend.value.use_ssl
      ssl_check_cert        = backend.value.ssl_check_cert
      ssl_cert_hostname     = backend.value.ssl_cert_hostname
      ssl_sni_hostname      = backend.value.ssl_sni_hostname
      min_tls_version       = backend.value.min_tls_version
      shield                = backend.value.shield
      request_condition     = backend.value.request_condition
    }
  }

  # more irrelevant stuff
}

Expected Behavior

What should have happened?

The plan should be planned or the apply should be applied.

Actual Behavior

What actually happened?

Any change whatsoever to the underlying Cloud Run service causes the Fastly provider to crash. I have to set for_each = [] to get the configuration to apply, then quickly revert and apply. This obviously causes outages because the backend goes away.

╷
│ Error: Request cancelled
│ 
│   with fastly_service_vcl.integration,
│   on cdn.tf line 12, in resource "fastly_service_vcl" "integration":
│   12: resource "fastly_service_vcl" "integration" {
│ 
│ The plugin.(*GRPCProvider).PlanResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-fastly_v5.13.0 plugin:

panic: value is unknown

goroutine 134 [running]:
github.com/hashicorp/go-cty/cty.Value.AsString({{{0xfc4158?, 0xc00026a27b?}}, {0xd48b80?, 0x1598d60?}})
        github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:1179 +0x111
github.com/fastly/terraform-provider-fastly/fastly.validateUniqueNames.func1({0xfc3a90?, 0xc000297890?}, 0x0?, {0xce7ea0?, 0xc00011a9b0?})
        github.com/fastly/terraform-provider-fastly/fastly/base_fastly_service.go:178 +0x171
github.com/hashicorp/terraform-plugin-sdk/v2/helper/customdiff.All.func1({0xfc3a90, 0xc000297890}, 0xe7e3df?, {0xce7ea0, 0xc00011a9b0})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/customdiff/compose.go:52 +0xe6
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc0003d84b0, {0xfc3a90, 0xc000297890}, 0xc000317040, 0xc00071cd70, 0xc000306e00, {0xce7ea0, 0xc00011a9b0}, 0x0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/schema.go:698 +0x4d4
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0xfc4238?, {0xfc3a90?, 0xc000297890?}, 0xc000317040, 0xd5b740?, {0xce7ea0?, 0xc00011a9b0?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/resource.go:990 +0xe5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0xc0003a26f0, {0xfc3a90?, 0xc000297770?}, 0xc000431c20)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/grpc_provider.go:858 +0xc52
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc000263720, {0xfc3a90?, 0xc000296cf0?}, 0xc0000a3900)
        github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/tf5server/server.go:825 +0x3fc
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0xe45d40?, 0xc000263720}, {0xfc3a90, 0xc000296cf0}, 0xc0000a3880, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:500 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000151800, {0xfc3a90, 0xc000296c60}, {0xfc8738, 0xc000002180}, 0xc0002fcfc0, 0xc0004a2210, 0x155a8c0, 0x0)
        google.golang.org/grpc@v1.63.2/server.go:1369 +0xe69
google.golang.org/grpc.(*Server).handleStream(0xc000151800, {0xfc8738, 0xc000002180}, 0xc0002fcfc0)
        google.golang.org/grpc@v1.63.2/server.go:1780 +0x106f
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        google.golang.org/grpc@v1.63.2/server.go:1019 +0x91
created by google.golang.org/grpc.(*Server).serveStreams.func2
        google.golang.org/grpc@v1.63.2/server.go:1030 +0x145

Error: The terraform-provider-fastly_v5.13.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Debug Output (optional)

Please provide a link to a GitHub Gist containing the complete debug output (see Terraform docs). Ensure any sensitive data is <REDACTED>.

https://gist.github.com/sethyates/a9fee5fb5bc85f3e739732437be6814e

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids (optional)

Is there anything atypical about your account or set up that we should know?

[saez0pub]

It seems to crash when the the backend name value is computed

[saez0pub]

Here is a full reproductible example:

$ cat <<EOF > main.tf
resource "random_string" "_" {
  length    = 9
  special   = false
  min_lower = 9
}

resource "fastly_service_vcl" "crash" {
  name               = "crash"

  domain {
    name = "test-fastly"
  }
  backend {
    address = "test"
    name    = random_string._.result
  }
}


provider "fastly" {
  api_key = "testing"
}

terraform {
  backend "local" {}
  required_version = ">= 1.5.0"
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = ">= 3.5.1"
    }
    fastly = {
      source  = "fastly/fastly"
      version = ">= 5.10.0"
    }
  }
}
EOF

$ terraform init
...
$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # random_string._ will be created
  + resource "random_string" "_" {
      + id          = (known after apply)
      + length      = 9
      + lower       = true
      + min_lower   = 9
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + numeric     = true
      + result      = (known after apply)
      + special     = false
      + upper       = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: Request cancelled
│
│   with fastly_service_vcl.crash,
│   on main.tf line 7, in resource "fastly_service_vcl" "crash":
│    7: resource "fastly_service_vcl" "crash" {
│
│ The plugin.(*GRPCProvider).PlanResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-fastly_v5.14.0 plugin:

panic: value is unknown

goroutine 53 [running]:
github.com/hashicorp/go-cty/cty.Value.AsString({{{0x1057bf438?, 0x1400024433b?}}, {0x1056812e0?, 0x105d819e0?}})
	github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:1179 +0x108
github.com/fastly/terraform-provider-fastly/fastly.validateUniqueNames.func1({0x1057bed70?, 0x140003da630?}, 0x0?, {0x10561f3c0?, 0x1400000e888?})
	github.com/fastly/terraform-provider-fastly/fastly/base_fastly_service.go:178 +0x13c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/customdiff.All.func1({0x1057bed70, 0x140003da630}, 0x1054776fb?, {0x10561f3c0, 0x1400000e888})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/customdiff/compose.go:52 +0xac
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0x140004244b0, {0x1057bed70, 0x140003da630}, 0x140003e8680, 0x14000794a50, 0x140002e2e00, {0x10561f3c0, 0x1400000e888}, 0x0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/schema.go:698 +0x3c4
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0x1057bf518?, {0x1057bed70?, 0x140003da630?}, 0x140003e8680, 0x105694140?, {0x10561f3c0?, 0x1400000e888?})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/resource.go:990 +0x9c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0x1400036e7c8, {0x1057bed70?, 0x140003da510?}, 0x140004b3ae0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/grpc_provider.go:858 +0xa54
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0x1400023d720, {0x1057bed70?, 0x14000271da0?}, 0x1400048c800)
	github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/tf5server/server.go:825 +0x2c4
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x1057821a0?, 0x1400023d720}, {0x1057bed70, 0x14000271da0}, 0x1400048c780, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:500 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0x14000195000, {0x1057bed70, 0x14000271d10}, {0x1057c3a58, 0x140004fc000}, 0x140003ceb40, 0x14000472210, 0x105d3d860, 0x0)
	google.golang.org/grpc@v1.63.2/server.go:1369 +0xc84
google.golang.org/grpc.(*Server).handleStream(0x14000195000, {0x1057c3a58, 0x140004fc000}, 0x140003ceb40)
	google.golang.org/grpc@v1.63.2/server.go:1780 +0xcd0
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/grpc@v1.63.2/server.go:1019 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func2
	google.golang.org/grpc@v1.63.2/server.go:1030 +0x160

Error: The terraform-provider-fastly_v5.14.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[sethyates]

Thanks @saez0pub. Fastly team, is there any update on fixing this?

[arianvp]

Running into this too:

╷
│ Error: Plugin did not respond
│ 
│   with fastly_tls_subscription.cache-staging,
│   on cache-staging.tf line 350, in resource "fastly_tls_subscription" "cache-staging":
│  350: resource "fastly_tls_subscription" "cache-staging" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with fastly_tls_subscription.cache,
│   on cache.tf line 302, in resource "fastly_tls_subscription" "cache":
│  302: resource "fastly_tls_subscription" "cache" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with fastly_service_vcl.channels,
│   on channels.tf line 108, in resource "fastly_service_vcl" "channels":
│  108: resource "fastly_service_vcl" "channels" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with fastly_tls_subscription.nixpkgs-tarballs,
│   on nixpkgs-tarballs.tf line 287, in resource "fastly_tls_subscription" "nixpkgs-tarballs":
│  287: resource "fastly_tls_subscription" "nixpkgs-tarballs" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may
│ contain more details.
╵

Stack trace from the terraform-provider-fastly_5.17.0 plugin:

panic: value is unknown

goroutine 541 [running]:
github.com/hashicorp/go-cty/cty.Value.AsString({{{0x1056d8ec8?, 0x1400000e631?}}, {0x10557f3c0?, 0x105d70000?}})
	github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:1179 +0x100
github.com/fastly/terraform-provider-fastly/fastly.resourceService.validateUniqueNames.func12({0x1056d8b68?, 0x140002b17d0?}, 0x140004bc040, {0x1054fd260?, 0x140003d2228?})
	github.com/fastly/terraform-provider-fastly/fastly/base_fastly_service.go:199 +0x12c
github.com/fastly/terraform-provider-fastly/fastly.resourceService.All.func15({0x1056d8b68, 0x140002b17d0}, 0x140004bc040, {0x1054fd260, 0x140003d2228})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/customdiff/compose.go:52 +0xa8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0x1400034fe30, {0x1056d8b68, 0x140002b17d0}, 0x14000840270, 0x14000867270, 0x1400033c800, {0x1054fd260, 0x140003d2228}, 0x0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/schema.go:698 +0x3b8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0x1056d8dd0?, {0x1056d8b68?, 0x140002b17d0?}, 0x14000840270, 0x140002b1800?, {0x1054fd260?, 0x140003d2228?})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/resource.go:990 +0x9c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0x1400033e7c8, {0x1056d8b68?, 0x140002b1710?}, 0x1400021d4a0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/schema/grpc_provider.go:858 +0xa18
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0x14000257b80, {0x1056d8b68?, 0x140002b0f00?}, 0x14000496e80)
	github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/tf5server/server.go:825 +0x2a8
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x105695200, 0x14000257b80}, {0x1056d8b68, 0x140002b0f00}, 0x14000496e00, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:500 +0x1c0
google.golang.org/grpc.(*Server).processUnaryRPC(0x140001a5000, {0x1056d8b68, 0x140002b0e40}, {0x1056dec60, 0x140004fe000}, 0x140004e47e0, 0x14000411c50, 0x105d38680, 0x0)
	google.golang.org/grpc@v1.63.2/server.go:1369 +0xb44
google.golang.org/grpc.(*Server).handleStream(0x140001a5000, {0x1056dec60, 0x140004fe000}, 0x140004e47e0)
	google.golang.org/grpc@v1.63.2/server.go:1780 +0xb08
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/grpc@v1.63.2/server.go:1019 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 20
	google.golang.org/grpc@v1.63.2/server.go:1030 +0x138

Error: The terraform-provider-fastly_5.17.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.