Terraform schemas for CUE

[roman-mazur]

Hello,

I wanted to share an experiment I conducted and see if I can get some feedback on it.

I have been using CUE to write Terraform configurations for quite a while. And I always was thinking that it would be nice to get the shemas of Terraform resources and data sources as CUE constrains in my code, so I can detect confiration mistakes a bit earlier.

Terraform does a fine job of validating its config, but it happens after it's exported from the code in CUE (when I execute terraform plan on the exported infra-config.tf.json).

This month I got some time to seriously approach the idea. Terraform has the providers schema -json command that generates a JSON document that describes what resources are exposed by the providers used in the Terraform config.
https://developer.hashicorp.com/terraform/cli/commands/providers/schema

This JSON document is not a JSON schema, but a bit like JSON schema. So I wrote some code that gets the schema for a particular Terraform provider, iterates over resource definitions there, transforms it to a JSON schema document (using CUE), and finally cue imports the generated JSON schemas, so that we get CUE definitions for the provider.

Here's an example of aws_instance (a Terraform resource that describes an EC2 instance):
https://github.com/roman-mazur/cuetf/blob/main/aws/res/aws_instance_gen.cue

Given AWS provider has ~2K different resources and data sources, I generated a lot of code in CUE.

Not all of the transformations succeeded though... I got some structure cycle errors which I will report separately, but a very big part of the AWS provider was imported this way. I also imported the Helm provider and 118 (out of 129) definitions of the Cloudflare provider.

Once I got the AWS definitions imported, I tried a very small example to see it in use.

package examples

import "rmazur.io/cuetf/aws"

awsServer: {
	aws.#Terraform & {
		provider: aws: region: "us-east-1"

		resource: {
			aws_instance: my_server: {
				ami:           "some-ami"
				instance_type: "t2.micro"
			}
		}
	}

	// Adding other resources.
	resource: null_resource: a_thing: {}
}

Doing cue export -e awsServer here gives the Terraform config. However, executing this command on my laptop takes ~15s.

$ time cue -e awsServer
...
cue export -e awsServer  16.19s user 0.84s system 132% cpu 12.890 total

Which brings us to the point that using the AWS provider schemas in CUE today is not practical :)
Getting feedback from terraform plan after cue export without these schemas will be much faster.

I wonder if this example can be a good target for the ongoing work in the performance optmization. My dream would be to have this under 1s. Let me know if this can be useful. Maybe it makes sense to add these defs to the Unity project?

Kind regards,
Roman

[roman-mazur]

One of the examples of errors I get in this experiment can be reproduced with the following script:

cd $(mktemp -d) && ( \
  set -e
  git clone https://github.com/roman-mazur/cuetf
  cd cuetf
  git checkout 24ffc1905717fd2b7c190248d0721555f5050a48
  cd internal/jsonschema/test/github && cue eval -e debug
)

This sample tries to convert terraform schema of a form

["list", ["object", {
  branch: "string"
  path:   "string"
  some_list: ["list", ["list", "string"]]
}]]

into a JSON schema (using the linked transformation). But fails to do it.

From what I understand, I run into some problem with recursive definitions, but I'm not sure how to resolve it.
If I change some_list here to have an item type that is different from what was used before (for example a primitive like string or a map), the transformation succeeds. For example, from

["list", ["object", {
  branch: "string"
  path:   "string"
  some_list: ["list", ["map", "string"]]
}]]

I get

{
        "type": "array",
        "items": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "branch": {
                    "type": "string"
                },
                "path": {
                    "type": "string"
                },
                "some_list": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "additionalProperties": {
                            "type": "string"
                        }
                    }
                }
            }
        }
    }

In the script to reproduce you change the cue invocation to have -t case=ok to see the successful scenario.

[roman-mazur]

Reported here: #3524

[roman-mazur]

I recently got some time to continue this experiment, using CUE version 0.14.1.
The current tip of the main branch in the cuetf repo has a few CUE definitions generated from the data provided with terraform provider schema:

https://github.com/roman-mazur/cuetf

With the latest version of CUE and several adjustments in the transform script, the number of errors reduced drastically: Helm, Cloudflare, and GitHub definitions are fully imported (including resources and data sources), the number of errors for AWS, Google, and Elastistack is much smaller, and the root cause seems to be related to #3524. Details on what definitions give an error are in the import logs.

So far, the repo is useful enough for me to maintain infra definitions related to my personal projects.

It's published on the central registry, and can be used with

deps: {
	"github.com/roman-mazur/cuetf@v0": {
		v:       "v0.0.2"
		default: true
	}
}

in your module.cue (if anyone wants to give it a try)

A public usage example can be found here showing the AWS EC2 instance definition.

And interesting aspect: the registry web page gives an error for the module. I guess it's because of the large amount of code there. Importing it does not cause any problems though.

I would appreciate your thoughts on

• whether such modules (of this size) have the right for existence in the central registry
• the general idea on getting the CUE definitions for Terraform providers (probably there are other implementations, I didn't follow, sorry)
• the issue with this particular recursion in CUE (evaluator: recursive definition with disjunction gives inappropriate structural cycle error #3524)

[myitcv]

@roman-mazur thank you for sharing this update, and apologies for the delay in following up here.

In response to your specific points:

whether such modules (of this size) have the right for existence in the central registry

Generally speaking my answer to this would be "yes" but like https://pkg.go.dev had to rethink things for very large Go modules and packages, we might well need to do that same for CUE modules. I think a discussion on the approach/module layout is probably the quickest way to make progress here.

the general idea on getting the CUE definitions for Terraform providers

Honestly this is a great approach. Some time ago we demoed #2462. But this approach hacked home-spun definitions, rather than generating from Terraform modules in a structured way like you have done.

Your approach to publishing CUE versions of Terraform modules is very similar to our approach of publishing curated modules which have an upstream: https://cue.dev/getting-started/schema-library/. Our upstreams are mainly JSON Schema, but the same thinking applies.

the issue with this particular recursion in CUE

I believe @rogpeppe replied to that issue, and has been working with @mpvl to schedule a fix. I'll reply in that issue to see if there is a means of working around the problem in the short term.

Would be great to setup time to discuss the approach/structure of your generated output, in particular what module and package boundaries make sense etc. We have some experience of this from our work with curated modules, but would be great to get your thoughts coming from this Terraform example.

[roman-mazur]

Thanks for the feedback. I will be happy to discuss more - maybe we can schedule some time next week.

[myitcv]

Sounds great. I believe @dominikdm has been in touch to help with just this.