generate/hcl: support module calls through `source` keyword #130

tormath1 · 2021-04-23T19:12:04Z

In this PR, we add the support for module calls. It supports local, remote and Terraform registry ones.

The function moduleInstall is called recursively in order to walk through the ModuleCalls of each module to feed a ManagedResource slice. It stores a trace of each installed element to avoid infinite loops and modules are cached into $XDG_CACHE directory.

Then it loops over the ManagedResource slice with the current behavior.

A few notes:

I'd like to use in-memory FS rather than actual FS but go-getter needs to be updated consequently (Abstract the os access via interface hashicorp/go-getter#83: I started to work on it btw)
I still need to find a way to properly testing this :)

Output example using this stack: https://github.com/cycloid-community-catalog/stack-gke

$ ./inframap generate --hcl generate/testdata/stack-gke
strict digraph G {
        "google_container_node_pool.pools"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_node_pool.pools";
        "kubernetes_config_map.ip-masq-agent"->"google_container_cluster.primary";
        "kubernetes_config_map.ip-masq-agent"->"google_container_node_pool.pools";
        "null_resource.wait_for_cluster"->"google_container_cluster.primary";
        "null_resource.wait_for_cluster"->"google_container_node_pool.pools";
        "google_service_account.cluster_service_account"->"random_string.cluster_service_account_suffix";
        "kubernetes_config_map.kube-dns"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns"->"google_container_node_pool.pools";
        "google_container_cluster.primary" [ shape=ellipse ];
        "google_container_node_pool.pools" [ shape=ellipse ];
        "google_service_account.cluster_service_account" [ shape=ellipse ];
        "kubernetes_config_map.ip-masq-agent" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-namservers" [ shape=ellipse ];
        "null_resource.delete_default_kube_dns_configmap" [ shape=ellipse ];
        "null_resource.wait_for_cluster" [ shape=ellipse ];
        "random_string.cluster_service_account_suffix" [ shape=ellipse ];

}

Closes: #56, #54

xescugc

Are you also still working on adding the afero.FS implemtation into the TF lib?

xescugc · 2021-05-19T08:03:27Z

generate/hcl.go

+	managedResources := make(map[string]*configs.Resource)
+	for rk, rv := range mod.ManagedResources {
+		managedResources[rk] = rv
+	}


One thing we are doing right now on the State is check if we have to prefix a resource with the module because the same resource could exist on another module aws_instance.front could be in multiple modules.

We only do that when we have more than 1 module we should try to do the same as we'll find the same issue.

xescugc · 2021-05-19T08:10:39Z

generate/hcl.go

+		for _, vers := range meta.Versions {
+			v, err := version.NewVersion(vers.Version)
+			if err != nil {
+				return fmt.Errorf("unable to create version from string: %w", err)
+			}
+
+			if latest == nil || v.GreaterThan(latest) {
+				latest = v
+			}
+
+			if call.Version.Required.Check(v) {
+				if match == nil || v.GreaterThan(match) {
+					match = v
+				}
+			}
+		}


Comment a bit this logic.

generate/hcl.go

xescugc · 2021-05-19T08:16:34Z

generate/hcl.go

+		dst := path.Join(cachePath, name)
+		// TODO: we should add a logic to invalidate
+		// the cache
+		if _, err := os.Stat(dst); os.IsNotExist(err) {
+			client := &getter.Client{
+				Src:  src,
+				Dst:  dst,
+				Pwd:  pwd,
+				Mode: getter.ClientModeDir,
+			}


As we have versions, we can assume versions are immutable, so you could use a dst := path.Join(cachePath, fmt.Sprintf("%s-%s", name, version))

xescugc · 2021-05-19T08:18:09Z

generate/hcl.go

+	// fill the final map of managed resources
+	// using the config freshly loaded
+	for rk, rv := range m.ManagedResources {
+		(*mRes)[rk] = rv
+	}
+


Potentially prefix it with them module if more than one or we'll have collision.

xescugc · 2021-05-19T08:31:28Z

cmd/root.go

+		PersistentPreRun: func(cmd *cobra.Command, args []string) {
+			if debug {
+				logsOut = os.Stdout
+			}
+			log.Init(logsOut, debug)
+		},


I would prefer not to include TC for this, copy the part you need (activation of TF logs) and open an issue to add Debugging to IM (if we do not have one already)

xescugc · 2021-09-02T10:26:21Z

Ping @Thomas-lhuillier hehe.

Thomas-lhuillier · 2021-09-02T10:37:25Z

What do I have to do? 😅

tormath1 · 2021-09-02T11:38:33Z

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Anyways, thanks for the ping ! Besides the comments to address, I guess there is not so much effort to provide to get this PR land into a new release.

Are you also still working on adding the afero.FS implemtation into the TF lib?

Nope. It was quite a mess to bring afero.FS support to the lib IIRC.

Thomas-lhuillier · 2021-09-02T11:54:48Z

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Makes sense 👍 Well, this all seems out of reach to me, maybe after a couple months/years training

xescugc · 2021-09-06T15:24:06Z

Yes I think it was an autocomplete error from GH and I did not check haha the ping is for @tormath1 (ok if i do @th the first one is Thomas haha)

tormath1 added 5 commits April 23, 2021 20:55

testdata: add Terraform from stack GKE

aef5207

mod: add go-getter and terracognita

425ed5c

generate/hcl: fetch terraform config required modules

ccf8e6c

cmd/root: add debug option to display TF logs

6d58168

sum: go mod tidy

272e1f9

tormath1 mentioned this pull request Apr 23, 2021

generate/hcl: support module calls through source keyword #67

Closed

xescugc self-requested a review April 26, 2021 07:02

xescugc requested changes May 19, 2021

View reviewed changes

xescugc force-pushed the master branch 2 times, most recently from c0bd35f to d9632dd Compare September 3, 2021 07:54

tormath1 and others added 4 commits September 7, 2021 20:46

fixup! mod: add go-getter and terracognita

974fd5a

fixup! sum: go mod tidy

b386772

fixup! cmd/root: add debug option to display TF logs

8320f1e

fixup! generate/hcl: fetch terraform config required modules

0409a56

tormath1 mentioned this pull request Oct 11, 2022

hcl: support module sources #56

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

generate/hcl: support module calls through `source` keyword #130

generate/hcl: support module calls through `source` keyword #130

Uh oh!

tormath1 commented Apr 23, 2021

Uh oh!

xescugc left a comment

Uh oh!

xescugc May 19, 2021

Uh oh!

xescugc May 19, 2021

Uh oh!

Uh oh!

xescugc May 19, 2021

Uh oh!

xescugc May 19, 2021

Uh oh!

xescugc May 19, 2021

Uh oh!

xescugc commented Sep 2, 2021

Uh oh!

Thomas-lhuillier commented Sep 2, 2021

Uh oh!

tormath1 commented Sep 2, 2021

Uh oh!

Thomas-lhuillier commented Sep 2, 2021

Uh oh!

xescugc commented Sep 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

generate/hcl: support module calls through source keyword #130

Are you sure you want to change the base?

generate/hcl: support module calls through source keyword #130

Uh oh!

Conversation

tormath1 commented Apr 23, 2021

Uh oh!

xescugc left a comment

Choose a reason for hiding this comment

Uh oh!

xescugc May 19, 2021

Choose a reason for hiding this comment

Uh oh!

xescugc May 19, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xescugc May 19, 2021

Choose a reason for hiding this comment

Uh oh!

xescugc May 19, 2021

Choose a reason for hiding this comment

Uh oh!

xescugc May 19, 2021

Choose a reason for hiding this comment

Uh oh!

xescugc commented Sep 2, 2021

Uh oh!

Thomas-lhuillier commented Sep 2, 2021

Uh oh!

tormath1 commented Sep 2, 2021

Uh oh!

Thomas-lhuillier commented Sep 2, 2021

Uh oh!

xescugc commented Sep 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

generate/hcl: support module calls through `source` keyword #130

generate/hcl: support module calls through `source` keyword #130