AWS plugin discovery scope excludes EKS::Cluster/Nodegroup, blocking recovery from partial-create bookkeeping failures

[mmassenzio]

Summary

When a formae apply --mode reconcile create partially fails due to a
transient plugin/API error on the Formae side (Cloud Control GetResource
polling timeout), AWS can finish creating the resource successfully while
Formae never records it as managed. For certain resource types (observed:
AWS::EKS::Nodegroup), there is then no supported recovery path back
into Formae's state — not even through the documented
discovery → extract → adopt workflow — because those resource types are
outside the AWS plugin's discovery scan scope.

Environment

• formae: 0.88.0
• AWS plugin: 0.1.16
• Resource type: AWS::EKS::Nodegroup (also affects AWS::EKS::Cluster)

Steps to reproduce

1. formae apply --mode reconcile a stack containing an AWS::EKS::Cluster
   with 3 AWS::EKS::Nodegroup resources.
2. Introduce (or wait for) a transient Cloud Control GetResource failure
   while polling create status for one or more node groups (in our case: a
   TLS handshake timeout against AWS Cloud Control). Agent logs:
   Plugin operator is missing in action after the timeout.
3. AWS finishes creating the node group(s) successfully in the background.
   Formae's command terminates Failed and marks those specific resources
   as failed creates; they are never added to Formae's managed inventory.
4. Confirm directly against AWS: the node group(s) are ACTIVE and healthy.
5. Confirm against Formae: list_resources/inventory resources for that
   stack shows only the node group(s) that finished polling before the
   timeout. The others are absent entirely (not listed as managed, not
   listed as unmanaged).
6. Trigger discovery explicitly (force_discover / wait for the scheduled
   scan). Query managed:false type:AWS::EKS::Nodegroup (or
   type:AWS::EKS::Cluster): zero results, even though the resources
   are live in the account discovery is configured against.

Root cause (as far as we can tell from the outside)

get_agent_stats shows the AWS plugin's discovery configuration includes an
explicit DiscoveryFilters allow-list:

AWS::EC2::Instance, AWS::EC2::SecurityGroup, AWS::EC2::NetworkInterface,
AWS::EC2::LaunchTemplate, AWS::AutoScaling::AutoScalingGroup,
AWS::EC2::VPCEndpoint, AWS::EC2::RouteTable, AWS::EC2::Subnet,
AWS::EC2::Volume, AWS::EFS::FileSystem, AWS::EFS::MountTarget,
AWS::IAM::Role, AWS::IAM::InstanceProfile,
AWS::ElasticLoadBalancingV2::LoadBalancer,
AWS::ElasticLoadBalancingV2::TargetGroup, AWS::Logs::LogGroup

AWS::EKS::Cluster and AWS::EKS::Nodegroup are not on this list, so the
plugin's List implementation for discovery never surfaces them as
candidates, regardless of how many times discovery runs. This looks like it
was scoped for the "catch out-of-band/foreign resources typically created
by autoscaling or consoles" use case, not for "recover a resource this same
plugin just created but failed to persist."

Impact

This directly undermines the tool's stated core differentiator ("formae
automatically discovers and update resources ... keeps the infrastructure
code automatically in sync with reality") for exactly the failure mode
where it matters most: a partially-successful apply. Once this happens for
an EKS cluster/node group, the operator is left with:

• no way to adopt the orphaned resource back into Formae state through any
  documented workflow, and
• a formae apply --mode reconcile of the original forma that will very
  likely attempt to re-create a resource with a name that already exists in
  AWS (untested exact error, but the resource identifier is a name that
  must be unique per cluster), rather than detecting and adopting it.

Suggested fixes (any of these would help)

1. Add AWS::EKS::Cluster and AWS::EKS::Nodegroup (and likely other
   "day-0 declared, not typically out-of-band" resource types) to the AWS
   plugin's default discovery filter, so partial-create bookkeeping
   failures are always recoverable via the existing discovery/adopt flow.
2. Harden the Cloud Control polling path so a transient
   network/TLS error while checking create status doesn't mark an
   in-flight create Failed outright — e.g. retry/backoff before giving up,
   or re-verify actual cloud state via a fresh GetResource/describe call
   before recording failure.
3. Provide an explicit "resync/repair a specific failed resource" command
   or MCP tool that re-checks cloud reality for a named failed resource
   from a specific command, independent of the discovery scan scope, and
   offers to adopt it if found.

Happy to share more detail (agent logs, command IDs) if useful — trimmed
account-specific identifiers here since this is a public repo.

[JeroenSoeters]

Thanks for reporting this @mmassenzio.

Would you mind sharing the output of formae status agent --output-consumer machine --output-schema json and could you grep the logs for any errors during discovery:

grep -aE "Starting resource discovery|Discovery: no plugin|No discoverable targets|Skipping non-discoverable|EKS::(Cluster|Nodegroup)|Missing parent property|Validation of required fields failed|Failed to list resources" \
   ~/.pel/formae/log/formae.log \
   | sed -E 's/\x1b\[[0-9;]*m//g' | tail -300

If that cycle has already rotated out, the same grep over gzip -dc ~/.pel/formae/log/formae.log.*.gz will find it.

Root cause (as far as we can tell from the outside)

The DiscoveryFilters is a deny-list, it is there to filter out resources that EKS itself manages as those by definition shouldn't be managed by formae.

Harden the Cloud Control polling path

We already have some hardening on the status-poll path in the pipeline, this should go out early next week with the next AWS plugin. However it doesn't explain why your agent cannot discover the cluster or the node group in your environment.

Provide an explicit "resync/repair a specific failed resource"

This is unfortunately not a trivial thing to do and not something that can be generalized across plugins. The problem is that when the initial response after the create fails we never store the native ID which is the key we synchronize resources on. It is on our radar though.

[mmassenzio]

Hi @JeroenSoeters - thanks for jumping on this so quickly!
Logs attached (there is a LOT of stuff, and I had to do some sanitization, hope they are somehow useful)

Worth noting that I'm a complete newbie to formae, so there may be obvious issues that I'm missing.
To add some more detail: once I figured out we couldn't "adopt" the NodeGroups that had been successfully created, I simply deleted them, and re-ran formae apply, and it all worked just fine.

I must say I was really impressed as to how I was able to quickly create/provision clusters across several regions (with some home-made scaffolding, but still, compared to TF, not even close).

formae-errors.log

[JeroenSoeters]

Thanks @mmassenzio, the logs are useful and they rule out the discovery-scope theory. I can see formae scanning AWS::EKS::Cluster and then AWS::EKS::Nodegroup with the expected ClusterName, so DiscoveryFilters was never blocking these.

The later us-east-2 scan does find your cluster, but by then you had already deleted the orphaned node groups and reapplied, so the logs don't cover the window we'd need to see.

What went wrong is the transient Cloud Control timeout. It left formae's status check waiting long enough that the agent gave up on an operation AWS then went on to finish successfully, so the node group's native ID was never recorded. This native ID is how formae tracks a resource, so the create was marked failed even though the node group was live in your account. The next AWS plugin release fixes both the case where a slow API response stalls the status check, and fixes that a completed create is always recorded with its native ID.

I'll keep this issue tied to these fixes and update it when the release is available.

Glad to hear you're enjoying formae! We've been using formae to manage our entire production estate for quite a while now and I wouldn't want to go back 😊

[mmassenzio]

@JeroenSoeters - thanks again for the extensive explanation, appreciated.
Yes, please let me know when I can try this again.

Having just gone through "terraform hell" (again!) I am really keen to adopt Formae as our main infra provisioning framework.