How to remove null values?

[mario-steinhoff-gcx]

Hello,

I use cue for kubernetes configuration and want to render helm charts via helm template and validate the resulting manifests against the cue-based kubernetes schema.

Rendering the template works and I can import manifests from helm stdout via yaml.UnMarshalStream().

Some helm charts (argocd in this example) produce YAML with empty keys because they are using this pattern, where the field itself is not additionally guarded by an if block:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  [...]
  annotations:
    {{- range $key, $value := .Values.server.metrics.serviceMonitor.annotations }}
    {{ $key }}: {{ $value | quote }}
    {{- end }}

Which renders to:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-server
  labels:
    helm.sh/chart: argo-cd-5.14.1
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/component: server
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
  annotations:
spec:
  [...]

After I run it through yaml umarshalling and marshalling like this:

command: helm: {
  helm: exec.Run & {
    stdin:  helmCommand.out.stdin
    cmd:    helmCommand.out.command
    stdout: string
  }
  helmTemplateResult: {
    for obj in yaml.UnmarshalStream(helm.stdout) {
      kubernetes: "\(obj.kind)": "\(obj.metadata.name)": obj
    }
  }
  print: cli.Print & {
    text: yaml.Marshal(helm.helmTemplateResult)
  }
}

I get this result:

kubernetes:
  ServiceMonitor:
    argocd-server:
      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        name: argocd-server
        labels:
          helm.sh/chart: argo-cd-5.14.1
          app.kubernetes.io/name: argocd-server
          app.kubernetes.io/instance: argocd
          app.kubernetes.io/component: server
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/part-of: argocd
        annotations: null
      spec:
        [...]

But the generated cue schema for annotations looks like this:

// Annotations is an unstructured key value map stored with a resource that may be
// set by external tools to store and retrieve arbitrary metadata. They are not
// queryable and should be preserved when modifying objects.
// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations
// +optional
annotations?: {[string]: string} @go(Annotations,map[string]string) @protobuf(12,bytes,rep)

And of course null is not a struct with string fields so it doesnt validate in cue but its still accepted by the kubernetes API server.

In jsonnet there is std.prune() to remove null fields, is there a way to solve this with cue?

[mario-steinhoff-gcx]

Building upon this comment, I came up with this:

#isValueStruct: {
	value:  _
	result: ((value & {}) | *_|_) != _|_
}

#isValueNull: {
	value:  _
	result: ((value & null) | *_|_) != _|_
}

#OmitNull: {
	orig: {}
	result: {...}
	result: {
		for k, v in orig {
			let valueIsStruct = (#isValueStruct & {value: v}).result
			let valueIsNull = (#isValueNull & {value: v}).result
			if valueIsStruct {
				"\(k)": (#OmitNull & {orig: v}).result
			}
			if !valueIsStruct && !valueIsNull {
				"\(k)": v
			}
		}
	}
}

Example: https://cuelang.org/play/?id=UHbZl6jGCRQ#cue@export@cue

Unfortunately when applied to my example argocd helm chart, it adds around 2 seconds to the execution time, e.g. without omitting nulls:

cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   5.40s user 0.51s system 151% cpu 3.905 total
cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   5.48s user 0.48s system 157% cpu 3.778 total
cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   5.41s user 0.49s system 151% cpu 3.899 total

with omitting nulls:

cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   7.40s user 0.60s system 149% cpu 5.343 total
cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   7.79s user 0.63s system 147% cpu 5.715 total
cue cmd -t cluster=cluster -t app=argocd -t workload=app  ./... >   7.58s user 0.59s system 150% cpu 5.414 total