Skip to content

Relax JSON substrait parsing #166

@benbellick

Description

@benbellick

What happens?

The extension fails to load JSON which contains unrecognized fields. This means that the extension is inflexible to backwards compatible changes to the substrait spec, as exhibited in substrait-io/substrait-python#114

To Reproduce

Execute from_substrait_json against e.g.:

{
  "extensionUrns": [
    {
      "extensionUrnAnchor": 1,
      "urn": "extension:io.substrait:functions_arithmetic"
    }
  ],
  "extensions": [
    {
      "extensionFunction": {
        "functionAnchor": 1,
        "name": "add:i32_i32",
        "extensionUrnReference": 1
      }
    },
    {
      "extensionFunction": {
        "functionAnchor": 2,
        "name": "subtract:i32_i32",
        "extensionUrnReference": 1
      }
    }
  ],
  "relations": [
    {
      "root": {
        "input": {
          "project": {
            "input": {
              "read": {
                "baseSchema": {
                  "names": [
                    "value1",
                    "value2"
                  ],
                  "struct": {
                    "types": [
                      {
                        "i32": {}
                      },
                      {
                        "i32": {}
                      }
                    ]
                  }
                },
                "namedTable": {
                  "names": [
                    "numbers"
                  ]
                }
              }
            },
            "expressions": [
              {
                "scalarFunction": {
                  "functionReference": 1,
                  "outputType": {
                    "i32": {}
                  },
                  "arguments": [
                    {
                      "value": {
                        "scalarFunction": {
                          "functionReference": 2,
                          "outputType": {
                            "i32": {}
                          },
                          "arguments": [
                            {
                              "value": {
                                "literal": {
                                  "i32": 20
                                }
                              }
                            },
                            {
                              "value": {
                                "literal": {
                                  "i32": 5
                                }
                              }
                            }
                          ]
                        }
                      }
                    },
                    {
                      "value": {
                        "literal": {
                          "i32": 10
                        }
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        "names": [
          "sum_result"
        ]
      }
    }
  ],
  "version": {
    "minorNumber": 75
  }
}

OS:

Mac OS 15.7

Substrait-Extension Version:

substrait-python v0.24.2

DuckDB Version:

1.2.2

DuckDB Client:

Python

Have you tried this on the latest master branch?

  • I have

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

  • I have

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions