Skip to content

Added $filters to AAS QL & enhanced Fragment Fieldidentifiers#532

Open
Martin187187 wants to merge 1 commit intoadmin-shell-io:IDTA-01002-3-2_workingfrom
Martin187187:filter
Open

Added $filters to AAS QL & enhanced Fragment Fieldidentifiers#532
Martin187187 wants to merge 1 commit intoadmin-shell-io:IDTA-01002-3-2_workingfrom
Martin187187:filter

Conversation

@Martin187187
Copy link
Copy Markdown
Collaborator

@Martin187187 Martin187187 commented Feb 10, 2026

  • implements Add optional FILTER to Query #517
  • added new Query Language Filter chapter in Query Language chapter.
  • Enhanced the FieldIdentifier Fragment Definition from just plain string to a BNF specification. => allow non leaf paths and disallow required paths.
  • updated Json schema & api accordingly.
  • Addition does not introduce breaking changes because it only adds new optional field
  • allow multiple filters instead of just 1 in the current security specification

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Feb 10, 2026

CLA assistant check
All committers have signed the CLA.

@Martin187187
Copy link
Copy Markdown
Collaborator Author

I would really like to remove the following section in the Json Security specification:

"FRAGMENT": {
          "type": "string"
        },
        "FILTER": {
          "$ref": "#/definitions/logicalExpression",
          "additionalProperties": false
        },
        "USEFILTER": {
          "type": "string"
        }, 

This is redundant because of the newly introduced FILTERLIST:

"FILTERLIST": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/SecurityQueryFilter"
          }
        } 

I didnt remove it to not cause any breaking changes.

Copy link
Copy Markdown

@mdanish98 mdanish98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Martin187187 ,

thanks a lot for this pull request.

I have reviewed this PR and provided the following review remarks mainly for naming inconsistency. Please note that the naming inconsistency might not be limited to what I pointed out, so please check other files and locations.

[source,json]
----
{
"condition": {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be either $condition or $formula depending on the final decision, as pointed out in another comment.

"pattern": "^id$"
},
"$condition": {
"$formula": {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some naming inconsistency:
Here in query-json-schema.json uses $formula (lines 750 & 761)
openapi.yaml still uses $condition (lines 802 & 812)
query-language.adoc also has $condition (e.g., line 690)

We should use either $condition or $formula to be consistent.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that $condition is the better choice, and I will update $formula accordingly. That said, the security layer currently relies on FORMULA, as defined in the specification, which introduces a naming inconsistency.

@@ -784,6 +801,13 @@ components:
pattern: ^id$
$condition:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Highlighting the naming inconsistency as explained in the above comment.

items:
$ref: '#/components/schemas/QueryFilter'
required:
- $condition
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Highlighting the naming inconsistency as explained in the above comment.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update all JSON examples that use $condition to use $formula. Also, I have noticed that some examples uses condition without a $ prefix, so please consider this also when you search and replace.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather use $condition. I there a reason why you prefer $formula?

Copy link
Copy Markdown

@mdanish98 mdanish98 Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't have any preferences, please use whatever makes sense and aligns with the AAS QL. This remarks is only to make sure whichever we decide, we use it everywhere and not a mix.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in Security in ACL it is FORMULA (and CONDITION) whereas in the query language only "condition" is used, there is no $formula, only a $condition in Part 2

@mdanish98
Copy link
Copy Markdown

Also, I am seeing some alerts from GitHub code scanning, please either fix it or dismiss it.

@Martin187187
Copy link
Copy Markdown
Collaborator Author

Hi @Martin187187 ,

thanks a lot for this pull request.

I have reviewed this PR and provided the following review remarks mainly for naming inconsistency. Please note that the naming inconsistency might not be limited to what I pointed out, so please check other files and locations.

Thanks alot for your review @mdanish98. I fixed the naming inconsistencies.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed a lot here because:

  • added Fragment Field Identifier that are used to express what to filter (different to normal field identifier
  • I noticed huge gaps between the json and bnf. I fixed the inconsistencies in the BNF so JSON schema has no breaking changes

@Martin187187
Copy link
Copy Markdown
Collaborator Author

I’ve updated this PR after identifying several inconsistencies and gaps between the JSON Schema and the BNF.

Specifically:

  • Aligned the BNF with the JSON Schema to ensure consistent terminology (e.g., avoiding different keywords for the same concept).
  • Improved and clarified the required logic in the JSON Schema.
  • Verified the updated schema against existing access rules and queries currently used in BaSyx to ensure compatibility and correctness.

If you’d like to review the test cases or see how I validated the schema, you can find them here:
https://github.com/Martin187187/basyx-go-components/tree/json-schema-tests/internal/common/security/schema_tests

Happy to provide additional test data or walk through the validation approach if needed.

"USEFORMULA": {
"type": "string"
},
"FRAGMENT": {
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a significant discrepancy between the security JSON schema and the API JSON schema. In the security specification, the fragment object is nested inside the filter object, but this is not the case in the API schema. I have now updated both documents to use the security version for consistency.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So is it a bug for 3.0.2?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it is fixed already

@Martin187187 Martin187187 marked this pull request as draft March 6, 2026 15:26
@Martin187187
Copy link
Copy Markdown
Collaborator Author

blocked by #548

@mdanish98
Copy link
Copy Markdown

mdanish98 commented Mar 16, 2026

Hi @Martin187187 , thanks for addressing the remarks, I will wait for the #548 to be concluded first and then resume the review.

@sebbader-sap
Copy link
Copy Markdown
Contributor

@mdanish98 #548 has been merged.

@sebbader-sap sebbader-sap requested a review from mdanish98 March 29, 2026 14:53
@sebbader-sap
Copy link
Copy Markdown
Contributor

@Martin187187 is the PR still in "draft" or now in the "ready for review" state?

@Martin187187
Copy link
Copy Markdown
Collaborator Author

@Martin187187 is the PR still in "draft" or now in the "ready for review" state?

I will have to resolve conflicts caused by the bugfixes. So it is currenlty only a draft.

@BirgitBoss
Copy link
Copy Markdown
Collaborator

Do we really add
documentation/IDTA-01002-3/modules/ROOT/pages/http-rest-api/test/query/test1.json
to the documentation?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably it would be better to include a .json file for better checks (same in Part 1 vor Value-Only Schema)

@BirgitBoss
Copy link
Copy Markdown
Collaborator

I recommend to add an annex with complete examples like in Security: https://industrialdigitaltwin.io/aas-specifications/IDTA-01004/v3.0.2/annex/text-access-rule-examples.html

"modelStringPattern": {
"FieldIdentifier": {
"type": "string",
"pattern": "^(?:\\$aas#(?:idShort|id|assetInformation\\.assetKind|assetInformation\\.assetType|assetInformation\\.globalAssetId|assetInformation\\.specificAssetIds\\[[0-9]*\\]\\.(?:name|value|externalSubjectId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?)|submodels\\[[0-9]*\\]\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))|\\$sm#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id)|\\$sme(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*)*)?#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|value|valueType|language)|\\$cd#(?:idShort|id)|\\$aasdesc#(?:idShort|id|assetKind|assetType|globalAssetId|specificAssetIds\\[[0-9]*\\]\\.(?:name|value|externalSubjectId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?)|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)|submodelDescriptors\\[[0-9]*\\]\\.(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)))|\\$smdesc#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)))$"

Check warning

Code scanning / QDJVMC

Typo Warning documentation

Typo: In word 'aasdesc'
"FragmentFieldIdentifier": {
"type": "string",
"pattern": "^(?:\\$aas#(?:idShort|id|assetInformation\\.assetKind|assetInformation\\.assetType|assetInformation\\.globalAssetId|assetInformation\\.specificAssetIds\\[[0-9]*\\]\\.(?:name|value|externalSubjectId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?)|submodels\\[[0-9]*\\]\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))|\\$sm#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id)|\\$sme(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*)*)?#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|value|valueType|language)|\\$cd#(?:idShort|id)|\\$aasdesc#(?:idShort|id|assetKind|assetType|globalAssetId|specificAssetIds\\[[0-9]*\\]\\.(?:name|value|externalSubjectId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?)|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)|submodelDescriptors\\[[0-9]*\\]\\.(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)))|\\$smdesc#(?:semanticId(?:\\.(?:type|keys\\[[0-9]*\\]\\.(?:type|value)))?|idShort|id|endpoints\\[[0-9]*\\]\\.(?:interface|protocolinformation\\.href)))$"
"pattern": "^(?:\\$aas#(?:idShort|assetInformation\\.assetType|assetInformation\\.globalAssetId|assetInformation\\.specificAssetIds\\[[0-9]*\\](?:\\.externalSubjectId(?:\\.keys\\[[0-9]*\\])?)?|submodels\\[[0-9]*\\](?:\\.keys\\[[0-9]*\\])?)|\\$sm#(?:semanticId(?:\\.keys\\[[0-9]*\\])?|idShort|id)|\\$sme(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*(?:\\.[A-Za-z](?:[A-Za-z0-9_-]*[A-Za-z0-9_])?(?:\\[[0-9]*\\])*)*)?(?:#(?:semanticId(?:\\.keys\\[[0-9]*\\])?|idShort|value|valueType|language))?|\\$cd#idShort|\\$aasdesc#(?:idShort|description|displayName|extension|administration|assetKind|assetType|globalAssetId|specificAssetIds\\[[0-9]*\\](?:\\.externalSubjectId(?:\\.keys\\[[0-9]*\\])?)?|endpoints\\[[0-9]*\\]|submodelDescriptors\\[[0-9]*\\](?:\\.(?:semanticId(?:\\.keys\\[[0-9]*\\])?|idShort|endpoints\\[[0-9]*\\]))?)|\\$smdesc#(?:semanticId(?:\\.keys\\[[0-9]*\\])?|idShort|endpoints\\[[0-9]*\\]))$"

Check warning

Code scanning / QDJVMC

Typo Warning documentation

Typo: In word 'smdesc'
<FragmentObject> <ws>
( ( "FORMULA:" <ws> <logicalExpression> <ws> ) | ( <UseFormula> <ws> ) )
( "FILTER:" <ws> <SecurityQueryFilter> )?
( "FILTERLIST:" <ws> ( <SecurityQueryFilter> <ws> )* )?

Check warning

Code scanning / QDJVMC

Typo Warning documentation

Typo: In word 'FILTERLIST'
"$ref": "#/definitions/SecurityQueryFilter",
"additionalProperties": false
},
"FILTERLIST": {

Check warning

Code scanning / QDJVMC

Typo Warning documentation

Typo: In word 'FILTERLIST'
Copy link
Copy Markdown
Collaborator Author

@Martin187187 Martin187187 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I resolved all conflicts. This is ready for final review now

"USEFORMULA": {
"type": "string"
},
"FRAGMENT": {
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it is fixed already

@Martin187187
Copy link
Copy Markdown
Collaborator Author

For review you can use https://regex101.com/ to verify regex changes and https://bnfplayground.pauliankline.com/ for bnf changes

@Martin187187 Martin187187 marked this pull request as ready for review April 7, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants