Proposal: Plan Validation mechanism for TaskRunner API

[theakshaypant]

Summary

In OpenFL TaskRunner API 1.8, secure aggregation was introduced with a limitation that it only worked with the WaitForAllPolicy straggler handling policy, and any misconfiguration wasn’t detected until runtime, leading to unnecessary reinitialization. To address this, a validation mechanism was introduced during the fx plan initialize step, and this proposal extends that mechanism using a structured validation manifest. This file defines compatibility rules and constraints for FL plans using triggers, requires, allowed, and forbidden keys, enabling early detection of invalid configurations. The system supports both single-file and modular multi-file formats for maintainability, ensuring that incompatible features are flagged upfront. While it supports only equality-based and AND-composite validations at initialization, it significantly improves plan reliability and user experience by enforcing constraints early in the federated learning workflow.

Motivation

In OpenFL TaskRunner API 1.8, secure aggregation was introduced with a constraint that it could only be used alongside WaitForAllPolicy straggler handling policy. Without any validations or verifications of the plan, if secure aggregation was enabled with any other straggler handling policy, an error would be raised only after the experiment is started; this would require the model owner to re-initialize and redistribute the plan with the appropriate value for straggler handling policy. As a stop gap measure, a plan verify method was introduced; when fx plan initialize command is run an error is raised if an incompatible straggler handling policy was used with secure aggregation.
Extending the same for other parameters in the FL plan, a need for validation of plan was identified before its distribution such that any incompatible features are not enabled together or any other similar constraints are met early in the FL process.

High-Level Design

Technical Details

In the current implementation, when fx plan initialize (Ref.) is called, the validation of plan happens when the plan is parsed. The proposal retains the existing flow and targets changes solely in its implementation.

validation.yaml

Given that it is hard (and long) to define all the constraints within a python script, we propose to add a validation.yaml file which is used to define and reference all the constraints for an FL plan.
This validation manifest serves two purposes:

• Reference for users to checks what features are incompatible or constraints for enabling a feature.
• Reference for openfl to validate the plan and raise an error if it is not valid.

Similar to the defaults directory, we introduce a openfl-workspace/workspace/plan/validations directory which would store all the constraints and compatibility checks for a plan.

$ pwd
/openfl/openfl-workspace/workspace/plan
$ tree 
.
├── defaults
│   ├── aggregator.yaml
│   ├── assigner.yaml
...
└── validations
    └── validation.yaml

File definition

The file is defined in a YAML format and contains a map of all validations that need to be done with the top level keys being an arbitrary string to only name the validation.
For each feature, we propose two ways to define the constraints.

• The constraints for each feature are defined in a single file

  secure_aggregation:
      ...
  feature_x:
      ...
  feature_y:
      ...

  • Pros:
    • Single file would be easier to lookup for users.
  • Cons:
    • The file can get big over a period of time and hard to lookup for users (or even maintain).
• We introduce a uses key word that is used to point to a reference file for definition of constraints for that feature

  $ cat validation.yaml
  secure_aggregation:
      uses: secure_aggregation.yaml
  feature_x:
      uses: feature_x.yaml
  feature_y:
      uses: feature_y.yml
  $ pwd
  /openfl/openfl-workspace/workspace/plan
  $ tree 
  .
  ├── defaults
  │   ├── aggregator.yaml
  │   ├── assigner.yaml
  ...
  └── validations
      ├── feature_x.yaml
      ├── feature_y.yml
      ├── secure_aggregation.yaml
      └── validation.yaml

  • The definition of the constraint in validation.yaml is ignored if uses keyword is present.
  • Pros:
    • The validation manifest remains small and easily maintainable.
    • If used in conjunction with well defined constraint names, users/maintainers can intuitively jump to the constraint file that want to check.
  • Cons:
    • The number of validation manifests can get high over a long period of time.

This proposal permits the use of both definitions.

Constraint definition

Apart from the uses key (for a separate file), we introduce the following keys for constraint definition.

• triggers: Map of triggers if met, the constraints are checked else no validation required for that feature.

  triggers:
      super_key.nested_key1.nested_key2.key1:
          trigger_value: value2
      key2:
          range: 
              - "> value3"
              - "< value4"
      key3:
          range: "value5 - value6" 

  • In the example above, we start validation for the constraint iff super_key.nested_key1.nested_key2.key1 == value2 && key2 > value3 && key2 < value4 && key3 >= value5 && key3 <= value6.
  • The usage of . in key indicates nested key in the plan so super_key.nested_key1.nested_key2.key1: would look like this in the FL plan.

    $ cat plan.yaml
    super_key:
        nested_key1:
            nested_key2:
                key1: value

  • The usage of . in the values does not mean anything and is matched as is with the value in plan.yaml.
  • trigger_value indicates the values for which the constraint check is done. Can be a string or a list.
  • range defines a range of values within which the constraint would be valid.

💡 Do we need a default_value for the trigger?
There are attributes which need not be specified in the plan.yaml and a default value is used for them.
Consider db_store_rounds in the Aggregator, it need not be given a value explicitly in the plan as there is value specified in the constructor = 1.
If there are constraints triggered by this db_store_rounds parameter, there is no way to validate them during plan initialization as there is a possibility of no value being set in the plan and the default value only copming into picture when Aggregator object is created at fx aggregator start which brings us back to the original problem of reinitization and redistribution of plan.
Specifying a default_value in the validation manifest takes care of these scenarios by using it for validation if no value is set in the plan.

At the same time, there is a default value specified in the defaults file for the same parameter = 2.
This leads to confusion about which default value is applied and highlights the need for a single source of truth for default values.

We should handle the duplication of default values in the constructor, defaults file and validation manifest.

• This can be done by keeping all the default values in defaults file and all the values being populated in the plan accordingly.

• We also eliminate the need for keeping the defaults in the validation manifest preventing duplication.

• As for the default values specified in the constructor, the named arguments can be changed to be used as keyword-only arguments. By doing so, the parameters

  • would receive the default value specified in the plan.
  • can still be used as a "named" argument
  • need not be passed in **kwargs

  def func(*, a, b):
      print(a)
      print(b)
  
  func("gg") # TypeError: func() takes 0 positional arguments but 1 was given
  func(a="gg") # TypeError: func() missing 1 required keyword-only argument: 'b'
  func(a="aa", b="bb", c="cc") # TypeError: func() got an unexpected keyword argument 'c'
  func(a="aa", b="bb", "cc") # SyntaxError: positional argument follows keyword argument
  func(a="aa", b="bb") # aa, bb
  

• requires: List of other constraints that need to be validated when the current definition is triggered.

  requires:
      - feature_x
      - feature_y

  • This implies if the triggers for the current constraint definition are met, feature_x and feature_y should be set to their trigger values and their respective constraints are also met.
  • This would be helpful in cases where a feature say feature_a requires for feature_x and feature_y to be enabled, instead of duplicating the constraints under feature_a, the constraints defined for x and y can be referenced here.
• allowed: Defines the list of keys that need to be set to certain values in order for the plan to be valid.

  allowed:
      super_key.nested_key.key1: value1
      key2:
          - value2
          - value3.value4

  • The defintion implies super_key.nested_key.key1 == value1 AND (key2 == value2 OR key2 == value3.value4).
  • As described earlier, only the key is assumed to be nested in case of "." being present in the string and not the value.
  • Value can either be a a single value or a list of values.
    • In case of a single value, it needs to be an exact match.
    • In case of a list, the key can have any one of the values in the list.
• forbidden: Defines the list of keys that cannot be set to certain values in order for the plan to be valid.

  forbidden:
      super_key.nested_key.key3: value5
      key4:
          - value6
          - value7.value8

  • Exactly opposite behavior of allowed in terms of validation.
  • The definition implies super_key.nested_key.key3 != value1 AND (key4 != value6 OR key4 != value7.value8).
  • As described earlier, only the key is assumed to be nested in case of "." being present in the string and not the value.
  • Value can either be a a single value or a list of values.
    • In case of a single value, it cannot be an exact match.
    • In case of a list, the key cannot have any one of the values in the list.
• range: Defines the range(s) in which the value can bet set.

  range:
      super_key.key1:
          - "> value1"
          - "< value2"
          - "value3 - value4"
  

  • The definition implies super_key.key1 > value 1 && super_key.key1 < value2 && super_key.key1 >= value3 && super_key.key1 <= value4.
  • As the earlier fields, the value can be passed as a single range or a conjunction of ranges.

💡 Why are both allowed and forbidden required?
Consider the secure aggregation requirement of having straggler handling as WaitForAllPolicy, it can be defined in two ways.

• Using allowed

  allowed:
     straggler_handling_policy.template:
         - openfl.component.WaitForAllPolicy
         - openfl.component.aggregator.WaitForAllPolicy
         - openfl.component.aggregator.straggler_handling.WaitForAllPolicy

• Using forbidden

  forbidden:
      straggler_handling_policy.template:
          - openfl.component.CutOffTimePolicy
          - openfl.component.aggregator.CutOffTimePolicy
          - openfl.component.aggregator.straggler_handling.CutOffTimePolicy
          - openfl.component.PercentagePolicy
          - openfl.component.aggregator.PercentagePolicy
          - openfl.component.aggregator.straggler_handling.PercentagePolicy

Depending on the values which are less (or easier to define), we can use either to define the constraints.
The two keys can also be used together such that

allowed:
   super_key.nested_key.key1: value1
   key2:
       - value2
       - value3.value4
forbidden:
   super_key.nested_key.key3: value5
   key4:
       - value6
       - value7.value8

This definition implies super_key.nested_key.key1 == value1 AND (key2 == value2 OR key2 == value3.value4) AND super_key.nested_key.key3 != value6 AND (key4 != value6 OR key4 != value7.value8)

Combining all the elements

For a single constraint, this is how the validation would look like with dependencies on constraints feature_x and feature_y.

secure_aggregation:
    triggers:
        aggregator.settings.secure_aggregation:
            trigger_value: true
    requires:
        - feature_x
        - feature_y
    allowed:
        straggler_handling_policy.template:
            - openfl.component.WaitForAllPolicy
            - openfl.component.aggregator.WaitForAllPolicy
            - openfl.component.aggregator.straggler_handling.WaitForAllPolicy
    forbidden:
        key1: forbidden1
        key2.key3: forbidden2

Incompatible features

• Secure aggregation: When enabled, straggler handling policy should be set to WaitForAllPolicy.
• Use Delta Updates: When enabled db_store_rounds should be set to greater than 1.
• Federated Evaludation: Number of rounds should be set to 1.
• Check for valid task names in the plans.
• Check for valid opt_treatment values.
• Flower compatibility.

Scope

• Only AND composites are allowed in the triggers.
  • Cannot define trigger1 OR trigger2, only trigger1 AND trigger2.
  • OR consitionals can be treated by defining separate constrain itself.
• Validations are limited to those which can be done at the time of plan initialisation.

Limitations

• Need to explicitly specify if there is a dependency on another constraint.

Open Questions

• Location of the validation.yaml file.
• Use multiple validation manifests or a single one.

Next Steps

1. Take a call if we need a validation manifest (name still needs some work on given a potential confusion with federated validation) or extend the current verify method to make it pure and add other incompatible features there.
2. Implementation of plan validation mechanism for parsing the validation manifest and performing the defined validations with only the secure aggregation constraint.

[teoparvanov]

As discussed offline, my suggestion would be to move validations.yaml in the core openfl package, as it isn't something that users are supposed to modify.

[theakshaypant]

While I understand your point, I do believe defaults and validations should be kept together. The point is valid for the yaml files in both these directories, users are not supposed to modify them.
If we want to move validation.yaml to the core openfl package, in my opinion, the same should be done for defaults as they belong together. Both of them are used for doing operations on the user defined plan (population and validation respectively) by core openfl and serve as a reference to the user if they stumble upon it.
Having said that, I propose to keep it in the same directory as defaults.

[teoparvanov]

@theakshaypant please also add more details regarding the need for the default_value field, and how you propose to tackle the ensuing duplication down the road.

[theakshaypant]

Updated the proposal.

• Added a note Why is default_value required in the trigger?
• Added a Next Steps section detailing how I propose we should handle the default value issue as well as the steps in which we should do the implementation of this proposal.

[MasterSkepticista]

Thanks for putting together this proposal @theakshaypant.

I get a feeling that plan validation is a disproportionately involved activity. I suggest limiting what we deliver first for the following reasons:

• The initial motivation was only to identify 'which combinations are invalid' - which is technically simpler than checking 'what combinations are valid' through allowed and required keys.
• Doesn't this method add burden on a contributor to also add entries to a file on 'what works'? In practice, features should work in conjunction with other features unless mentioned otherwise.
• We have one too many yamls to deal with, which we want to eliminate. Adding validation checklist files and the necessary code infrastructure around it doesn't help our goal of trimming configuration files.
• Ideally, all features must work orthogonally to other features. An extensive allow/deny/require list (partly) incentivizes development of half-baked features. This puts cognitive overhead on the user as well as the developers on 'how to make it work under invalid combinations'. In most cases, OpenFL should either fully support a feature, or not at all (i.e. we wait until we reach a design that is not narrow). Exceptions based on fundamental incompatibilities are indeed allowed if a feature author builds a case for it.

As of today, very few OpenFL features have (*) asterisks on them. It is OK in my opinion, to limit the scope of verify(...) to reject invalid combinations as stated by the feature authors. It means that unless stated, those combinations will be supported by OpenFL.

The current verify(...) function you have written is pretty readable and captures essence of what we want to validate, properly. It could benefit by making the function pure such that it takes a config and checks invalid combinations, raising very helpful error messages. In addition, we want this to be auto-documented on readthedocs, for which we have a pipeline (I can walk you through that). The RTD page will be a mirror of what the verify(...) function does.

Overall, my recommendation is for us to be cautious of the scope of verification, and keep it as simple as possible (it is already a great start with one function that user need not worry about). We have to be cautious as a team, that verify does not become a ground for enabling features in use cases that are parochial.

[theakshaypant]

Doesn't this method add burden on a contributor to also add entries to a file on 'what works'? In practice, features should work in conjunction with other features unless mentioned otherwise.

The idea behind this proposal is to address the "mentioned otherwise" feature set. Users will not necessarily refer the documentation in detail for a feature they are enabling. The lack of documentation for such features also supports the cause for having this validation mechanism (looking at use_delta_updates and db_store_rounds > 1 requirement).
In my opinion, it does not add a burden on a contributor as we do not expect an entry to be made for every feature that is implemented in OpenFL.
While I do understand your perspective, I believe your statements hold true if/when the validation manifest does not adhere to a standard and is being poorly maintained.

We have one too many yamls to deal with, which we want to eliminate. Adding validation checklist files and the necessary code infrastructure around it doesn't help our goal of trimming configuration files.

While I acknowledge the sentiment behind reducing the number of yaml files, I believe the validation manifest should not be bundled with pre-existing configuration files in this context. For any user, one might only reference the yamls for a single workspace and the others can be considered as a unnecessary. On the other hand, the validation manifest would impact (and be used for) any (and all) experiment irrespective of the other configuration files they may or may not have used (or referenced).

Ideally, all features must work orthogonally to other features. An extensive allow/deny/require list (partly) incentivizes development of half-baked features. This puts cognitive overhead on the user as well as the developers on 'how to make it work under invalid combinations'.

I would not agree that an allow/deny/require list incentivizes half-baked features. If anything, the absence of such list, hides these "shortcomings" from the user. In my eyes instead of looking at the manifest as 'how to make it work under invalid combinations', it should be looked as 'how to avoid invalid combinations'.
I would reiterate a point I made earlier, your point is true if/when the validation manifest does not adhere to a standard and is being poorly maintained.

In most cases, OpenFL should either fully support a feature, or not at all (i.e. we wait until we reach a design that is not narrow). Exceptions based on fundamental incompatibilities are indeed allowed if a feature author builds a case for it.

Agreed, 100%. However, with this validation manifest we are trying to address and monitor the exceptions that you mentioned.

Overall, my recommendation is for us to be cautious of the scope of verification, and keep it as simple as possible (it is already a great start with one function that user need not worry about).
We have to be cautious as a team, that verify does not become a ground for enabling features in use cases that are parochial.

Having said that, I like your suggestion of having a pure verify method and this proposal can be looked at as an extension of the same.

[MasterSkepticista]

Thanks for the detailed clarification @theakshaypant

• It wasn't apparent to me in your proposal, thanks for clarifying (that a user need not update the compatibility matrix on new feature adds)
• Not entirely convinced... It can be a simple key-value structure within the validation module itself which would act as a form of compatibility matrix. It may also allow validations to also happen when plans are modified after initialization. This is possible because all collaborators may agree on a plan value change after the experiment has been distributed. (In use cases outside the service)
• I think I agree with your perspective. But I am talking about a different effect. Ideally we do not want a compatibility matrix in the first place because all features work orthogonally - except those which cannot fundamentally work in conjunction, for which a validity check is beneficial. Example: Secure aggregation with dropouts requires reasonably large changes, it is not a fundamental limitation. Can we guess how long until straggler + secure aggregation validation check is eliminated?

Overall, we both do agree on having a thin verification check. As for implementation (yaml or not), I am OK with the consensus approach here