Skip to content

Adding new checking rulesets

Jump to bottom
Szeto edited this page Jun 11, 2025 · 36 revisions

New metadata checking rules (checks) are developed in this repository before being moved upstream into ukf-meta. These checks are developed alongside tests designed to exercise and validate them.

In the early stages of development, the testbed serves as a playground for experimenting with new checks. Once developed and tested, checks are contributed back to ukf-meta. After this point, the only contributions to the testbed will be the new tests, which help to build an ever-evolving and comprehensive test suite.

Testbed structure

The testbed directories are structured in the following way to support development:

Directory Description
validators/overlays/development/classes/* Directory to add new checks (e.g., XSL check files), instantiate those checks as beans, and define them as part of the pipeline stages (checks) to run. Effectively a playground to help develop new checks. Everything in the classes directory ends up on the project's classpath at runtime
tests/* Directory that holds the tests associated with the checks under development. These tests are contributed back to the testbed

The testbed includes the following file to support development:

File Description
validators/overlays/development/classes/development-stages.xml Spring beans file to define new check beans and add them to the list of stages to run

Development of new metadata checks

The following steps describe the process for developing new checks and associated tests using the testbed. Each step is described in more detail in the sections that follow:

  1. Create a new topic branch for your development work.
  2. Add any new metadata checks.
  3. Add new bean definitions for the new checks and reference these beans in the development pipeline.
  4. Add new tests.
  5. After completing the development of the checks and tests, contribute the new checks to ukf-meta.
  6. Update the testbed's main branch to be consistent with the new checks in ukf-meta
  7. Rebase the topic branch on top of the main branch and commit new tests to the topic branch.
  8. Merge tests from the topic branch into the main branch.

1. Create a new topic branch

Create a new topic branch:

git checkout -b my-new-check

In the final state of this branch, the merge back into main will include either one or both of the following:

  1. new tests
  2. updates to existing tests

Any new checks you are developing (e.g., new XSL checks) will be contributed upstream into the ukf-meta project and will not be merged back into the main branch

2. Add new checks

Currently, it is expected that the rules for a new check will be encoded using XSL. In the future, additional checks may be developed in this manner, such as a new Java stage that necessitates the comprehensive, holistic, testing than the testbed provides.

Using XSL as an example, add any new XSL metadata check files directly into the validators/overlays/development/classes directory. For example:

validators/overlays/development/classes/check_mdattr_ec_anonymous.xsl

3. Add bean definitions and reference in the development pipeline

In validators/overlays/development/classes/development-stages.xml add any bean definitions that represent the new checks. For example, to add a new bean definition that references an XSL check:

<bean id="check_mdattr_ec_anonymous" parent="mda.XSLValidationStage"
    p:XSLResource="classpath:check_mdattr_ec_anonymous.xsl"/>

And reference the bean in the development_stages composite stage:

<bean id="development_stages" parent="mda.CompositeStage">
    <property name="stages">
        <util:list>
            <ref bean="check_mdattr_ec_anonymous"/>
        </util:list>
    </property>
</bean>

Running the testbed at this point would exercise the new checks over the existing suite of tests (see the README for how to run the testbed).

Override existing beans

development-stages.xml is a valid place to override existing beans and modify their functionality for testing purposes. Additionally, you can copy existing XSL checks into validators/overlays/development/classes/ to test any modifications or required changes. You will need to create a new bean that references the new check. When overriding a bean, you do not need to include that bean in the development_stages composite stage; this will cause its execution to trigger twice: once from the original list of stages and once from the development_stages.

For example, if you wanted to override the check_saml_strings stage and add a new element to check, you would simply override the bean in development-stages.xml:

<bean id="check_saml_strings" parent="mda.SAMLStringElementCheckingStage">
        <property name="elementNames">
            <set>
                <ref bean="md-Company"/>
                ...
                <ref bean="shibmd-Scope"/>
                <!-- New Element Checks Here -->
                <ref bean="new-element-check"/>
            </set>
        </property>
    </bean>

Please note that the individual check you are modifying may be part of a composite check. If that is the case, you will need to copy the composite check bean into development-stages.xml in addition to the bean for the check you are updating. This step is necessary for Spring to properly recognise your test.

You may need to change the reference of your import with prefix classpath: if needed

<xsl:import href="classpath:_rules/check_framework.xsl"/>

Breaking existing tests

Tests often resemble integration tests more than they do unit tests. This means that new metadata checks may cause existing tests to fail, not just those currently being developed. When this happens, and the failure does not indicate a need for adjustments (i.e., the new checks are expected to identify these issues), you should update the yaml options file for any affected tests to correspond to the new expected outcome based on all checks. For example, the options file can contain more than one expected error (again, see Test Options for more options):

expected:
  - status: error
    component_id: checkSchemas
    message: 'cvc-complex-type.2.4.a: Invalid content was found starting with element
      ''{"urn:oasis:names:tc:SAML:metadata:ui":DiscoHints}''. One of ''{"urn:oasis:names:tc:SAML:2.0:metadata":KeyDescriptor' is expected.'
  - status: error
    component_id: check_mdui_xslt
    message: DiscoHints appearing outside Extensions element

Occasionally, a new check or test may reveal issues with existing checks or tests that requires consideration before it can be resolved. This is a normal part of the process and contributes to greater confidence in the overall test results.

Retesting after update

Any modifications made to the validators/overlays/development/classes/ directory will necessitate rebuilding the testbed for them to take effect. (see the README for how to build and run the testbed)

4. Add new tests

Tests should exercise the new checks being developed to verify the logic operates as expected. Tests are nearly always SAML EntityDescriptor elements (for now, other types are not supported). The example in Example Test EntityDescriptor can be cut and paste as a basis for your test. SAML Metadata tests are located inside tests/xml/, organized in subdirectories that represent the rules being tested. For instance, entity category rules can be created in:

tests/xml/entity-categories/<specific-test>

Tests should be created for both good and bad metadata variants. Good tests should pass all checks currently defined in the testbed and should provide a canonical example of elements or attributes that are being validated by the check. On the other hand, bad tests should be designed to trigger validation rules in the check, for example, either an error, info, or warning result. Each of the bad tests should be accompanied by a sidecar options file (in YAML format) that describes the test options along with the expected outcomes. For more information, please refer to the section on Test Options. For example, a test directory may contain the following files:

tests/xml/entity-categories/anonymous/good_sp.xml
tests/xml/entity-categories/anonymous/bad_sp_no_dispname.xml
tests/xml/entity-categories/anonymous/bad_sp_no_dispname.yaml

With the contents of the YAML file for the bad test:

expected:
  - status: error
    component_id: check_mdattr_ec_anonymous
    message: SP asserts anonymous entity category but has no DisplayName element.

5. Contribute the new metadata checks to ukf-meta

At the point checks are ready to contribute to ukf-meta, the topic branch will likely contain the following changes (the final destination of the change is also specified):

Change Destined for testbed's main branch Destined for ukf-meta's main branch
New metadata checks in validators/overlays/development/classes/ No Yes
New development beans and stages in validators/overlays/development/classes/development-stages.xml No No
New tests in tests/ Yes No

The new checks will need to be contributed upstream to ukf-meta. This typically involves copying the new XSL checks into either ukf-meta/mdx/_rules or ukf-meta/mdx/uk, and enabling the checks by referencing them in the appropriate pipeline stage. You should follow the established process for making changes to ukf-meta — this includes adding the new files to a topic branch, requesting a review, and finally merging the branch into the main branch of ukf-meta.

6. Update the testbed to include new checks from ukf-meta

Once the new checks have been committed to ukf-meta, the main branch of the testbed needs to be updated to pull in those checks from upstream into the validators/overlays/ukf-meta-prod overlay. To do this, on the main branch of the testbed run the script:

ukf-testbed/import-prod

Then, commit and push those changes:

git commit -m "Update to current ukf-meta production branch"
git push

7. Rebase the topic branch on top of main and commit the tests

Rebase the topic branch on top of the main branch:

git checkout main
git pull
git checkout <topic-branch>
git rebase main

Assuming the rebase was successful, on the topic branch:

  1. Add the new checks pulled in from ukf-meta to default_validator_stages in validators/overlays/all/default-validator-stages.xml.
    • These should be the checks which were previously added to development_stages in validators/overlays/development/classes/development-stages.xml. This will enable those checks for the testbed.
  2. reset the validators/overlays/development/classes/ directory back to what it was before development work began. That is:
    • Remove any XSL files
    • Remove any new bean definitions in development-stages.xml and remove references to those beans in the development_stages pipeline.

You should then make the following commits:

  1. commit the new tests in tests/ as one commit.
  2. commit the addition of those tests to the default testbed pipeline in validators/overlays/all/default-validator-stages.xml as the final commit.

This leave two commits on the topic branch, one that adds the tests and one that enables those tests in the testbed. All the work developing checks is discarded from the testbed as it now exists in ukf-meta. For example, the following commits would be expected:

Commit No. Change Description
1 tests/* The tests associated with the new checks
2 validators/overlays/all/classes/default-validator-stages.xml The checks we want to run in the testbed

8. Merge the new tests into main

Merge the topic branch back into the main branch and delete the topic branch. (likely using the GitHub interface if you created a PR for this branch)

Clone this wiki locally