Support development of new checks #101
Description
New checks on metadata have historically been developed in a rather ad hoc fashion in a variety of places. One of the goals of the testbed is to provide a single place to do this in a more consistent way. This has been hard to do in the initial testbed setup because the emphasis has been on handling the 0.9/0.10/Xalan/no-Xalan combinations, but once we have progressed through at least part of #100 that will be less of a concern and it's worth coming back to this.
As a basic approach, I think we should assume that new checks are initially developed on a branch in this repository. That means that they can be developed alongside their corresponding tests and, once the check has been implemented in the production deployment, the tests developed here (as well as any required changes to other tests) can be cherry-picked onto the main branch.
(In principle, I'd prefer that to be a merge operation but you'd need to manually reconcile the two versions of the check and that seems like more complication for little benefit. It may be worth revisiting the details once we have worked through a real example or two.)
I'd like this to be compatible with a longer term move towards using policy files (essentially, lists of checks) taken directly from the production deployment, replacing the current setup where we construct such lists locally in a duplicative way which may be error-prone. My thought is therefore that the pipeline configuration is broken up into a couple of components which can be locally overridden in a development branch, along with some conventions as to where to place the development content.
The current scheme has:
default-validator.xmlonly exists in thealloverlay, and definespipelineto include onlydefault_validator_stages.default-validator.xmlincludesdefault-validator-stages.xml(again, currently only defined in thealloverlay) which definesdefault_validator_stagesas a list of checks. This is the part most likely to be replaced by a reference to a policy file from the production deployment, once we have one which fits the bill.
The simplest way of extending this current system to allow for new check development would, I think, be to change the pipeline definition from calling the single default_validator_stages to calling that and then a second CompositeStage which would contain a list of checks under development. Purely as a straw man, let's call that development_stages.
To provide a clean separation of the under-development checks within the testbed, I think a good approach would be to not define that second list in the all overlay but in a new overlay. A straw man for the overlay name would be development and it would be applied in every validator, probably quite far down the build process if not actually at the end. In the main branch, the overlay would include a single XML configuration defining the CompositeStage bean referenced by pipeline. It would be empty.
In a development branch, you'd put all of the definitions for the new check or checks into an override version of the file defining development_stages within the development overlay. If they are XSL-based checks, the XSL files can live in the appropriate place in the same overlay. The tests associated with the checks live outside the overlay in the usual place (/tests/xml/...). I hope this would make it easier to pull just those into the main branch when the production code picks up the code for the actual check.
(Long term, the production code will also pick up a reference to the new check at this point; shorter term that needs to be added into default_validator_stages manually when the tests are pulled in.)
The development branch is discarded at this point. As I mentioned earlier, in principle I'd prefer a merge here rather than a cherry-pick/delete operation but it's not yet clear to me how to get there from where we currently are. Once we have appropriate policy files we can look at that again.