Combined and TRX reporters

[TamaMcGlinn]

Added a combined reporter and a TRX format reporter; the latter is what our Microsoft vNext TFS buildserver uses to show the results in graphs. Example usage:

with Ada.Text_IO; use Ada.Text_IO;
with Ada.Command_Line;

with AUnit.Run;
with AUnit.Options;
with AUnit_Test.Tests; use AUnit_Test.Tests;
with AUnit;

with AUnit.Reporter.Text;
with AUnit.Reporter.TRX;
with AUnit.Reporter.Combine;

procedure AUnit_Test.Test_Runner is

   function Runner is new AUnit.Run.Test_Runner_With_Status (Suite);

   Reporter : AUnit.Reporter.Combine.Combined_Reporter;
   XML_Gen : AUnit.Reporter.TRX.Reporter_Generator;

   Options : constant AUnit.Options.AUnit_Options :=
      (Global_Timer     => True,
       Test_Case_Timer  => True,
       Report_Successes => False,
       Filter           => null);

   use type AUnit.Status;

begin

   declare
      Text_Reporter : aliased AUnit.Reporter.Text.Text_Reporter;
      XML_Reporter : aliased AUnit.Reporter.Reporter'Class := XML_Gen.Get_Reporter;
   begin
      Reporter.Add_Reporter (XML_Reporter'Access);
      Reporter.Add_Reporter (Text_Reporter'Access);
      if Runner (Reporter, Options) = AUnit.Success then
         Put_Line ("--------------Unit tests succeeded--------------");
         Ada.Command_Line.Set_Exit_Status(Ada.Command_Line.Success);
      else
         Put_Line ("--------------Unit tests failed-----------------");
         Ada.Command_Line.Set_Exit_Status(Ada.Command_Line.Failure);
      end if;
   end;

end AUnit_Test.Test_Runner;

Because of the combined reporter, we have both the graphical output and also the same results output to the console when our build is run locally.

[TamaMcGlinn]

Some screenshots of the TRX file being displayed by vNext (of 1 failing unit test; a succeeding one is less interesting)

Note the time the test took is correctly passed on, but vNext doesn't display values below 1 ms (in this case it took around 300 μs).

I don't expect AdaCore to be able to test the vNext aspect of this; however, I am sure it will be useful to other users of aunit.

[fedor-rybin]

This won't work for restricted runtimes, Ada.Containers are not present there

[TamaMcGlinn]

Is there an easy way for me to test that? I tried adding for Runtime ("Ada") use "zfp"; to the project using aunit, but then even AUnit.Time_Measure's usage of Ada.Calendar won't work.

[TamaMcGlinn]

After reviewing aunit_shared.gpr it seems clear this currently works fine.

I guess the proper way to do this would be to add another enumeration like the current Except, Calend, Memory and FileIO and then create a consistent interface to the Vectors package with two implementations, "vectors" would pass through to Ada.Containers.Vectors, while "novectors" would have some custom implementation.

Another option would be to use AUnit.Memory directly, or rather, to implement AUnit.Vectors in terms of AUnit.Memory, so that the above enumeration isn't necessay. Thoughts on which is better/simpler?

By the way, I saw that in aunit_shared.gpr around line 52 when "cert" => the value for Calend remains "calendar", which is implicitly saying that Ada.Calendar is certified. It seems better to make that explicit if that is correct. If this were normal Ada code, I would say that these four enumerations are begging to be made into a record type, so that we could make the compiler spot such omissions, but I'm not sure that applies to gpr files.

[TamaMcGlinn]

A third option I think is to move the trx reporter into a separate directory which is only included as a source when the runtime is set to full. This is the only option that doesn't have me unnecessarily supporting runtimes I don't use myself.

[fedor-rybin]

R should be of type Reporter_Access, otherwise there would be a compilation error:

aunit-reporter-combine.adb:20:26: implicit conversion of anonymous access parameter not allowed

[fedor-rybin]

Overall I'm not sure that Combined_Reporter adds much value. Now that Result is nod modified by Reporter, it is easy to use AUnit.Run.Test_Runner_With_Results and then call any ammount of Reporters in sequence:

with AUnit.Run;
with Math_Suite;
with AUnit.Reporter.XML;
with AUnit.Reporter.Text;
with AUnit.Test_Results;

procedure Test_Math is
   procedure Runner is new AUnit.Run.Test_Runner_With_Results
     (Math_Suite.Suite);

   Txt_Reporter : AUnit.Reporter.Text.Text_Reporter;
   XML_Reporter : AUnit.Reporter.XML.XML_Reporter;

   Results : AUnit.Test_Results.Result;
begin
   Runner (Txt_Reporter, Results);                     -- First reporter
   AUnit.Reporter.XML.Report (XML_Reporter, Results);  -- Second reporter
end Test_Math;

[TamaMcGlinn]

@fedor-rybin thank you for your comments; I have removed the combined reporter entirely, so I made a new pull request with just the trx reporter.