Testing of calculator №6.1. Testing using etalons

Original in Russian: http://programmingmindstream.blogspot.com/2014/05/61.html

Table of contents

In the previous chapter, we have moved from GUI-testing to testing of business logic by defining business-logic class TCalculator.

In this article, we’ll discuss and implement “Testing using etalons”… Testing using etalons is based on tests from the previous chapter. However, it performs an important function I’ll tell about later. Cutting it short, using etalons means saving the values and results of test into a file that is afterwards compared with the etalon. If files do not coincide, test fails. A question arises, where do we get etalon file? We have two options: either we create it with our own hands or (as I did), if etalon does not exist, we create it automatically basing on the test result file, since we assume that our tests are correct a fortiori.

I’ll try to explain in details using the example of writing the etalon test for the operation + or, as we’ve named it in our calculator, function ADD.

To begin with, we write a new class TCalculatorOperationViaEtalonTest, which looks much like the previous class TCalculatorOperationTest. But, this time we do not check the coincidence with the etalon file instead of the logic.

type
  TCalculatorOperationViaEtalonTest = class(TTestCase)
   published
    procedure TestDiv;
    procedure TestMul;
    procedure TestAdd;
    procedure TestSub;
  end;//TCalculatorOperationViaEtalonTest
 
procedure TCalculatorOperationViaEtalonTest.TestAdd;
var
  x1, x2  : string;
begin
  x1:= cA;
  x2:= cB;
 
  CheckTrue(AddArgumentsToLog(g_Logger, x1, x2, TCalculator.Add(x2, x1), Self));
end;

In order to save the results of testing into the file and compare it to the etalon, we introduce one more class TLogger that will be used through global variable g_Logger.

unit Tests.Logger;
 
interface
 
uses
  TestFrameWork;
 
const
  cEtalonSuffix = '.etalon';
  cTestSuffix = '.out';
  cTestFolder = 'TestSet';
 
type
  TLogger = class
  strict private
    FTestFile : TextFile;
    FTestFilePath,
    FEtalonFilePath : string;
  private
    function TestOutputFolderPath: string;
    function Is2FilesEqual(const aFilePathTest, aFilePathEtalon: string): Boolean;
    function IsExistEtalonFile: Boolean;
  public
    class constructor Create;
    class destructor Destroy;
    procedure OpenTest(aTestCase: TTestCase);
    procedure ToLog(const aParametr: string);
    function CheckWithEtalon: Boolean;
  end;//TLogger
 
var
  g_Logger : TLogger;
 
implementation
 
uses
  SysUtils,
  System.Classes,
  Winapi.Windows;
 
{ TLogger }
 
function TLogger.CheckWithEtalon: Boolean;
begin
  Assert(FTestFilePath<>'');
  Assert(FEtalonFilePath<>'');
 
  CloseFile(FTestFile);
 
  if IsExistEtalonFile then
    Result := Is2FilesEqual(FTestFilePath, FEtalonFilePath)
  else
    Result := CopyFile(PWideChar(FTestFilePath),PWideChar(FEtalonFilePath),True);
end;
 
class destructor TLogger.Destroy;
begin
  FreeAndNil(g_Logger);
end;
 
class constructor TLogger.Create;
begin
  g_Logger := TLogger.Create;
end;
 
function TLogger.Is2FilesEqual(const aFilePathTest,
                                     aFilePathEtalon: string): Boolean;
var
  l_msFileTest, l_msFileEtalon: TMemoryStream;
begin
  Result := False;
  l_msFileTest := TMemoryStream.Create;
  try
    l_msFileTest.LoadFromFile(aFilePathTest);
    l_msFileEtalon := TMemoryStream.Create;
    try
      l_msFileEtalon.LoadFromFile(aFilePathEtalon);
      if l_msFileTest.Size = l_msFileEtalon.Size then
        Result := CompareMem(l_msFileTest.Memory, l_msFileEtalon.memory, l_msFileTest.Size);
    finally
      FreeAndNil(l_msFileEtalon);
    end;
  finally
    FreeAndNil(l_msFileTest);
  end
end;
 
function TLogger.IsExistEtalonFile: Boolean;
begin
  Result:= FileExists(FEtalonFilePath);
end;
 
procedure TLogger.OpenTest(aTestCase: TTestCase);
var
  l_FileName : string;
begin
  l_FileName := aTestCase.ClassName + aTestCase.GetName;
  FTestFilePath := TestOutputFolderPath + l_FileName + cTestSuffix;
  FEtalonFilePath := TestOutputFolderPath + l_FileName + cEtalonSuffix;
 
  if not DirectoryExists(TestOutputFolderPath) then
    ForceDirectories(TestOutputFolderPath);
 
  AssignFile(FTestFile, FTestFilePath);
  Rewrite(FTestFile);
end;
 
function TLogger.TestOutputFolderPath: string;
begin
  Result := ExtractFilePath(ParamStr(0)) + cTestFolder + '\'
end;
 
procedure TLogger.ToLog(const aParametr: string);
begin
  Writeln(FTestFile, aParametr + ' ');
end;
 
end.

Thus, after executing the test for the first time we get two resulting files:
- TCalculatorOperationViaEtalonTestTestAdd.out, file to form “our test”
- TCalculatorOperationViaEtalonTestTestAdd.etalon, file that became (once more, we assume our tests are correct) the etalon. We merged the etalon to GIT.

The format of .out and .etalon files for “Plus” operation:

5 – the first argument
10 – the second argument
15 – the result

The whole code of a new class TCalculatorOperationViaEtalonTest:

unit CalculatorOperationViaEtalonTest;
 
interface
 
uses
  TestFrameWork,
  Calculator
  ;
 
 type
  TCalculatorOperationViaEtalonTest = class(TTestCase)
   published
    procedure TestDiv;
    procedure TestMul;
    procedure TestAdd;
    procedure TestSub;
  end;//TCalculatorOperationViaEtalonTest
 
implementation
 
  uses
    SysUtils,
    Tests.Logger;
 
const
 cA = '5';
 cB = '10';
{ TCalculatorOperationViaEtalonTest }
function AddArgumentsToLog(aLogger: TLogger;
                           aX1, aX2, aResult: string;
                           aTestCase: TTestCase): Boolean;
begin
  aLogger.OpenTest(aTestCase);
  aLogger.ToLog(aX1);
  aLogger.ToLog(aX2);
  aLogger.ToLog(aResult);
  Result := aLogger.CheckWithEtalon;
end;
 
procedure TCalculatorOperationViaEtalonTest.TestDiv;
var
  x1, x2 : string;
begin
  x1:= cA;
  x2:= cB;
 
  CheckTrue(AddArgumentsToLog(g_Logger, x1, x2, TCalculator.Divide(x2, x1), Self));
end;
 
procedure TCalculatorOperationViaEtalonTest.TestSub;
var
  x1, x2  : string;
begin
  x1:= cA;
  x2:= cB;
 
  CheckTrue(AddArgumentsToLog(g_Logger, x1, x2, TCalculator.Sub(x2, x1), Self));
end;
 
procedure TCalculatorOperationViaEtalonTest.TestMul;
var
  x1, x2  : string;
begin
  x1:= cA;
  x2:= cB;
 
  CheckTrue(AddArgumentsToLog(g_Logger, x1, x2, TCalculator.Mul(x2, x1), Self));
end;
 
procedure TCalculatorOperationViaEtalonTest.TestAdd;
var
  x1, x2  : string;
begin
  x1:= cA;
  x2:= cB;
 
  CheckTrue(AddArgumentsToLog(g_Logger, x1, x2, TCalculator.Add(x2, x1), Self));
end;
 
initialization
 TestFramework.RegisterTest(TCalculatorOperationViaEtalonTest.Suite);
end.

Finally, the most interesting – why we actually need etalon tests.
1. Since etalon test is committed in GIT we can always check if something gets wrong while introducing some changes.
2. The “most” important. In large teams work is divided between developers and testers. If logic of an application changes, for example, Plus function sums 2 figures and adds 8 instead of just summing 2 figures as before the changes have took place, then we face this fact.

class function TCalculator.Add(const A, B: string): string;
var
  x1, x2, x3 : single;
begin
  x1 := StrToFloat(A);
  x2 := StrToFloat(B);
  x3 := x1 + x2 + 8;
  Result := FloatToStr(x3);
end;

Now, when we run our testing it will look as follows:

It means all our tests of “Plus” failed. As a result, developer has to redo 3 tests. However, let’s just try to correct the etalon.

We had:
5 10 15

We have:
5
10
23

The result:

As you can see, tests with etalons have passed successfully. It means, when changing logic of an application testers also can introduce changes to tests WITHOUT HAVING to change the source code. This enables a very flexible use.

P.S.
While writing the article I tried to find the discovering of the theme in RuNet. It is not developed in RuNet, but a similar approach to testing is described by Kernighan and Pike in the chapter about testing. Though, they offer to measure test executing time and efficiency while running. We’ll come down to it in our coming articles.

Repository