Depression, or Falsity of hResult and other ErrorCode’s

Original in Russian: http://programmingmindstream.blogspot.ru/2014/09/blog-post_23.html

So, “third time unlucky”…

More precisely, for the third time the launch of internal product failed.

My fault.

Although it had “all the trimmings”, tests, factories and so no and so forth.

Tests of all sorts – pass.

The real soft – does not work in the real conditions.

READ_ERROR and then WRITE_ERROR. Under distributed access.

Generally, I will not write to blog until I puzzle it out, reconsider it and give “a treatment plan”.

For there is no point in it.

For “theory without practice is useless”.

There’s no point in writing about “cows in vacuum” when “your own cows do not work”.

For now – we’ve created a “load test”.

I left it working for a night.

I would look at it the next morning.

It is possible we’ll write “another load test”.

For quite the time I haven’t seen such epic fails.

I may tell you one thing “not about myself” – hResult and other “ErrorCode’s” are in a way “not a great gimmick”. Of course, it depends on how you use it… I have not always used it properly…

I had:

 SetFilePos(hFile, aPos);
 FileWrite(hFile, @SomeValue, SizeOf(SomeValue));
 SetFilePos(hFile, aPos);
 FileRead(hFile, @SomeOtherValue, SizeOf(SomeOtherValue));
 Assert(SomeValue = SomeOtherValue);

I have:

 SetFilePos(hFile, aPos);
 FileWrite(hFile, @SomeValue, SizeOf(SomeValue));
 SetFilePos(hFile, aPos);
 SysCheck(FileRead(hFile, @SomeOtherValue, SizeOf(SomeOtherValue)));
 // - here it began to fall “sometimes”, despite the fact that SomeOtherValue = SomeValue
 Assert(SomeValue = SomeOtherValue);

- it is a surprise that without SysCheck – the check passed.

It means, the error has been returned but the CHECK passed.

It works in this way:

 SetFilePos(hFile, aPos);
 FileWrite(hFile, @SomeValue, SizeOf(SomeValue));
 SetFilePos(hFile, aPos);
 try
  SysCheck(FileRead(hFile, @SomeOtherValue, SizeOf(SomeOtherValue)));
  // - here it began to fall “sometimes”, despite the fact that SomeOtherValue = SomeValue
 finally
  Assert(SomeValue = SomeOtherValue);
  // - here it does NOT FALL, although it “falls” more, on SysCheck
 end;

It means, “sometimes what’s intended to be read is read”, but with an error.

WITHOUT checking for errors everything worked but “malfunctioned”, with the “check for errors” it began to “fall more often”.

It was so in terms of distributed heterogeneous environment, using “dead” or “half-dead” computers, different Windows versions up to the “prehistoric” ones.

Why? Yet not clear.

Anyway, I blame my own “butter fingers” instead of “Microsoft guys”.

For it is easy to “point fingers at others”.

Simple, but not constructive.

And, by the way, if we write:

SetFilePos(hFile, aPos);
FileWrite(hFile, @SomeValue, SizeOf(SomeValue));
l_TryCount := 0;
while (l_TryCount < 100) do
begin
 Inc(l_TryCount);
 SetFilePos(hFile, aPos);
 try
  SysCheck(FileRead(hFile, @SomeOtherValue, SizeOf(SomeOtherValue)));
  // - here it began to fall “sometimes”, despite the fact that SomeOtherValue = SomeValue
 except
  if (l_TryCount < 100) then
   continue
  else
   raise;
 end;//try..except
 break;
end;
 Assert(SomeValue = SomeOtherValue);

Again, it “falls less often”.

The reason “for thinking”.

Unfortunately, it does not repeat on “synthetic tests”.

In order to top it off, LockRegion/UnlockRegion also engage in it.

Naturally, they are “correctly established” and “properly” framed SysCheck etc.

BUT it looks like their presence is the issue.

Without them, it “seems to work” but with “concurrent access” it works bad, which is obvious.

We’ll move to a new testing level.

P.S. The whole code given above is, of course, “pseudocode” used to illustrate problems. More likely, “comas” are put there incorrectly. SysCheck at FileWrite is also skipped on purpose. Believe me, we have it.

In addition, WrittenSize and ReadSize are checked there.

However, these details are also skipped – on purpose.

P.P.S. The code without SysCheck and OleCheck works for already 15 years, but malfunctions (now and then, I mean what I say, we get incorrect data; it’s unpleasant, but “possible to live”). That is actually the reason why I went in for puzzling it out and writing SysCheck and OleCheck.

As a result, I have got “the whole nine yards”.

Nine yards of what? I don’t know yet.

Once again, it does not repeat at all client stations. May be due to the “butter fingers”.

In short – “do not forget about errors codes”, though “processing” of them is “not always clear”.

When I establish the guilt of my “butter fingers” I will surely write.

P.P.P.S. I’ve also added logging of problem operations. And… And… I’ve got a “Schrödinger's cat”. Logging began to influence “business logic”, at least in terms of “timing delays”, which is obvious.

Again – a “separate issue”.

P.P.P.P.S. One more thing. The code given above is for one client. One writes, the other one reads is not the case. I understand using it for different clients, but not for one client. I hope, I will soon understand.

P.P.P.P.S. Another more thing.

I have already checked the version of this kind:

function DoRead(...): LongBool;
begin
 FileRead(hFile ...);
 // - we “forgot” to return the result
end;
...
SysCheck(DoRead(...));
// - we check for “garbage”

“At a rough guess” there are no “uninitialized variables”.

P.P.P.P.P.S. By the way, my tests are still working… No errors :-( It “inspires sadness”. I’ll see how it ends in the morning.

P.P.P.P.P.S. As it turned out, two things ensure problems:

1. Access through the UNC paths, i.e. paths like - \\server\resource\path\filename.
2. Using of LockFile as a must.

P.P.P.P.P.P.S. Yesterday tests worked without errors. Data in the 3 Gbs area has been processed.

Today I launched tests on two computers. Tomorrow I’ll see the results.

Then I will launch on 3, 4, 5 and so on.

P.P.P.P.P.P.P.S. Today I also found two “bottlenecks” - AllocNewFATAlement and AllocNewCluster. In their turn, they lead to LockFile. We lock the header that stores the information about storage structure. All users “beat against these blockings” while writing.

I already know how to solve it.

We should pro-actively allocate a number (five, ten, twenty) of FATElement and Cluster at once instead of one. As one piece. Given that a file usually contains more than one or two or even ten clusters, it is effective. We should also keep a list of free ones locally by a client. Then, when a client closes, they should be put back to the list of free, “not used” ones. This is done so that other client could use it afterwards.

Sure, there is a possibility that we lose items if a client provokes the fail.

But we get to the “bottleneck” less often, because we can write:

if AllocatedFatElements.Empty then
begin
 // - there is inter-PROCESS “bottleneck”
 Lock;
 try
  Result := AllocNewFatElement;
  for l_Index := 0 to 10 do
   AllocatedFatElements.Add(AllocNewFatElement);
 finally
  Unlock;
 end//try..finally
end
else
 // - here is ONLY inter-THREADED “bottleneck” (because AllocatedFatElements is – naturally – protected by a number of threads)
 Result := AllocatedFatElements.GetLastAndDeleteIt;

Instead of:

Lock;
// - there is ALWAYS a multi-PROCESS “bottleneck”
try
 Result := AllocNewFatElement;
finally
 Unlock;
end;//try..finally

It is the same for clusters.

Even if the elements will  “hang”, t will not get worse and the storage will not be broken. It will have “holes”.

Taking into account the fact that at night the “night Update” takes place (if it is possible), the storage is repacked - in any event. It means the “holes” will disappear - in any case – by morning.

As a result, we’ll have a repacked persistent part with no holes and the “empty” variable part.

During the day clients will write their documents’ versions into the variable part.

The process repeats the next night.

This is, of course, in case there are no working users connected to the base.

A special post is - here. http://programmingmindstream.blogspot.ru/2014/09/blog-post_25.html

Note that this is “all about internal products”. I won’t tell about third-party products.

For those who’ve read to the end, the task looks as follows:

“Deficit” :-(