Knowledge Base

WD98: Table Sum Rounds Incorrectly on Large Numbers

Article ID: 177893

Article Last Modified on 10/31/2006


APPLIES TO


This article was previously published under Q177893

SYMPTOMS

When you sum a number in a table that is greater than 15 decimal places, the number is rounded to the nearest 10.

For example, the following 16 digit number rounds to the nearest 10th.

6553565535655351 + 1 = 6553565535655350

CAUSE

This is a limitation of the floating-point processor. The exact behavior of a floating-point calculation is hardware dependent.

Additional query words: OFF98 floating point

Keywords: kbprb kbtable KB177893