PRECISION

Set Numeric PRECISION

This directive determines the number of significant digits to be maintained to the right of the decimal point.

Syntax

PRECISION numeric-value

numeric-value is any integer in the range of 0 to 14 or 127.

Remarks

Numeric-value of 127 is the same as FLOATING POINT and is available starting with release level 8.0.

PRECISION does not affect the maximum number of digits allowed in a fixed-point number, which remains 14. Values with more than 14 digits must be represented in floating point format.

Rounding to the PRECISION amount occurs whenever a variable gets set as the result of a mathematical operation or a numeric variable is PRINTed.

PRECISION does not affect the storage of values resulting from a LET directive (unless the LET contains a mathematical calculation) or INPUT and READ assignments. These values are not rounded when stored in memory; they are stored as defined. If the numeric values are PRINTed, the output is round but the stored values remain unaffected.

If PRECISION is not specified, the default is set at 2.

PRECISION is reset to the default of 2 by BEGIN, CLEAR, END, LOAD, RESET, RUN, or STOP directives.

PRECISION is automatically set at 127 when the FLOATING POINT directive is active.

Examples

PRECISION 4

Given the assignments A=2.112233; B=2.112233*2; C=A*2; D=9.9999445; and E=9.99995, PRECISION 4 has the following effects:

A is stored as 2.112233	A is output as 2.1122
B is stored as 4.2245	B is output as 4.2245
C is stored as 4.2245	C is output as 4.2245
D is stored as 9.9999445 D is output as 9.9999
E is stored as 9.99995	E is output as 10

The following program demonstrates an interesting anomaly:

10 PRECISION 4
20 A=1.0050, B=1.01
30 PRECISION 2
40 PRINT "A=",A,"    B=",B
50 IF A=B THEN
      PRINT "THEY ARE EQUAL"
   ELSE
      PRINT "THEY ARE NOT EQUAL"
60 LOOP = LOOP + 1
70 PRECISION 4
80 A=1.0051
90 IF LOOP < 2 THEN GOTO 30

When A equals 1.0051 and A is rounded, then A will equal B, but when A equals 1.0050 and is rounded A will not equal B. Thoroughbred Basic subtracts the numbers and tests the result, so a result of 0 is needed to determine equality. With rounding, the 0.0049 difference rounds to 0 and the 0.0050 difference rounds to 0.01. When PRM NOROUND is active, the numbers must be the same.

See also

BEGIN, CLEAR, END, FLOATING POINT, LOAD, RUN, and STOP directives
PRC system variable

From the Thoroughbred Basic Language Reference, Volume II: F–Q, version 8.9.1. Open this page in the PDF