Why language models get COMP-3 arithmetic wrong

Packed decimal in one paragraph

COMP-3 (packed decimal) stores two decimal digits per byte, one per nibble, with the final nibble holding the sign: C for positive, D for negative, F for unsigned. A field declared PIC S9(7)V99 COMP-3 has nine digits and occupies five bytes. The value +12,345.67 is stored as the digits 001234567 followed by the sign, so the bytes are 00 12 34 56 7C in hex. The decimal point is not stored at all. The V in the PICTURE tells the compiler where to align it.

Everything that follows comes from that last sentence. The scale of every numeric field is a compile-time fact from the data division, and the result of an arithmetic statement depends on the scales of every operand and of the receiving field.

Truncation is the default

Consider a fee calculation. To a reader who sees only the procedure division, it multiplies a balance by a rate.

The receiving field WS-FEE has two decimal places and no ROUNDED phrase, so the result is truncated, not rounded: 0.4999 becomes 0.49. High-order digits are truncated too. If the product exceeds 99,999.99 and there is no ON SIZE ERROR phrase, the leading digits are silently dropped. Both behaviors are standard COBOL, and both are the opposite of what most programming languages do by default.

1       01  WS-BALANCE        PIC S9(9)V99   COMP-3.
2       01  WS-RATE           PIC SV9(5)     COMP-3.
3       01  WS-FEE            PIC S9(5)V99   COMP-3.
4       01  WS-FEE-R          PIC S9(5)V99   COMP-3.
5 
6           COMPUTE WS-FEE = WS-BALANCE * WS-RATE.
7           COMPUTE WS-FEE-R ROUNDED = WS-BALANCE * WS-RATE
8               ON SIZE ERROR PERFORM 9100-FEE-OVERFLOW
9           END-COMPUTE.

What a language model does with this

We tested this pattern repeatedly while building our own optional description layer. Given the procedure division and asked to translate it, a general language model typically produces something like fee = balance * rate in Java with double, or with BigDecimal and RoundingMode.HALF_UP on both lines. The first version introduces binary floating point error. The second rounds where the COBOL truncates. Neither handles the size error branch on the first statement, because there is none, and the silent high-order truncation disappears.

This is not a matter of model quality. The information required is not in the text the model was given. The receiving field's PICTURE may live in a copybook four COPY statements away. The behavior of intermediate results depends on the compiler's ARITH option (COMPAT allows 18 digit intermediates, EXTEND allows 31), and sign handling of packed fields depends on NUMPROC. A model reading text is predicting the most likely translation, and the most likely translation in the training data is the one that rounds.

What a deterministic engine does instead

PYD ENTERPRISES resolves every operand to its declaration, including through copybooks and REDEFINES, and annotates each arithmetic node in the AST with operand scales, receiving field precision, rounding mode and size error handling. Compiler options are part of the scan configuration, read from the build JCL or the Endevor processor where available, so the intermediate precision rules match production.

The exported formula carries that information explicitly. A target team implementing it in Java gets a contract, not a guess.

The same annotations drive the equivalence fixtures. For each arithmetic rule the generator picks inputs that land exactly on a rounding boundary, inputs that overflow the receiving field, and negative values that exercise the sign nibble, because those are the three places a hand-written port most often drifts from the original.

1// Rule FEE-001: FEECALC.cbl lines 6 to 6, truncate to 2 dp, high-order truncation at 5 integer digits
2BigDecimal fee = balance.multiply(rate)
3    .setScale(2, RoundingMode.DOWN)
4    .remainder(new BigDecimal("100000"));
5 
6// Rule FEE-002: FEECALC.cbl lines 7 to 9, ROUNDED (half away from zero), size error branch
7BigDecimal raw = balance.multiply(rate).setScale(2, RoundingMode.HALF_UP);
8if (raw.abs().compareTo(new BigDecimal("99999.99")) > 0) {
9    feeOverflow();          // 9100-FEE-OVERFLOW; WS-FEE-R keeps its previous value
10} else {
11    feeRounded = raw;
12}

The rounding mode question

Plain ROUNDED in IBM Enterprise COBOL rounds half away from zero, which corresponds to Java's RoundingMode.HALF_UP (Java's HALF_UP is also away from zero for negative values). COBOL 2002 added explicit ROUNDED MODE phrases such as NEAREST-EVEN, and Enterprise COBOL supports them. If the source uses one, the extractor carries it through. If a target team reaches for banker's rounding because it is the default in their decimal library, the equivalence suite will fail on the first half-cent boundary fixture, which is the point of generating those fixtures.

Where language models do help

None of this means language models are useless in modernization. They are good at naming things. Given a rule the deterministic engine already isolated, with resolved types and source lines, a model the customer hosts can suggest that 2100-DRIVER-AGE-FACTOR should be called Driver age rating factor and write a one-paragraph description for the business analyst. That is the only role we give them: describe what the engine found, never decide what the code does. The comparison with generic LLM migration goes further into the trade-offs, and the COMP-3 pattern entry lists the target mappings for common arithmetic forms.