Page MenuHomePhabricator

[Fixed Point Arithmetic] Validation Test for Saturated Division and Comparison Fix
Needs ReviewPublic

Authored by leonardchan on May 17 2018, 7:59 AM.



This patch includes changes for division on saturated fixed point types. Overflow occurs when the resulting value cannot fit into the number of data bits for the resulting type.

For signed division, we cap at the max value of the type when the dividend sign is positive, divisor sign is negative, but the quotient is still positive. Reciprocally, we cap at the min value if the dividend is negative, divisor is positive, and quotient is negative.

For unsigned division, overflow occurs if the resulting value exceeds the max possible value that this type can hold.

The logic for comparisons between fixed point types was also fixed to account for padding bits. Since the padding bits contains values we do not care about, we mask the fixed point data bits in the underlying integral that we do care about and compare them.

Diff Detail

rC Clang