x86: Fix issue with PACKUSWB when the value to convert is exactly 0x00ff #6514
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PACKUSWB instruction performs a lane-wise conversion of signed 16-bit integers to 8-bit integers, saturating the results if they are out-of-range (i.e., to the range 0 to 255).
In the SLEIGH implementation of the saturation operation, if the signed 16-bit integer is exactly equal to
0x00ff
it will be converted to the byte0x00
(instead of0xff
).The issue with the value
0x00ff
in particular is that that both(sword s> 0xff:2)
and(sword s< 0xff:2)
are false causing neither of the conditions. In the patch I choose to change the second condition tos<=
(matching the comment above the macro), however changing the first condition would also fix the issue.e.g.:
660f67db
"PACKUSWB XMM3, XMM3" with XMM3=0x00ff_00ffx86:LE:64:default
(Existing): { XMM3 = 0x0 }x86:LE:64:default
(This patch): { XMM3 = 0xff_ff_00_00_00_00_00_00_ff_ff }(Note: The repetition of the bytes is correct because XMM3 is both the source and destination register).