Part 2 to fix x86_64 fp128 calling convention.

[chh]

Summary

Part 1 was submitted in http://reviews.llvm.org/D15134.
Bugs:

• https://llvm.org/bugs/show_bug.cgi?id=24109
• https://llvm.org/bugs/show_bug.cgi?id=23897

Changes:

• X86RegisterInfo.td, X86RecognizableInstr.cpp: Add FR128 register class.
• X86CallingConv.td: Pass f128 values in XMM registers or on stack.
• X86InstrCompiler.td, X86InstrInfo.td, X86InstrSSE.td: Add instruction selection patterns for f128.
• X86ISelLowering.cpp: When target has MMX registers, configure MVT::f128 in FR128RegClass, with TypeSoftenFloat action, and custom actions for some opcodes. Add missed cases of MVT::f128 in places that handle f32, f64, or vector types. Add TODO comment to support f128 type in inline assembly code.
• Add unit tests for x86_64 fp128 type.

[chh]

Same changes as the previous diff, but use -U999999.

[davidxl]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8589

Can you explain this a little more?

8593

Should it be ... && (N1VT == N1OpVT || N1OpVT != MVT::f128)?

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

169

Why hasType(VT) returns false for FR128? Is it expected?

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3535

unconditionally expand? Do you need to add some assertion on the type expected?

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

52

why this change?

68

Why returning false for some cases and true for others?

78

Explain this change?

145

Should the problem be fixed in SoftenFloatOperand method?

[chh]

Add more comments and changes per davlidxl's suggestions.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8589

Done. Please see new comment in code.

8593

Good suggestion. I might be too conservative here.
Now I changed the condition to only N1Op0VT != MVT::f128,
and all my tests are still working.

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

169

See also my previous answers and comments at line 143.
The question is whether ComRC returned by TRI->getCommonSubClass should guarantee ComRC->hasType(Node->getSimpleValueType(ResNo), or FR128 should be defined to work with existing getCommonSubClass. I couldn't find a clean way to change either FR128 or getCommonSubClass, without breaking many other things. So, here it seems to me the smallest fix on the user of ComRC (UseRC).

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

3535

Is there some invalid constant type?
I think ExpandConstant(CP) should already check necessary node types.

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

52

The new default value R will make other code simpler like SetSoftenedFloat(R, R),
which just wants to mark this node as softened without changing it.

68

See new inline comment at line 62.

78

Added more comments in code, line 62.
Hope that will make more sense to the comments about FABS, FCOPYSIGN, FNEG, ConstantFP.

145

SoftenFloatOperand works with ScanOperands if !KeepFloat.
So, the new condition here is when KeepFloat, we should call ReplaceValueWith.
It might be possible to fix every SoftenFloatOperand method and also simplify ScanOperands, but that looks like a more complex and risky move.

[davidxl]

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

169

What is VT in this case? What reg class is ComRC here? How about the DstRC below TLI->getRegClassFor(VT)?

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

291

Change the name to ExpandConstantFP128?

3535

I don't see what expected EVTs are checked in this method?

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

64

Does the following change affects other types other than f128?

[echristo]

Adding Owen and Quentin to this review. I'd feel more comfortable if they were to comment on the long term maintainability of this.

Thanks!

-eric

[chh]

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

169

One of the test cases showing this problem is
test/CodeGen/X86/avx512-calling-conv.ll,
compiled with -mtriple=x86_64-apple-darwin -mcpu=knl

VT is v16i8 (23)
ComRC is the new FR128RegClassID (74)
DstRC is not set yet, but the correct
TLI->getRegClassFor(VT) is VR128RegClassID (75)

It means that my current definition of FR128 has registers in VR128,
and ComRC could be FR128 where VR128 was the only candidate before.
But of course FR128 with my current instruction selection patterns won't
work for v16i8, and it should not.

• I tried in my early development of this patch to store f128 value in existing register classes, without a new FR128. But that approach has even more troubles.
• I also tried to copy some VR128 instruction patterns to FR128 to make this work without this patch in EmitCopyFromReg. But that requires a lot of duplication in instruction selection pattern, and I had not even resolved all the problems.
• One thing I think might be possible, as my original TODO item, is to change the semantic of getCommonSubClass to guarantee ComRC->hasType(VT), or an extra flag for getCommonSubclass to guarantee ComRC->hasType(VT). I was hoping to break this patch into smaller pieces, and those refactoring tasks seem to be good candidates for follow up patches.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

291

Although it is triggered now only for f128 type, but it should work for other types say f80, f96, etc. Actually, the constant value is not floating type, but an integer (binary bits) of the floating point value from CP->getConstantIntValue.

3535

I think there is no target-independent type restriction here.
I see TLI.isFPImmLegal check in ISD::ConstantFP, but don't see similar limitation for non-FP types.
ExpandConstant could have more value range limit if necessary.

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

64

Not now. Only x86_64 f128 is configured this way. But in the future new configurations can use this logic to keep value legal (in register) but with soften operators.

[davidxl]

Quentin, can you help do a quick review of the patch and give
suggestions on high level direction of the patch?

thanks,

David

[qcolombet]

Hi Chih-Hung,

I believe the current patch can be split in several smaller patches. For instance, improving the AsmPrinter and changing the ABI seem pretty orthogonal to not have to come in one patch.

Now, as a high level comment, this patch seems quite involved for something that sounds fairly common from the legalizer point of view. I.e., we have a legal type, fp128, with illegal instructions (soften float). Have you checked how we deal with those cases usually? If so, could you explain what does and does not work?
I have not spent a lot of time on thinking or reviewing the patch, so I admit I can tell nonsense, but the fell I have on the patch is that we are doing it wrong.

Regarding the ABI changes, the patch obviously breaks the existing code that have been generated by clang for those types. I would like to have a sense of how risky are that changes. Could you list the targets (OS) that use the variants you are changing?

Thanks,
-Quentin

include/llvm/Target/TargetLowering.h

1907

That change makes me shiver.
We probably do something against the design if we need to do that.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8592

Based on the comment, this seems like a hack that impact all targets to fix a x86_64 backend limitation.
The bottom line is that I am guessing we should push that change.

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

147

By definition the common sub class of A and B must be able to hold both the types of A and B, since the related registers are both in A and in B (i.e., could be access by A or B register names).
The bottom line is that comment must be always wrong and the target must do the right thing to ensure this is true, IMO.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1205

How could that be different?!
That looks wrong to have to check for both conditions.

[chh]

This patch should change output of fp128 for Android x86_64 only,
to fix mainly the problem in https://llvm.org/bugs/show_bug.cgi?id=23897.

No regression test failure was found for other targets, but since I am not sure if all other targets have enough regression tests on fp128, I tried to make changes only depend on the new configuration or work for all configurations.

It's possible to break this patch into several ones, but then only the last one should turn on the configuration change for Android x86_64. Then, all patches before the last one will be incomplete and not really tested. It's not right to change the Android x86_64 configuration before the last patch, because right now, fp128 for Android x86_64 works by itself, just not compatible with gcc. A partial change of the fp128 code will fail to compile libm. For example, the AsmPrinter change was actually discovered when libm was compiled with -g. The current code cannot handle new fp128 constant value.

The change was larger than typical bug patch because I think this is the first configuration to keep fp128 value in SSE registers. Before this, most x86 configurations used fp80 on floating point stack, and fp128 on x86 was split into two 64-bit registers. Hence, fp128 was passed in two 64-bit registers or on stack, but we need to pass them in SSE to be compatible with gcc and the ABI. Current nice legalization abstraction for floating point values worked mostly for most target, except a few cases that we have seen hacked for ppc. For fp128 in SSE registers, we need more enhancements to this legalization abstraction layer, or I would have to write up another new kind of action maybe 90% similar to soften floating point, but do something differently.

Some changes to ad-hoc optimizations related to fp128 were made only now because the existing code have not handled fp128 before. The fp128 values were split into two registers and/or converted to library function calls and do not hit the optimization code. Even the ppc f128 type is in two f64 registers. These were mostly found in the compilation of libm, which heavily cast fp128 to/from i128 to use faster bit-wise operations. I have added test cases for those problems I found. So hopefully if any of such changes were reverted, some regression test will fail.

include/llvm/Target/TargetLowering.h

1907

I think an alternative is to keep ValueTypeActions private here, but to hack into computeRegisterProperties of lib/CodeGen/TargetLoweringBase.cpp to define different actions for Android x86_64 fp128. That function now has one hack for ppcf128, but that is depending on a special MVT::ppcf128. If we add target dependent code into computeRegisterProperties, for only Android x86_64, not all f128, then that might look more like a hack than making ValueTypeActions accessible to child classes.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8592

Yes, i think this impact all x86_64 f128 values in one register.
We didn't hit this problem because no target keeps f128 in one register.
What is the alternative change?

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

147

I can think about this more. I was putting off that kind of change to getCommonSubClass as a TODO in the future, but that was more than a month ago.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

1205

This is needed only for the new special configuration of Android x86_64 fp128, which is TypeLegal to be in a register, but with a type action to soften the operations to library calls. It's TypeAction is not TypeLegal.

[qcolombet]

Hi Chih-Hung,

[chh]

A few email discussions about this patch was sent to [e-mail] and [e-mail], but they were not shown here. I don't know the cause, but please check llvm-commits mailing list about this topic. Thanks.

[chh]

Sync with latest LLVM source and
add -attr=+mmx to test cases since new default mmx is off.
Add one more parameter to getCommonSubClass and firstCommonClass,
to guarantee that returned common sub class will contain the specified simple value type.
This extra parameter is used by EmitCopyFromReg in InstrEmitter.cpp.

[martell]

In D11438#271749, @chh wrote:

I usually found missed handling of f128 type from testing libm long double on Android target, and those cases appeared in many different places.

Hi @chh,
I'm currently looking into a similar calling convention bug. I just came across this patch as you updated it today
I'll start testing your patch soon but I feel you would have some insight into this bug
https://llvm.org/bugs/show_bug.cgi?id=24398
based on the content of this patch ?

[chh]

Martell, I don't have Windows or mingw-w64 system to test now, specifically I don't know what gcc generates for that system. If you were compiling test code with gcc and clang, and link with libraries compiled by gcc, then you could hit calling convention problem. You could compare first assembly code output from both compilers.
Please take a look and try the test cases in
https://llvm.org/bugs/show_bug.cgi?id=23897
https://llvm.org/bugs/show_bug.cgi?id=24111

http://reviews.llvm.org/D11437 has the clang changes to fix the calling convention for f128, and this patch is for the llvm part. If mingw-w64 does not use 128-bit floating point, then my patches shouldn't affect your problem, but you can find the source locations to fix mingw-w64 issues.

[martell]

I've posted my reply to the llvm-developers list to not clog your review
with noise :)

[chh]

All reviewers, please take a look of the latest diff.
Please add comment for anything you think must be changed before submit.
I test it with Android and on Linux the LLVM regression tests.
After this long review period, I think a field test would be more effective to find any error.
Thanks.

[davidxl]

It seems to me the patch does not use the right approach in SoftenFloatResult and SoftenFloatOperands. The patch tries to workaround the the problem that the softened result does not have the right type (same size integer instead of float type). To workaround the problem, the patch skips 'softening' completely for some opcodes such as BITCAST, SELECT, and as a result, has to change the prototype of SoftenFloatResult and allows operands scanning which should not be necessary if done properly.

It seems to be the right approach is to make sure those variants of SoftenFloatRes_<OPCODE> to return and set the right 'softened' float type for f128. There are two main places that need to be changed: TargetLowering::makeLibCall and BitConvertToInteger by making them f128 aware. Does it make sense?

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

725

Why is keepFloat computed differently from SoftenFloatResult?

lib/CodeGen/SelectionDAG/LegalizeTypes.h

388

If SetSoftenedFloat was called with the right value, then there is no need to do special handling here. This is the reason why I think SoftenFloatResult should not special handling those op code.

[chh]

David, I have reconsidered a few alternatives and checked again current llvm architecture.

First a few observations for other people not following all the issues before:

The core problem is that llvm classifies types into
legally-in-register-with-HW-instruction or
illegal-and-should-be-expanded-promoted-or-soften.
f128 is classified as illegal for most targets and soften to i128,
which is passed/returned in two 64-bit registers.
The SoftenFloat* functions not only convert operations to library function calls,
but also the result type to i128. 
But on x86_64, calling convention for i128 is different from f128.
GCC and llvm are correct for i128, but llvm incorrectly passed f128 the same as i128.

To give f128 its own calling convention, I still don't have other solution but
to create its own register class. That makes it a new kind of type not well fit
into current scheme.

It does not seem feasible to me to keep using SoftenFloat* functions and
convert f128 to i128, and then add other hacks to use different calling convention
for real i128 and f128-converted-i128. If it is possible, please let me know
how to do it.

If I understand correctly, your suggestion is to fix some SoftenFloatRes_* functions
to return f128 for f128 in this case instead of skipping them.
If we want to fix them and avoid the changes to LegalizeTypes.cpp, they will
also need to scan-and-convert operands, because current LegalizeTypes.cpp simply
"goto NodeDone" after calling SoftenFloatResult.
Wouldn't that change duplicate the ScanOperands actions from LegalizeTypes.cpp to
LegalizeFloatTypes.cpp?

I did not like the duplication of ScanOperands, so I added new return value to
SoftenFloatResult to indicate if ScanOperands is necessary. Then, some
node types no longer need any change inside SoftenFloatResult.

Maybe what I should do is adding more comments to SoftenFloat* functions to
explain that they now do not always convert floating point types to integer types.
In fact, they convert nodes to the TransformToType[VT], which could be f128 or i128.
Current comment of getTypeToTransformTo says:

/// For types supported by the target, this is an identity function....

That can be extended to as "supported by the target or passed/returned in registers".

Another alternative is to keep the exact meaning of LegalizeTypeAction::TypeSoftenFloat
and add a new action like TypeSoftenOnlyFloatOps. We still need to change comments of
several functions to deal with the new legally in register but no HW support f128 type.
For other targets, with the same calling convention for f128 and i128,
they can keep on using TypeSoftenFloat, which still converts nodes to the
TransformToType[VT] but that is assumed to be an integer type.
For x86_64, the new TypeSoftenOnlyFloatOps action will assume that TransformToType[VT]
could be the same as VT and VT could be legally in register.

At the end, SoftenFloat* and SoftenOnlyFloatOps* functions could share some code
to minimize duplication. The shared code would have some flag to serve similar purpose
as the KeepFloat variable in my patch.
Many places that handle LegalizeTypeAction::TypeSoftenFloat will need to be changed
to also handle LegalizeTypeAction::TypeSoftenOnlyFloatOps.
That doesn't seem to be a simpler patch either.

BTW, I am going to LLVM dev meeting too.
So if I don't have time left this week, I will continue next week.

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

725

SoftenFloatResult converts its given SDNode N,
so its KeepFloat is computed from N's value type:

EVT VT = N->getValueType(ResNo);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
bool KeepFloat = VT.isSimple() && VT == NVT && TLI.isTypeLegal(VT);

SoftenFloatOperand converts N's operand,
so its KeepFloat is computed from the OpNo operand:

EVT OpVT = N->getOperand(OpNo).getValueType();
bool KeepFloat = OpVT.isSimple() && TLI.isTypeLegal(OpVT);

The extra check of VT == TLI.getTypeToTransformTo(....)
is not really necessary when we have only x86_64 f128 that
can be legal and transformed to itself at the same time.
I can remove it, or keep it as an redundant check in both places
to filter out any other new unexpected configuration.

Is this the difference you noticed?
Would you rather keep or remove (VT == NVT ) at both places?

lib/CodeGen/SelectionDAG/LegalizeTypes.h

388

In LegalizeTypesGeneric.cpp, you can see that a caller of GetSoftenedFloat
assumes that a softened float should be split into two integers.
We don't want that. So either we modify the meaning of TargetLowering::TypeSoftenFloat
and GetSoftenedFloat or we need to use a new action type like TypeSoftenOnlyFloatOps.

[chh]

Sync with latest source and update comments about TypeSoftenFloat, SoftenFloat*, KeepFloat, and GetSoftenedFloat.
New comments should clarify that only non-HW-supported operations need to be converted and use integer type.
We should think about non-HW-supported operations instead of "illegal" types.
SoftenFloat is really about converting float operations, not necessarily converting to integers.

[davidxl]

I am still not convinced that you need to special handling certain opcodes in SoftenFloatResult and returns a bool to indicate if operands need to be scanned.

The difference I can see is that without this patch, CopyToReg and CopyFromReg opcodes are expected to be already type legalized. With your patch, can you add these functions:

SoftenFloatRes_CopyToReg and SoftenFloatRes_CopyFromReg -- it does not need to do anything but it makes sure that the 'softened' result is stored. Currently you simply short-circuit it with

default:
if (keepFloat)

return false;

[chh]

Use davidxl's suggestion and add case ISD::Register, CopyFromReg, CopyToReg into SoftenFloatResult. They are now only used when KeepFloat, to keep the node unchanged. All other not handled node opcode will still by default abort the compiler.

[davidxl]

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

60

KeepFloat does not convey the real meaning. Is is better to call it LegalInHWReg?

68

With the handling of CopyFromReg CopyToReg, there does not seem to be a need to special case SELECT, SELECT_CC by briefly looking at the code.

In fact, I think we should push the logic all down to the leaf and get rid of return true/false change. It is the change of 'flow' and special handling of operand scanning for f128 that makes the current patch look intrusive.

143

Which sub-methods need special handling here?

157

Just return SDValue(N, ResNo)?

[chh]

• Rename KeepFloat to LegalInHWReg.
• Clarify some comments.
• Embed into original switch cases the handle of LegalInHWReg for the soften of opcode Register, CopyFromReg, CopyToReg, ConstantFP, FABS, FCOPYSIGN, and FNEG.

[chh]

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

68

David,

Before my patch, DAGTypeLegalizer::run's loop over results always
(1) goto NodeDone with a changed (new) node, or
(2) goto ScanOperands without changed node,

for the TargetLowering::TypeLegal case.

All SoftenFloatRes_* functions made new nodes of integer types.

For x86_64 fp128, we really don't want to change the f128 type so that instruction selection can be correct or more efficient based on f128 type instead of i128.
For BITCAST, SELECT, and SELECT_CC opcode, we need to do down and check their operands as if f128 is TypeLegal. So there is no simpler way to treat it the same as other SoftFloatRes_* functions.
For Register, CopyFromReg, CopyToReg, ConstantFP, FABS, FCOPYSIGN, and FNEG they are simple enough to skip ScanOperands, so we can let them return the same SDNode and pretend that the result node was changed and goto NodeDone. So I simplified all of them into the old switch cases without new extra functions.

143

See updated comment. Examples are those calling TLI.makeLibCall.

153

Okay, changed to an assert, suppress compiler warning, and simple return.

[davidxl]

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

70

I got curious why these opcodes need to be skipped and picked ISD::SELECT to look at.

Commented line 70 out does trigger error -- but further debugging indicate the bug can be in the original code. In particular, PromoteIntRes_SETCC should call ReplaceValueWith, but does not.

Fixing SETCC handling makes the test case TestMax pass.

Can you verify this is a bug triggered by f128 change?

147

Where is the short-cut? Is it the new default handling in SoftenFloatOperand? Making short cut there and have weak handshaking seems fragile.

736

What are the new opcodes that reach here? Please make such opcode explict and add assertions for unexpected ones.

775–776

This seems redundant -- see above early return.

[chh]

Will upload diff 11 soon with suggested changes.

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

70

Yes, I can reproduce that error too. I believe that it happens only with
my new x86_64 f128 configuration, but I have no way to verify for all targets
and illegal types. It's arguably a "bug" as it worked with existing ScanOperands.

However, the reason to skip some Opcode is not to workaround the problem of
missed calls to ReplaceValueWith. We want to keep some Opcode with f128 type
for instruction selection to correct calling convention or more efficient code.
When we do not change SDNode for f128 type, we need to tell LegalizeTypes to
treat it as a "legal" type and go to ScanOperands, so all other existing cases
will work with this unchanged SDNode.

147

Yes, in SoftenFloatOperand, I do not try to repeatedly replace a SDNode
when its operand type can be kept as float and not changed.
Current LegalizeTypes and LegalizeFloatTypes depend on ScanOperands to
replace SDNode where DAGTypeLegalizer::*Result functions "missed" calls
to ReplaceValueWith. That is kind of strange and somewhat inefficient.

However, changing that architecture is too big a risk and work for me to
fix x86_64 f128 calling convention and make sure the change work for
all targets now and in the future. I don't think there are enough regression
tests for all target and types that depend on LegalizeTypes. I found all broken
cases when trying to compile libm on Android.

Hence, I use this safer and "smaller" patch to match existing architecture
and limit the impact to only SoftenFloat* and LegalInHWReg cases.
It could be a separate patch to clean up the complicated loops inside LegalizeTypes,
or LegalizeTypes could be rewritten as we know that there are non-HW-supported operations
on some types but not really "illegal types".

736

Unspecified cases will dump and abort as before.

775–776

Fixed.

[chh]

Update some comments, remove redundant code, and explicitly list legal
operand types in SoftenFloatOperand.

[davidxl]

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

147

I don't quite like the short cut here either. The original architected flow is reasonable to me: basically

1. After softenfloat result, the old SDValue is mapped to the new result
2. the worklist based loop follow DU and use instrctions of the old result get processed -- operands are scanned.
3. during scanning, the use of the old value will be replaced with the new value.
4. if scan operands result in a new node, replaceValueWith then needs to be eagerly called here on demand.

I prefer you do not change the flow in SoftenFloatOperands and make a patch here to workaround it. The resulting change may look bigger (as more opcodes need to be handled), but it is what it is.

I also suggest you extract the following patches out seperately

1. softenFloatResult and softenFloatOperands
2. f128 constantFP
3. register class small refactoring
4. the rest (may also be breakable )

[chh]

David,

I will upload a new patch with some of your older suggested changes first.
Please see details in the new code diff and upload comment.

I met many difficulties implementing your latest inline comments
in LegalizeFloatTypes.cpp, so there is no change with that idea yet.
Please see my reply to the inline comment.

For separating this large patch, I have not tried yet,
but it might be okay to split at least into two parts:
(2) The new unit test files *.ll and X86CallingConv.td, X86ISelLowering.cpp, X86InstrInfo.td, X86InstrSSE.td, X86RegisterInfo.td.
(1) All the other files.

We should be able to check in (1) and expect no change to all targets.
Patch (2) will fix Android x86_64 problem, but still no change to other targets.
I am assuming that we want some time to test (1) before submitting (2).
Further split of (1) might be possible, but that would take more review
and test cycles if there is little regression. When there is any regression,
it should be easy to revert all changes in (1) and then debug-and-retry, right?
I don't see a way to split (2), as we don't want to have the Android target
in a new incomplete state different from the current one.

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

147

The difficulties I found in the current architecture are
(1) Avoid duplicating a lot of code from SoftenFloatRes_* to SoftenFloatOp_*.
(2) Make sure the changes in multiple SoftenFloatRes_* and SoftenFloatOp_* will be fixed correctly at the same time in the future.

In SoftenFloatOperand, we can see that it calls ReplaceValueWith
at the end for all kinds of opcode. The number of calls to ReplaceValueWith
is not as many as we expected. Hence, it seems to me the norm is to share
code at higher level than pushing down into SoftenFloatOp_* and SoftenFloatRes_*.
It seems reasonable to call ReplaceValueWith at the end of
SoftenFloatOperand and SoftenFloatResult.

I tried to remove this part, but the result was either
(a) calling ReplaceValuewith inside SoftenFloatRes_*, or
(b) copying many SoftenFloatRes_* code into SoftenFloatOp_* and then call ReplaceValueWith in SoftenFloatOp_*. The copied code is not all simple and small. For example, SoftenFloatRes_FCOPYSIGN is complicated, and I don't see how simpler it is to duplicate it into SoftenFloatOp_FCOPYSIGN, or abstract it out into a shared function.

What you mentioned in another email earlier might be better to
move some SoftenFloatOp_* responsibility into SoftenFloatRes_*.
Then, we have less code duplication, and we need to make sure
that ReplaceValueWith is called whenever something is changed
by SoftenFloatRes_*. I could try that when I have more time.

My concern (2) is more scaring. During my test of those code movement,
I made a few mistakes and the compiler could run and generate inefficient
code because some ReplaceValueWith call was missed or some fp128 was
not kept in register due to a short-cut was missed. So I would prefer
a way easier and smaller change to verify no no-impact to existing targets,
by conditioned on isLegalInHWReg, even if some can be applied to other targets.

[chh]

Push short cuts inside SoftenFloatResult down to SoftenFloatResultOp_* functions,
which now take extra argument ResNo. SoftenFloatResult is cleaner and faster,
SoftenFloatResultOp_* functions have some repeated patterns.

Abstract out the function isLegalInHWReg used in multiple places.
Spell out more short-cut opcode inside SoftenFloatOperand when isLegalInHWReg.
Add assert of isLegalInHWReg after call of SoftenFloatResult when result is not changed.

[chh]

Split out part 1 in http://reviews.llvm.org/D15134.

[chh]

Sync with latest llvm source and new changes suggested in http://reviews.llvm.org/D15134.

[chh]

Now the diff does not contain changes in submitted part1.

[davidxl]

lib/Target/X86/X86ISelLowering.cpp

14274

This needs some explanation. Why can the Op1's value type be i128?

26794

Why TODO here? 'x' constraint should work.

26907

Explain TODO here.

lib/Target/X86/X86InstrInfo.td

955–960

Unrelated format change here.

[davidxl]

lib/Target/X86/X86InstrSSE.td

8878

Move the comment above the pattern def.

1. movaps is shorter, not 'should be'
2. regarding 'faster' part -- put a reference there. In fact, f128 operations should be considered in integer domain so movdqa should be used to avoid domain bypass penalty.

8884

pand is for SIMD integer. andps is shorter though.

[chh]

lib/Target/X86/X86ISelLowering.cpp

14274

Removed it. It was an condition only triggered by my older hacks.
Not it should not happen.

26794

f128 and i128 were not in SSE_REG before.
So I think this is a new feature that can be added later.

26907

Same as above. f128 and i128 not in SSE_REG before this change.

lib/Target/X86/X86InstrInfo.td

955–960

They are changed to align up with new line 962.

lib/Target/X86/X86InstrSSE.td

8878

I updated comment to keep only the shorter and SSE reasons.
I was not sure about 'faster with movaps',
which seems to be used more in clang than gcc.
My main reasons are shorter and available in SSE for Android's applications.

8884

I also choose andps over pand for shorter and availability in SSE,
not sure about performance difference when combined with other instructions.
I think by not splitting f128 into two registers, we already saved more code and execution time.

[davidxl]

lib/Target/X86/X86InstrInfo.td

955–961

It seems you are adding extra space before :

lib/Target/X86/X86InstrSSE.td

8887

Is there any coding style guidelines for table gen code? There are long lines that are wrapped ..

test/CodeGen/X86/fp128-calling-conv.ll

16

Is this a relevant test? non-f128 related tests can be submitted in a different patch.

test/CodeGen/X86/fp128-cast.ll

13

There are also lots of irrelevant tests added in this file .

[chh]

lib/Target/X86/X86InstrInfo.td

955–961

Yes, it seems to be the style at line 964-969 too.

lib/Target/X86/X86InstrSSE.td

8887

Note sure if there is special rule for table gen code.
http://llvm.org/docs/CodingStandards.html says 80 columns.
There are few exceptions in this file,
but now I wrapped all my new lines to less than 80 characters.

test/CodeGen/X86/fp128-calling-conv.ll

16

Okay, i will put non-fp128 type tests to another patch.

test/CodeGen/X86/fp128-cast.ll

13

Okay, i will put non-fp128 type tests to another patch.

[davidxl]

test/CodeGen/X86/fp128-cast.ll

155

Please move this test case to a different patch.

test/CodeGen/X86/fp128-compare.ll

5

Move this test case out of the patch -- -there are more than 20 or so test cases below that need to be separated out.

[chh]

Will reduce fp128-compare.ll in the next diff.

test/CodeGen/X86/fp128-cast.ll

155

This is about f128 calling convention to return f128 in SSE.
Are you sure about removing this one?

[davidxl]

test/CodeGen/X86/fp128-cast.ll

155

you are right -- this one is relevant.

Please check the rest.

[chh]

Removed test cases without f128 type.

[davidxl]

great. I will look at the tests in more details.

David

[davidxl]

test/CodeGen/X86/fp128-calling-conv.ll

6

Is this variable used?

9

Is it used?

12

Are the two parts swapped? GCC seems to generates: 3FFF0000000000000000000000000000

test/CodeGen/X86/fp128-cast.ll

13

TestFPExtF32_F128

26

TestFPExt ..

39

Missing a test for conversion to unsigned I32

127

might be better to relax the check a little : testq %rax, %rax should be fine too.

136

Can you simplify the variable names ?

167

There is no guarantee adcq will be after movq ...

test/CodeGen/X86/fp128-compare.ll

24

missing check of 'set<cc>'

34

is this correct?

46

Missing check

test/CodeGen/X86/fp128-i128.ll

5

Better comment?

93

var names can be cleaned up to be shorter.

126

seems irrelevant.

138

Seems irrelevant.

206

Can this test case be simplified more?

[chh]

test/CodeGen/X86/fp128-calling-conv.ll

6

No, Removed now.

9

No. Removed now.

12

That looked strange but correct. I copied that from clang's dump, and the llvm output assembly code is the same as gcc's. http://llvm.org/docs/LangRef.html says big-endian is used for hexadecimal floating point constants.

test/CodeGen/X86/fp128-cast.ll

39

Added conversion to uint32 and uint64.

136

These were generated by clang for my simplified C code from libm.
They are useful to show the clang transformations.
I will add the C code example as comments.

167

Okay, relaxed the check patterns.

test/CodeGen/X86/fp128-compare.ll

34

Yes. It's a strange optimization, which returns 1 if %cmp is negative as __lttf2 will return when %d1 < %d2.

test/CodeGen/X86/fp128-i128.ll

93

These IL were copied from libm compiled code. Clang has its way to convert C union structure references. I will add original C code as comment.

126

Okay, removed the i64 test.

138

We need to test i128 too, since this patch put also i128 into the FR128 register class. The i128 instruction was generated from f128 C code.

206

This was from libm C code, which triggered one error related to f128 without a complete patch. So I added it. I tried to reduce the original C code but then that won't trigger the problem.

[davidxl]

test/CodeGen/X86/fp128-cast.ll

137

Why do we care what transformations have been done to get the IR? The IR code should by itself readable -- so while the C example is useful, I still prefer the naming in IR simplified.

test/CodeGen/X86/fp128-compare.ll

35

ok. But it is possible with test + sets, right? may be adding a comment so that people know how to fix the test if it breaks in the future?

test/CodeGen/X86/fp128-i128.ll

8

__float128

24

long double --> __float128?

45

The pattern checked is pretty long -- I worry it may break in the future. Is it possible to relax it some how?

58

long double --> __float128

82

long double --> __float128

107

long double --> __float128

[chh]

test/CodeGen/X86/fp128-cast.ll

137

The comment now seems to be showing at wrong place in this code diff.
The biggest confusing name is u.sroa.0.4.extract.shift in TestBits128.
So I will shorten long names in this function for now.

The C code is important for anyone in the future to test, if not having time to rebuild and test the whole AOSP with libm.
Any simplification of the IR might work, but clang could generate different bit operators for those long double union types and trigger problems with any ad-hoc f128 optimization.

test/CodeGen/X86/fp128-compare.ll

35

Sure, added comment.
If it is broken in the future, maybe it would be easier to continue using this trick. :-)

test/CodeGen/X86/fp128-i128.ll

8

Unfortunately, clang does not accept __float128 keyword, although it can emit f128 for llvm.

24

no __float128 in clang.

45

I tried to reduce the C code, but any reduction won't trigger the complicated IL that reached a point that my partial fix core dumped. Maybe we can take out a few CHECK-NEXT requirements.

On the other hard, I was terrified by so many ad-hoc optimizations of floating points for the usage patterns in libm.
I guess llvm tried to match or do better then gcc and libm tried to use every possible bit operations.
So maybe it is better for Android or anyone depends on f128 type to have more check rules here.
Whoever changes float optimization in the future has better fully understand and update these tests.

[davidxl]

test/CodeGen/X86/fp128-i128.ll

9

This should be fixed in clang FE. By default, long double is extended FP, not quadFP --- so do fix the comment to avoid confusion.

[chh]

test/CodeGen/X86/fp128-i128.ll

9

Used __float128 here and added more comments at the top of file.

[davidxl]

LGTM.

[chh]

Fix infinite loop in SelectionDAGBuilder.cpp, caught by new regression tests in this patch.
It happens only with new f128 type, whose VT == TLI.getTypeToTransformTo(Ctx, VT).

[chh]

David, thanks for the review suggestions and approval.
There is one recent regression caught by two of my unit tests,
so I need to fix it too in the updated diff.

include/llvm/Target/TargetLowering.h

private:
   /// For each condition code (ISD::CondCode) keep a LegalizeAction that
   /// indicates how instruction selection should deal with the condition code.
   ///
   /// Because each CC action takes up 2 bits, we need to have the array size be
   /// large enough to fit all of the value types. This can be done by rounding
   /// up the MVT::LAST_VALUETYPE value to the next multiple of 16.
   uint32_t CondCodeActions[ISD::SETCC_INVALID][(MVT::LAST_VALUETYPE + 15) / 16];
 
+protected:
   ValueTypeActionImpl ValueTypeActions;
 
 private:
   LegalizeKind getTypeConversion(LLVMContext &Context, EVT VT) const;
 
 private:
   std::vector<std::pair<MVT, const TargetRegisterClass*> > AvailableRegClasses;
 
   /// Targets can specify ISD nodes that they would like PerformDAGCombine
   /// callbacks for by calling setTargetDAGCombine(), which sets a bit in this
   /// array.
   unsigned char
   TargetDAGCombineArray[(ISD::BUILTIN_OP_END+CHAR_BIT-1)/CHAR_BIT];
 
   /// For operations that must be promoted to a specific type, this holds the
   /// destination type.  This map should be sparse, so don't hold it as an
   /// array.
   ///
   /// Targets add entries to this map with AddPromotedToType(..), clients access
   /// this with getTypeToPromoteTo(..).
   std::map<std::pair<unsigned, MVT::SimpleValueType>, MVT::SimpleValueType>

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitFCOPYSIGN(SDNode *N) {
 
   // copysign(x, copysign(y,z)) -> copysign(x, z)
   if (N1.getOpcode() == ISD::FCOPYSIGN)
     return DAG.getNode(ISD::FCOPYSIGN, SDLoc(N), VT,
                        N0, N1.getOperand(1));
 
   // copysign(x, fp_extend(y)) -> copysign(x, y)
   // copysign(x, fp_round(y)) -> copysign(x, y)
-  if (N1.getOpcode() == ISD::FP_EXTEND || N1.getOpcode() == ISD::FP_ROUND)
+  // Do not optimize out type conversion of f128 type yet.
+  // Store and truncate operators for f128 are very limited.
+  EVT N1VT = N1->getValueType(0);
+  EVT N1Op0VT = N1->getOperand(0)->getValueType(0);
+  if ((N1.getOpcode() == ISD::FP_EXTEND || N1.getOpcode() == ISD::FP_ROUND) &&
+      (N1VT == N1Op0VT || (N1VT != MVT::f128 && N1Op0VT != MVT::f128)))
     return DAG.getNode(ISD::FCOPYSIGN, SDLoc(N), VT,
                        N0, N1.getOperand(0));
 
   return SDValue();
 }
 
 SDValue DAGCombiner::visitSINT_TO_FP(SDNode *N) {
   SDValue N0 = N->getOperand(0);

lib/CodeGen/SelectionDAG/InstrEmitter.cpp

for (SDNode *User : Node->uses()) {
             const TargetRegisterClass *RC = nullptr;
             if (i+II.getNumDefs() < II.getNumOperands()) {
               RC = TRI->getAllocatableClass(
                 TII->getRegClass(II, i+II.getNumDefs(), TRI, *MF));
             }
             if (!UseRC)
               UseRC = RC;
             else if (RC) {
               const TargetRegisterClass *ComRC =
                 TRI->getCommonSubClass(UseRC, RC);
+              // Current TRI->getCommonSubClass might return a register class
+              // ComRC such that ComRC->hasType(Node->getSimpleValueType(ResNo))
+              // is not true. This could happen when RC or UseRC is FR128.
+              // Until that behavior of FR128 is changed,
+              // we need to handle such case later before using UseRC.
               // If multiple uses expect disjoint register classes, we emit
               // copies in AddRegisterOperand.
               if (ComRC)
                 UseRC = ComRC;
             }
           }
         }
       }
       MatchReg &= Match;
       if (VRBase)
         break;
     }
 
   const TargetRegisterClass *SrcRC = nullptr, *DstRC = nullptr;
   SrcRC = TRI->getMinimalPhysRegClass(SrcReg, VT);
 
   // Figure out the register class to create for the destreg.
   if (VRBase) {
     DstRC = MRI->getRegClass(VRBase);
-  } else if (UseRC) {
-    assert(UseRC->hasType(VT) && "Incompatible phys register def and uses!");
+  } else if (UseRC && UseRC->hasType(VT)) {
+    // If TRI->getCommonSubClass(UseRC, RC) returns some register class,
+    // e.g. FR128, that does not contain VT, it should not be used for DstRC.
     DstRC = UseRC;
   } else {
     DstRC = TLI->getRegClassFor(VT);
   }
 
   // If all uses are reading from the src physical register and copying the
   // register is either impossible or very expensive, then don't create a copy.
   if (MatchReg && SrcRC->getCopyCost() < 0) {

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

private:
   SDValue ExpandBSWAP(SDValue Op, SDLoc dl);
   SDValue ExpandBitCount(unsigned Opc, SDValue Op, SDLoc dl);
 
   SDValue ExpandExtractFromVectorThroughStack(SDValue Op);
   SDValue ExpandInsertToVectorThroughStack(SDValue Op);
   SDValue ExpandVectorBuildThroughStack(SDNode* Node);
 
   SDValue ExpandConstantFP(ConstantFPSDNode *CFP, bool UseCP);
+  SDValue ExpandConstant(ConstantSDNode *CP);
 
   std::pair<SDValue, SDValue> ExpandAtomic(SDNode *Node);
 
   void ExpandNode(SDNode *Node);
   void PromoteNode(SDNode *Node);
 
 public:
   // Node replacement helpers
SelectionDAGLegalize::ExpandConstantFP(ConstantFPSDNode *CFP, bool UseCP) {
   }
   SDValue Result =
       DAG.getLoad(OrigVT, dl, DAG.getEntryNode(), CPIdx,
                   MachinePointerInfo::getConstantPool(DAG.getMachineFunction()),
                   false, false, false, Alignment);
   return Result;
 }
 
+/// Expands the Constant node to a load from the constant pool.
+SDValue SelectionDAGLegalize::ExpandConstant(ConstantSDNode *CP) {
+  SDLoc dl(CP);
+  EVT VT = CP->getValueType(0);
+  SDValue CPIdx = DAG.getConstantPool(CP->getConstantIntValue(),
+                                      TLI.getPointerTy(DAG.getDataLayout()));
+  unsigned Alignment = cast<ConstantPoolSDNode>(CPIdx)->getAlignment();
+  SDValue Result =
+    DAG.getLoad(VT, dl, DAG.getEntryNode(), CPIdx,
+                MachinePointerInfo::getConstantPool(DAG.getMachineFunction()),
+                false, false, false, Alignment);
+  return Result;
+}
+
 /// Expands an unaligned store to 2 half-size stores.
 static void ExpandUnalignedStore(StoreSDNode *ST, SelectionDAG &DAG,
                                  const TargetLowering &TLI,
                                  SelectionDAGLegalize *DAGLegalize) {
   assert(ST->getAddressingMode() == ISD::UNINDEXED &&
          "unaligned indexed stores not implemented!");
   SDValue Chain = ST->getChain();
   SDValue Ptr = ST->getBasePtr();

 void SelectionDAGLegalize::LegalizeOp(SDNode *Node) {
   DEBUG(dbgs() << "\nLegalizing: "; Node->dump(&DAG));
 
   if (Node->getOpcode() == ISD::TargetConstant) // Allow illegal target nodes.
     return;
 
 #ifndef NDEBUG
   for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i)
-    assert(TLI.getTypeAction(*DAG.getContext(), Node->getValueType(i)) ==
-             TargetLowering::TypeLegal &&
+    assert((TLI.getTypeAction(*DAG.getContext(), Node->getValueType(i)) ==
+              TargetLowering::TypeLegal ||
+            TLI.isTypeLegal(Node->getValueType(i))) &&
            "Unexpected illegal type!");
 
   for (const SDValue &Op : Node->op_values())
-    assert((TLI.getTypeAction(*DAG.getContext(),
-                              Op.getValueType()) == TargetLowering::TypeLegal ||
-                              Op.getOpcode() == ISD::TargetConstant) &&
-                              "Unexpected illegal type!");
+    assert((TLI.getTypeAction(*DAG.getContext(), Op.getValueType()) ==
+              TargetLowering::TypeLegal ||
+            TLI.isTypeLegal(Op.getValueType()) ||
 #endif
 
   // Figure out the correct action; the way to query this varies by opcode
   TargetLowering::LegalizeAction Action = TargetLowering::Legal;
   bool SimpleFinishLegalizing = true;
   switch (Node->getOpcode()) {
   case ISD::INTRINSIC_W_CHAIN:
   case ISD::INTRINSIC_WO_CHAIN:
void SelectionDAGLegalize::ExpandNode(SDNode *Node) {
   case ISD::ConstantFP: {
     ConstantFPSDNode *CFP = cast<ConstantFPSDNode>(Node);
     // Check to see if this FP immediate is already legal.
     // If this is a legal constant, turn it into a TargetConstantFP node.
     if (!TLI.isFPImmLegal(CFP->getValueAPF(), Node->getValueType(0)))
       Results.push_back(ExpandConstantFP(CFP, true));
     break;
   }
+  case ISD::Constant: {
+    ConstantSDNode *CP = cast<ConstantSDNode>(Node);
+    Results.push_back(ExpandConstant(CP));
+    break;
+  }
   case ISD::FSUB: {
     EVT VT = Node->getValueType(0);
     if (TLI.isOperationLegalOrCustom(ISD::FADD, VT) &&
         TLI.isOperationLegalOrCustom(ISD::FNEG, VT)) {
       const SDNodeFlags *Flags = &cast<BinaryWithFlagsSDNode>(Node)->Flags;
       Tmp1 = DAG.getNode(ISD::FNEG, dl, VT, Node->getOperand(1));
       Tmp1 = DAG.getNode(ISD::FADD, dl, VT, Node->getOperand(0), Tmp1, Flags);
       Results.push_back(Tmp1);

lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

return
     VT == MVT::ppcf128 ? Call_PPCF128 :
     RTLIB::UNKNOWN_LIBCALL;
 }
 
 //===----------------------------------------------------------------------===//
 //  Result Float to Integer Conversion.
 //===----------------------------------------------------------------------===//
 
-void DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
+bool DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
   DEBUG(dbgs() << "Soften float result " << ResNo << ": "; N->dump(&DAG);
         dbgs() << "\n");
-  SDValue R = SDValue();
+  SDValue R(N, ResNo);
+
+  // Some types, e.g. x86_64's f128, want to be legally in registers
+  // but need some operations converted to library calls or integer
+  // bitwise operations
+  EVT VT = N->getValueType(ResNo);
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
+  bool KeepFloat = VT.isSimple() && VT == NVT && TLI.isTypeLegal(VT);
+
+  // When the value can be kept in register, like f128 in FR128 for x86_64,
+  // we do not soften some op code but will go on to legalize the operands.
+  if (KeepFloat)
+    switch (N->getOpcode()) {
+      case ISD::BITCAST:
+      case ISD::SELECT:
+      case ISD::SELECT_CC:
+        return false;
+      // On x86_64 FABS, FNEG, and FCOPYSIGN can be simplified to
+      // native bitwise operations.
+      case ISD::FABS:
+      case ISD::FCOPYSIGN:
+      case ISD::FNEG:
+      // We can load better from the constant pool.
+      // Change it and not to scan the operand again.
+      case ISD::ConstantFP:
+        // Mark this node as softened.
+        SetSoftenedFloat(R, R);
+        return true;
+    }
 
   switch (N->getOpcode()) {
   default:
+    // For CopyToReg, CopyFromReg, etc.
+    if (KeepFloat)
+      return false;
 #ifndef NDEBUG
     dbgs() << "SoftenFloatResult #" << ResNo << ": ";
     N->dump(&DAG); dbgs() << "\n";
 #endif
     llvm_unreachable("Do not know how to soften the result of this operator!");
 
     case ISD::MERGE_VALUES:R = SoftenFloatRes_MERGE_VALUES(N, ResNo); break;
     case ISD::BITCAST:     R = SoftenFloatRes_BITCAST(N); break;
#endif
     case ISD::FPOWI:       R = SoftenFloatRes_FPOWI(N); break;
     case ISD::FREM:        R = SoftenFloatRes_FREM(N); break;
     case ISD::FRINT:       R = SoftenFloatRes_FRINT(N); break;
     case ISD::FROUND:      R = SoftenFloatRes_FROUND(N); break;
     case ISD::FSIN:        R = SoftenFloatRes_FSIN(N); break;
     case ISD::FSQRT:       R = SoftenFloatRes_FSQRT(N); break;
     case ISD::FSUB:        R = SoftenFloatRes_FSUB(N); break;
     case ISD::FTRUNC:      R = SoftenFloatRes_FTRUNC(N); break;
-    case ISD::LOAD:        R = SoftenFloatRes_LOAD(N); break;
+    case ISD::LOAD:        R = SoftenFloatRes_LOAD(N, KeepFloat); break;
     case ISD::SELECT:      R = SoftenFloatRes_SELECT(N); break;
     case ISD::SELECT_CC:   R = SoftenFloatRes_SELECT_CC(N); break;
     case ISD::SINT_TO_FP:
     case ISD::UINT_TO_FP:  R = SoftenFloatRes_XINT_TO_FP(N); break;
     case ISD::UNDEF:       R = SoftenFloatRes_UNDEF(N); break;
     case ISD::VAARG:       R = SoftenFloatRes_VAARG(N); break;
   }
 
   // If R is null, the sub-method took care of registering the result.
-  if (R.getNode())
+  if (R.getNode()) {
     SetSoftenedFloat(SDValue(N, ResNo), R);
+    // Some soften sub-methods do not call ReplaceValueWith and depend
+    // on the ScanOperands in LegalizeTypes.cpp to replace every operand.
+    // But SoftenFloatOperand is not smart enough to replace every operand.
+    // See CodeGen/X86/fp128-abs.ll, fp128-load.ll.
+    if (R.getNode() != N && KeepFloat)
+      ReplaceValueWith(SDValue(N, ResNo), R);
+  }
+  return true;
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatRes_BITCAST(SDNode *N) {
   return BitConvertToInteger(N->getOperand(0));
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatRes_MERGE_VALUES(SDNode *N,
                                                       unsigned ResNo) {
   SDValue Op = DisintegrateMERGE_VALUES(N, ResNo);
   return BitConvertToInteger(Op);
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatRes_BUILD_PAIR(SDNode *N) {
   // Convert the inputs to integers, and build a new pair out of them.
   return DAG.getNode(ISD::BUILD_PAIR, SDLoc(N),
return TLI.makeLibCall(DAG, GetFPLibCall(N->getValueType(0),
                                            RTLIB::TRUNC_F32,
                                            RTLIB::TRUNC_F64,
                                            RTLIB::TRUNC_F80,
                                            RTLIB::TRUNC_F128,
                                            RTLIB::TRUNC_PPCF128),
                          NVT, &Op, 1, false, SDLoc(N)).first;
 }
 
-SDValue DAGTypeLegalizer::SoftenFloatRes_LOAD(SDNode *N) {
+SDValue DAGTypeLegalizer::SoftenFloatRes_LOAD(SDNode *N, bool KeepFloat) {
   LoadSDNode *L = cast<LoadSDNode>(N);
   EVT VT = N->getValueType(0);
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
   SDLoc dl(N);
 
   SDValue NewL;
   if (L->getExtensionType() == ISD::NON_EXTLOAD) {
     NewL = DAG.getLoad(L->getAddressingMode(), L->getExtensionType(),
                        NVT, dl, L->getChain(), L->getBasePtr(), L->getOffset(),
                        L->getPointerInfo(), NVT, L->isVolatile(),
                        L->isNonTemporal(), false, L->getAlignment(),
                        L->getAAInfo());
     // Legalized the chain result - switch anything that used the old chain to
     // use the new one.
+    if (N != NewL.getValue(1).getNode())
       ReplaceValueWith(SDValue(N, 1), NewL.getValue(1));
     return NewL;
   }
 
   // Do a non-extending load followed by FP_EXTEND.
   NewL = DAG.getLoad(L->getAddressingMode(), ISD::NON_EXTLOAD,
                      L->getMemoryVT(), dl, L->getChain(),
                      L->getBasePtr(), L->getOffset(), L->getPointerInfo(),
                      L->getMemoryVT(), L->isVolatile(),
                      L->isNonTemporal(), false, L->getAlignment(),
                      L->getAAInfo());
   // Legalized the chain result - switch anything that used the old chain to
   // use the new one.
   ReplaceValueWith(SDValue(N, 1), NewL.getValue(1));
-  return BitConvertToInteger(DAG.getNode(ISD::FP_EXTEND, dl, VT, NewL));
+  auto ExtendNode = DAG.getNode(ISD::FP_EXTEND, dl, VT, NewL);
+  if (KeepFloat)
+    return ExtendNode;
+  return BitConvertToInteger(ExtendNode);
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatRes_SELECT(SDNode *N) {
   SDValue LHS = GetSoftenedFloat(N->getOperand(1));
   SDValue RHS = GetSoftenedFloat(N->getOperand(2));
   return DAG.getSelect(SDLoc(N),
                        LHS.getValueType(), N->getOperand(0), LHS, RHS);
 }
SDValue DAGTypeLegalizer::SoftenFloatRes_VAARG(SDNode *N) {
   SDLoc dl(N);
 
   SDValue NewVAARG;
   NewVAARG = DAG.getVAArg(NVT, dl, Chain, Ptr, N->getOperand(2),
                           N->getConstantOperandVal(3));
 
   // Legalized the chain result - switch anything that used the old chain to
   // use the new one.
+  if (N != NewVAARG.getValue(1).getNode())
     ReplaceValueWith(SDValue(N, 1), NewVAARG.getValue(1));
   return NewVAARG;
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatRes_XINT_TO_FP(SDNode *N) {
   bool Signed = N->getOpcode() == ISD::SINT_TO_FP;
   EVT SVT = N->getOperand(0).getValueType();
   EVT RVT = N->getValueType(0);
   EVT NVT = EVT();

 //===----------------------------------------------------------------------===//
 //  Operand Float to Integer Conversion..
 //===----------------------------------------------------------------------===//
 
 bool DAGTypeLegalizer::SoftenFloatOperand(SDNode *N, unsigned OpNo) {
   DEBUG(dbgs() << "Soften float operand " << OpNo << ": "; N->dump(&DAG);
         dbgs() << "\n");
   SDValue Res = SDValue();
+  EVT OpVT = N->getOperand(OpNo).getValueType();
+  bool KeepFloat = OpVT.isSimple() && TLI.isTypeLegal(OpVT);
+
+  // When an operand is float but can be kept in register,
+  // many of the SoftenFloatOp_* functions will call GetSoftenedFloat
+  // and get the unsoftened float operands, which should be fine.
 
   switch (N->getOpcode()) {
   default:
+    // When result type is legal, N could have op code like CopyToReg,
+    // CopyFromReg, Register, etc. They do not need to be softened.
+    // Assume all child nodes were softened if changed in place.
+    if (KeepFloat)
+      return false;
 #ifndef NDEBUG
     dbgs() << "SoftenFloatOperand Op #" << OpNo << ": ";
     N->dump(&DAG); dbgs() << "\n";
 #endif
     llvm_unreachable("Do not know how to soften this operator's operand!");
 
   case ISD::BITCAST:     Res = SoftenFloatOp_BITCAST(N); break;
   case ISD::BR_CC:       Res = SoftenFloatOp_BR_CC(N); break;
   case ISD::FP_EXTEND:   Res = SoftenFloatOp_FP_EXTEND(N); break;
   case ISD::FP_TO_FP16:  // Same as FP_ROUND for softening purposes
   case ISD::FP_ROUND:    Res = SoftenFloatOp_FP_ROUND(N); break;
   case ISD::FP_TO_SINT:  Res = SoftenFloatOp_FP_TO_SINT(N); break;
   case ISD::FP_TO_UINT:  Res = SoftenFloatOp_FP_TO_UINT(N); break;
   case ISD::SELECT_CC:   Res = SoftenFloatOp_SELECT_CC(N); break;
   case ISD::SETCC:       Res = SoftenFloatOp_SETCC(N); break;
-  case ISD::STORE:       Res = SoftenFloatOp_STORE(N, OpNo); break;
+  case ISD::STORE:
+    Res = SoftenFloatOp_STORE(N, OpNo);
+    // Do not try to analyze or soften this node again if the value is
+    // or can be hold in a register. In that case, Res.getNode() should
+    // be euqal to N.
+    if (Res.getNode() == N && KeepFloat)
+      return false;
+    // Otherwise, we need to reanalyze and lower the new Res nodes.
+    break;
   }
 
   // If the result is null, the sub-method took care of registering results etc.
   if (!Res.getNode()) return false;
 
   // If the result is N, the sub-method updated N in place.  Tell the legalizer
   // core about this.
   if (Res.getNode() == N)
     return true;
 
   assert(Res.getValueType() == N->getValueType(0) && N->getNumValues() == 1 &&
          "Invalid operand expansion");
 
+  if (N != Res.getNode())
     ReplaceValueWith(SDValue(N, 0), Res);
   return false;
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatOp_BITCAST(SDNode *N) {
   return DAG.getNode(ISD::BITCAST, SDLoc(N), N->getValueType(0),
                      GetSoftenedFloat(N->getOperand(0)));
 }
 

lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   // Float to Integer Conversion Support: LegalizeFloatTypes.cpp
   //===--------------------------------------------------------------------===//
 
   /// GetSoftenedFloat - Given a processed operand Op which was converted to an
   /// integer of the same size, this returns the integer.  The integer contains
   /// exactly the same bits as Op - only the type changed.  For example, if Op
   /// is an f32 which was softened to an i32, then this method returns an i32,
   /// the bits of which coincide with those of Op.
+  /// If Op can be legally stored in a register, it might not have been
+  /// converted to an integer. In that cast, the given op is returned.
   SDValue GetSoftenedFloat(SDValue Op) {
     SDValue &SoftenedOp = SoftenedFloats[Op];
+    if (!SoftenedOp.getNode() &&
+        Op.getValueType().isSimple() &&
+        TLI.isTypeLegal(Op.getValueType()))
+      return Op;
     RemapValue(SoftenedOp);
     assert(SoftenedOp.getNode() && "Operand wasn't converted to integer?");
     return SoftenedOp;
   }
   void SetSoftenedFloat(SDValue Op, SDValue Result);
 
   // Result Float to Integer Conversion.
-  void SoftenFloatResult(SDNode *N, unsigned OpNo);
+  bool SoftenFloatResult(SDNode *N, unsigned OpNo);
   SDValue SoftenFloatRes_MERGE_VALUES(SDNode *N, unsigned ResNo);
   SDValue SoftenFloatRes_BITCAST(SDNode *N);
   SDValue SoftenFloatRes_BUILD_PAIR(SDNode *N);
   SDValue SoftenFloatRes_ConstantFP(ConstantFPSDNode *N);
   SDValue SoftenFloatRes_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue SoftenFloatRes_FABS(SDNode *N);
   SDValue SoftenFloatRes_FMINNUM(SDNode *N);
   SDValue SoftenFloatRes_FMAXNUM(SDNode *N);
private:
   SDValue SoftenFloatRes_FPOWI(SDNode *N);
   SDValue SoftenFloatRes_FREM(SDNode *N);
   SDValue SoftenFloatRes_FRINT(SDNode *N);
   SDValue SoftenFloatRes_FROUND(SDNode *N);
   SDValue SoftenFloatRes_FSIN(SDNode *N);
   SDValue SoftenFloatRes_FSQRT(SDNode *N);
   SDValue SoftenFloatRes_FSUB(SDNode *N);
   SDValue SoftenFloatRes_FTRUNC(SDNode *N);
-  SDValue SoftenFloatRes_LOAD(SDNode *N);
+  SDValue SoftenFloatRes_LOAD(SDNode *N, bool KeepFloat);
   SDValue SoftenFloatRes_SELECT(SDNode *N);
   SDValue SoftenFloatRes_SELECT_CC(SDNode *N);
   SDValue SoftenFloatRes_UNDEF(SDNode *N);
   SDValue SoftenFloatRes_VAARG(SDNode *N);
   SDValue SoftenFloatRes_XINT_TO_FP(SDNode *N);
 
   // Operand Float to Integer Conversion.
   bool SoftenFloatOperand(SDNode *N, unsigned OpNo);

lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

for (unsigned i = 0, NumResults = N->getNumValues(); i < NumResults; ++i) {
         PromoteIntegerResult(N, i);
         Changed = true;
         goto NodeDone;
       case TargetLowering::TypeExpandInteger:
         ExpandIntegerResult(N, i);
         Changed = true;
         goto NodeDone;
       case TargetLowering::TypeSoftenFloat:
-        SoftenFloatResult(N, i);
-        Changed = true;
+        // Some fp128 type nodes might not be softened, to keep in
+        // registers for other operations. If not changed, go to
+        // check the operands.
+        Changed = SoftenFloatResult(N, i);
+        if (Changed)
           goto NodeDone;
+        goto ScanOperands;
       case TargetLowering::TypeExpandFloat:
         ExpandFloatResult(N, i);
         Changed = true;
         goto NodeDone;
       case TargetLowering::TypeScalarizeVector:
         ScalarizeVectorResult(N, i);
         Changed = true;
         goto NodeDone;
#endif
 
   // In a debug build, scan all the nodes to make sure we found them all.  This
   // ensures that there are no cycles and that everything got processed.
 #ifndef NDEBUG
   for (SDNode &Node : DAG.allnodes()) {
     bool Failed = false;
 
     // Check that all result types are legal.
+    // A value type is illegal if its TypeAction is not TypeLegal,
+    // and TLI.RegClassForVT does not have a register class for this type.
+    // For example, the x86_64 target has f128 that is not TypeLegal,
+    // to have softened operators, but it also has FR128 register class to
+    // pass and return f128 values. Hence a legalized node can have f128 type.
     if (!IgnoreNodeResults(&Node))
       for (unsigned i = 0, NumVals = Node.getNumValues(); i < NumVals; ++i)
-        if (!isTypeLegal(Node.getValueType(i))) {
-          dbgs() << "Result type " << i << " illegal!\n";
+        if (!isTypeLegal(Node.getValueType(i)) &&
+            !TLI.isTypeLegal(Node.getValueType(i))) {
+          dbgs() << "Result type " << i << " illegal: ";
+          Node.dump();
           Failed = true;
         }
 
     // Check that all operand types are legal.
     for (unsigned i = 0, NumOps = Node.getNumOperands(); i < NumOps; ++i)
       if (!IgnoreNodeResults(Node.getOperand(i).getNode()) &&
-          !isTypeLegal(Node.getOperand(i).getValueType())) {
-        dbgs() << "Operand type " << i << " illegal!\n";
+          !isTypeLegal(Node.getOperand(i).getValueType()) &&
+          !TLI.isTypeLegal(Node.getOperand(i).getValueType())) {
+        dbgs() << "Operand type " << i << " illegal: ";
+        Node.getOperand(i).dump();
         Failed = true;
       }
 
     if (Node.getNodeId() != Processed) {
        if (Node.getNodeId() == NewNode)
          dbgs() << "New node not analyzed?\n";
        else if (Node.getNodeId() == Unanalyzed)
          dbgs() << "Unanalyzed node not noticed?\n";
void DAGTypeLegalizer::SetPromotedInteger(SDValue Op, SDValue Result) {
   AnalyzeNewValue(Result);
 
   SDValue &OpEntry = PromotedIntegers[Op];
   assert(!OpEntry.getNode() && "Node is already promoted!");
   OpEntry = Result;
 }
 
 void DAGTypeLegalizer::SetSoftenedFloat(SDValue Op, SDValue Result) {
-  assert(Result.getValueType() ==
-         TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType()) &&
+  // f128 of x86_64 could be kept in SSE registers,
+  // but sometimes softened to i128.
+  assert((Result.getValueType() ==
+          TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType()) ||
+          Op.getValueType() ==
+          TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType())) &&
          "Invalid type for softened float");
   AnalyzeNewValue(Result);
 
   SDValue &OpEntry = SoftenedFloats[Op];
-  assert(!OpEntry.getNode() && "Node is already converted to integer!");
+  // Allow repeated calls to save f128 type nodes
+  // or any node with type that transforms to itself.
+  // Many operations on these types are not softened.
+  assert((!OpEntry.getNode()||
+          Op.getValueType() ==
+          TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType())) &&
+         "Node is already converted to integer!");
   OpEntry = Result;
 }
 
 void DAGTypeLegalizer::SetPromotedFloat(SDValue Op, SDValue Result) {
   assert(Result.getValueType() ==
          TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType()) &&
          "Invalid type for promoted float");
   AnalyzeNewValue(Result);

lib/CodeGen/SelectionDAG/LegalizeTypesGeneric.cpp

void DAGTypeLegalizer::ExpandRes_BITCAST(SDNode *N, SDValue &Lo, SDValue &Hi) {
   // Handle some special cases efficiently.
   switch (getTypeAction(InVT)) {
     case TargetLowering::TypeLegal:
     case TargetLowering::TypePromoteInteger:
       break;
     case TargetLowering::TypePromoteFloat:
       llvm_unreachable("Bitcast of a promotion-needing float should never need"
                        "expansion");
-    case TargetLowering::TypeSoftenFloat:
-      // Convert the integer operand instead.
-      SplitInteger(GetSoftenedFloat(InOp), Lo, Hi);
+    case TargetLowering::TypeSoftenFloat: {
+      // Expand the floating point operand only if it was softened.
+      // Otherwise, it could be f128 that can be copied to i128 in register.
+      auto SoftenedOp = GetSoftenedFloat(InOp);
+      if (SoftenedOp == InOp)
+        break;
+      SplitInteger(SoftenedOp, Lo, Hi);
       Lo = DAG.getNode(ISD::BITCAST, dl, NOutVT, Lo);
       Hi = DAG.getNode(ISD::BITCAST, dl, NOutVT, Hi);
       return;
+    }
     case TargetLowering::TypeExpandInteger:
     case TargetLowering::TypeExpandFloat: {
       auto &DL = DAG.getDataLayout();
       // Convert the expanded pieces of the input.
       GetExpandedOp(InOp, Lo, Hi);
       if (TLI.hasBigEndianPartOrdering(InVT, DL) !=
           TLI.hasBigEndianPartOrdering(OutVT, DL))
         std::swap(Lo, Hi);

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

case ISD::SINT_TO_FP: {
                                  APFloat::rmNearestTiesToEven);
       return getConstantFP(apf, DL, VT);
     }
     case ISD::BITCAST:
       if (VT == MVT::f16 && C->getValueType(0) == MVT::i16)
         return getConstantFP(APFloat(APFloat::IEEEhalf, Val), DL, VT);
       if (VT == MVT::f32 && C->getValueType(0) == MVT::i32)
         return getConstantFP(APFloat(APFloat::IEEEsingle, Val), DL, VT);
-      else if (VT == MVT::f64 && C->getValueType(0) == MVT::i64)
+      if (VT == MVT::f64 && C->getValueType(0) == MVT::i64)
         return getConstantFP(APFloat(APFloat::IEEEdouble, Val), DL, VT);
+      if (VT == MVT::f128 && C->getValueType(0) == MVT::i128)
+        return getConstantFP(APFloat(APFloat::IEEEquad, Val), DL, VT);
       break;
     case ISD::BSWAP:
       return getConstant(Val.byteSwap(), DL, VT, C->isTargetOpcode(),
                          C->isOpaque());
     case ISD::CTPOP:
       return getConstant(Val.countPopulation(), DL, VT, C->isTargetOpcode(),
                          C->isOpaque());
     case ISD::CTLZ:

lib/CodeGen/SelectionDAG/TargetLowering.cpp

case ISD::BITCAST:
     // thing demanded, turn this into a FGETSIGN.
     if (!TLO.LegalOperations() &&
         !Op.getValueType().isVector() &&
         !Op.getOperand(0).getValueType().isVector() &&
         NewMask == APInt::getSignBit(Op.getValueType().getSizeInBits()) &&
         Op.getOperand(0).getValueType().isFloatingPoint()) {
       bool OpVTLegal = isOperationLegalOrCustom(ISD::FGETSIGN, Op.getValueType());
       bool i32Legal  = isOperationLegalOrCustom(ISD::FGETSIGN, MVT::i32);
-      if ((OpVTLegal || i32Legal) && Op.getValueType().isSimple()) {
+      if ((OpVTLegal || i32Legal) && Op.getValueType().isSimple() &&
+           Op.getOperand(0).getValueType() != MVT::f128) {
+        // Cannot eliminate/lower SHL for f128 yet.
         EVT Ty = OpVTLegal ? Op.getValueType() : MVT::i32;
         // Make a FGETSIGN + SHL to move the sign bit into the appropriate
         // place.  We expect the SHL to be eliminated by other optimizations.
         SDValue Sign = TLO.DAG.getNode(ISD::FGETSIGN, dl, Ty, Op.getOperand(0));
         unsigned OpVTSizeInBits = Op.getValueType().getSizeInBits();
         if (!OpVTLegal && OpVTSizeInBits > 32)
           Sign = TLO.DAG.getNode(ISD::ZERO_EXTEND, dl, Op.getValueType(), Sign);
         unsigned ShVal = Op.getValueType().getSizeInBits()-1;

lib/CodeGen/TargetLoweringBase.cpp

while (true) {
     LegalizeKind LK = getTypeConversion(C, MTy);
 
     if (LK.first == TypeLegal)
       return std::make_pair(Cost, MTy.getSimpleVT());
 
     if (LK.first == TypeSplitVector || LK.first == TypeExpandInteger)
       Cost *= 2;
 
+    // Do not loop with f128 type.
+    if (MTy == LK.second)
+      return std::make_pair(Cost, MTy.getSimpleVT());
+
     // Keep legalizing the type.
     MTy = LK.second;
   }
 }
 
 //===----------------------------------------------------------------------===//
 //  Loop Strength Reduction hooks
 //===----------------------------------------------------------------------===//

lib/Target/X86/X86CallingConv.td

def RetCC_X86_32_VectorCall : CallingConv<[
   CCDelegateTo<RetCC_X86Common>
 ]>;
 
 // X86-64 C return-value convention.
 def RetCC_X86_64_C : CallingConv<[
   // The X86-64 calling convention always returns FP values in XMM0.
   CCIfType<[f32], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f64], CCAssignToReg<[XMM0, XMM1]>>,
+  CCIfType<[f128], CCAssignToReg<[XMM0, XMM1]>>,
 
   // MMX vector types are always returned in XMM0.
   CCIfType<[x86mmx], CCAssignToReg<[XMM0, XMM1]>>,
   CCDelegateTo<RetCC_X86Common>
 ]>;
 
 // X86-Win64 C return-value convention.
 def RetCC_X86_Win64_C : CallingConv<[
def CC_X86_64_C : CallingConv<[
   CCIfType<[v2i1],  CCPromoteToType<v2i64>>,
   CCIfType<[v4i1],  CCPromoteToType<v4i32>>,
   CCIfType<[v8i1],  CCPromoteToType<v8i16>>,
   CCIfType<[v16i1], CCPromoteToType<v16i8>>,
   CCIfType<[v32i1], CCPromoteToType<v32i8>>,
   CCIfType<[v64i1], CCPromoteToType<v64i8>>,
 
   // The first 8 FP/Vector arguments are passed in XMM registers.
-  CCIfType<[f32, f64, v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+  CCIfType<[f32, f64, f128, v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
             CCIfSubtarget<"hasSSE1()",
             CCAssignToReg<[XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7]>>>,
 
   // The first 8 256-bit vector arguments are passed in YMM registers, unless
   // this is a vararg function.
   // FIXME: This isn't precisely correct; the x86-64 ABI document says that
   // fixed arguments to vararg functions are supposed to be passed in
   // registers.  Actually modeling that would be a lot of work, though.
   CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
                           CCIfSubtarget<"hasFp256()",
                           CCAssignToReg<[YMM0, YMM1, YMM2, YMM3,
                                          YMM4, YMM5, YMM6, YMM7]>>>>,
 
   // The first 8 512-bit vector arguments are passed in ZMM registers.
   CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
             CCIfSubtarget<"hasAVX512()",
             CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3, ZMM4, ZMM5, ZMM6, ZMM7]>>>>,
 
   // Integer/FP values get stored in stack slots that are 8 bytes in size and
   // 8-byte aligned if there are no more registers to hold them.
   CCIfType<[i32, i64, f32, f64], CCAssignToStack<8, 8>>,
 
   // Long doubles get stack slots whose size and alignment depends on the
   // subtarget.
-  CCIfType<[f80], CCAssignToStack<0, 0>>,
+  CCIfType<[f80, f128], CCAssignToStack<0, 0>>,
 
   // Vectors get 16-byte stack slots that are 16-byte aligned.
   CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCAssignToStack<16, 16>>,
 
   // 256-bit vectors get 32-byte stack slots that are 32-byte aligned.
   CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
            CCAssignToStack<32, 32>>,
 

lib/Target/X86/X86ISelLowering.cpp

for (unsigned i = 0; i != array_lengthof(IntVTs); ++i) {
     setOperationAction(ISD::SUBE, VT, Custom);
   }
 
   setOperationAction(ISD::BR_JT            , MVT::Other, Expand);
   setOperationAction(ISD::BRCOND           , MVT::Other, Custom);
   setOperationAction(ISD::BR_CC            , MVT::f32,   Expand);
   setOperationAction(ISD::BR_CC            , MVT::f64,   Expand);
   setOperationAction(ISD::BR_CC            , MVT::f80,   Expand);
+  setOperationAction(ISD::BR_CC            , MVT::f128,  Expand);
   setOperationAction(ISD::BR_CC            , MVT::i8,    Expand);
   setOperationAction(ISD::BR_CC            , MVT::i16,   Expand);
   setOperationAction(ISD::BR_CC            , MVT::i32,   Expand);
   setOperationAction(ISD::BR_CC            , MVT::i64,   Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::f32,   Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::f64,   Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::f80,   Expand);
+  setOperationAction(ISD::SELECT_CC        , MVT::f128,  Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::i8,    Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::i16,   Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::i32,   Expand);
   setOperationAction(ISD::SELECT_CC        , MVT::i64,   Expand);
   if (Subtarget->is64Bit())
     setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i32, Legal);
   setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i16  , Legal);
   setOperationAction(ISD::SIGN_EXTEND_INREG, MVT::i8   , Legal);
X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
   setOperationAction(ISD::SELECT          , MVT::i1   , Promote);
   // X86 wants to expand cmov itself.
   setOperationAction(ISD::SELECT          , MVT::i8   , Custom);
   setOperationAction(ISD::SELECT          , MVT::i16  , Custom);
   setOperationAction(ISD::SELECT          , MVT::i32  , Custom);
   setOperationAction(ISD::SELECT          , MVT::f32  , Custom);
   setOperationAction(ISD::SELECT          , MVT::f64  , Custom);
   setOperationAction(ISD::SELECT          , MVT::f80  , Custom);
+  setOperationAction(ISD::SELECT          , MVT::f128 , Custom);
   setOperationAction(ISD::SETCC           , MVT::i8   , Custom);
   setOperationAction(ISD::SETCC           , MVT::i16  , Custom);
   setOperationAction(ISD::SETCC           , MVT::i32  , Custom);
   setOperationAction(ISD::SETCC           , MVT::f32  , Custom);
   setOperationAction(ISD::SETCC           , MVT::f64  , Custom);
   setOperationAction(ISD::SETCC           , MVT::f80  , Custom);
+  setOperationAction(ISD::SETCC           , MVT::f128 , Custom);
   if (Subtarget->is64Bit()) {
     setOperationAction(ISD::SELECT        , MVT::i64  , Custom);
     setOperationAction(ISD::SETCC         , MVT::i64  , Custom);
   }
   setOperationAction(ISD::EH_RETURN       , MVT::Other, Custom);
   setOperationAction(ISD::CATCHRET        , MVT::Other, Custom);
   setOperationAction(ISD::CLEANUPRET      , MVT::Other, Custom);
   // NOTE: EH_SJLJ_SETJMP/_LONGJMP supported here is NOT intended to support
if (!Subtarget->useSoftFloat() && X86ScalarSSEf64) {
     addLegalFPImmediate(APFloat(-0.0f)); // FLD0/FCHS
     addLegalFPImmediate(APFloat(-1.0f)); // FLD1/FCHS
   }
 
   // We don't support FMA.
   setOperationAction(ISD::FMA, MVT::f64, Expand);
   setOperationAction(ISD::FMA, MVT::f32, Expand);
 
-  // Long double always uses X87.
+  // Long double always uses X87, except f128 in MMX.
   if (!Subtarget->useSoftFloat()) {
+    if (Subtarget->is64Bit() && Subtarget->hasMMX()) {
+      addRegisterClass(MVT::f128, &X86::FR128RegClass);
+      ValueTypeActions.setTypeAction(MVT::f128, TypeSoftenFloat);
+      setOperationAction(ISD::FABS , MVT::f128, Custom);
+      setOperationAction(ISD::FNEG , MVT::f128, Custom);
+      setOperationAction(ISD::FCOPYSIGN, MVT::f128, Custom);
+    }
+
     addRegisterClass(MVT::f80, &X86::RFP80RegClass);
     setOperationAction(ISD::UNDEF,     MVT::f80, Expand);
     setOperationAction(ISD::FCOPYSIGN, MVT::f80, Expand);
     {
       APFloat TmpFlt = APFloat::getZero(APFloat::x87DoubleExtended);
       addLegalFPImmediate(TmpFlt);  // FLD0
       TmpFlt.changeSign();
       addLegalFPImmediate(TmpFlt);  // FLD0/FCHS
X86TargetLowering::LowerCallResult(SDValue Chain, SDValue InFlag,
   CCInfo.AnalyzeCallResult(Ins, RetCC_X86);
 
   // Copy all of the result registers out of their specified physreg.
   for (unsigned i = 0, e = RVLocs.size(); i != e; ++i) {
     CCValAssign &VA = RVLocs[i];
     EVT CopyVT = VA.getLocVT();
 
     // If this is x86-64, and we disabled SSE, we can't return FP values
-    if ((CopyVT == MVT::f32 || CopyVT == MVT::f64) &&
+    if ((CopyVT == MVT::f32 || CopyVT == MVT::f64 || CopyVT == MVT::f128) &&
         ((Is64Bit || Ins[i].Flags.isInReg()) && !Subtarget->hasSSE1())) {
       report_fatal_error("SSE register return with SSE disabled");
     }
 
     // If we prefer to use the value in xmm registers, copy it out as f80 and
     // use a truncate to move it from fp stack reg to xmm reg.
     bool RoundAfterCopy = false;
     if ((VA.getLocReg() == X86::FP0 || VA.getLocReg() == X86::FP1) &&
if (VA.isRegLoc()) {
       if (RegVT == MVT::i32)
         RC = &X86::GR32RegClass;
       else if (Is64Bit && RegVT == MVT::i64)
         RC = &X86::GR64RegClass;
       else if (RegVT == MVT::f32)
         RC = &X86::FR32RegClass;
       else if (RegVT == MVT::f64)
         RC = &X86::FR64RegClass;
+      else if (RegVT == MVT::f128)
+        RC = &X86::FR128RegClass;
       else if (RegVT.is512BitVector())
         RC = &X86::VR512RegClass;
       else if (RegVT.is256BitVector())
         RC = &X86::VR256RegClass;
       else if (RegVT.is128BitVector())
         RC = &X86::VR128RegClass;
       else if (RegVT == MVT::x86mmx)
         RC = &X86::VR64RegClass;
static SDValue LowerFABSorFNEG(SDValue Op, SelectionDAG &DAG) {
   if (IsFABS)
     for (SDNode *User : Op->uses())
       if (User->getOpcode() == ISD::FNEG)
         return Op;
 
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
 
+  bool IsF128 = (VT == MVT::f128);
+
   // FIXME: Use function attribute "OptimizeForSize" and/or CodeGenOpt::Level to
   // decide if we should generate a 16-byte constant mask when we only need 4 or
   // 8 bytes for the scalar case.
 
   MVT LogicVT;
   MVT EltVT;
   unsigned NumElts;
 
   if (VT.isVector()) {
     LogicVT = VT;
     EltVT = VT.getVectorElementType();
     NumElts = VT.getVectorNumElements();
+  } else if (IsF128) {
+    // SSE instructions are used for optimized f128 logical operations.
+    LogicVT = MVT::f128;
+    EltVT = VT;
+    NumElts = 1;
   } else {
     // There are no scalar bitwise logical SSE/AVX instructions, so we
     // generate a 16-byte vector constant and logic op even for the scalar case.
     // Using a 16-byte mask allows folding the load of the mask with
     // the logic op, so it can save (~4 bytes) on code size.
     LogicVT = (VT == MVT::f64) ? MVT::v2f64 : MVT::v4f32;
     EltVT = VT;
     NumElts = (VT == MVT::f64) ? 2 : 4;
SDValue Mask =
                   false, false, false, Alignment);
 
   SDValue Op0 = Op.getOperand(0);
   bool IsFNABS = !IsFABS && (Op0.getOpcode() == ISD::FABS);
   unsigned LogicOp =
     IsFABS ? X86ISD::FAND : IsFNABS ? X86ISD::FOR : X86ISD::FXOR;
   SDValue Operand = IsFNABS ? Op0.getOperand(0) : Op0;
 
-  if (VT.isVector())
+  if (VT.isVector() || IsF128)
     return DAG.getNode(LogicOp, dl, LogicVT, Operand, Mask);
 
   // For the scalar case extend to a 128-bit vector, perform the logic op,
   // and extract the scalar result back out.
   Operand = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, LogicVT, Operand);
   SDValue LogicNode = DAG.getNode(LogicOp, dl, LogicVT, Operand, Mask);
   return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, VT, LogicNode,
                      DAG.getIntPtrConstant(0, dl));
 }
 
 static SDValue LowerFCOPYSIGN(SDValue Op, SelectionDAG &DAG) {
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   LLVMContext *Context = DAG.getContext();
   SDValue Op0 = Op.getOperand(0);
   SDValue Op1 = Op.getOperand(1);
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
   MVT SrcVT = Op1.getSimpleValueType();
+  bool IsF128 = (VT == MVT::f128);
 
   // If second operand is smaller, extend it first.
   if (SrcVT.bitsLT(VT)) {
     Op1 = DAG.getNode(ISD::FP_EXTEND, dl, VT, Op1);
     SrcVT = VT;
   }
   // And if it is bigger, shrink it first.
   if (SrcVT.bitsGT(VT)) {
     Op1 = DAG.getNode(ISD::FP_ROUND, dl, VT, Op1, DAG.getIntPtrConstant(1, dl));
     SrcVT = VT;
   }
 
   // At this point the operands and the result should have the same
   // type, and that won't be f80 since that is not custom lowered.
+  assert((VT == MVT::f64 || VT == MVT::f32 || IsF128) &&
+         "Unexpected type in LowerFCOPYSIGN");
 
   const fltSemantics &Sem =
-      VT == MVT::f64 ? APFloat::IEEEdouble : APFloat::IEEEsingle;
+      VT == MVT::f64 ? APFloat::IEEEdouble :
+          (IsF128 ? APFloat::IEEEquad : APFloat::IEEEsingle);
   const unsigned SizeInBits = VT.getSizeInBits();
 
   SmallVector<Constant *, 4> CV(
-      VT == MVT::f64 ? 2 : 4,
+      VT == MVT::f64 ? 2 : (IsF128 ? 1 : 4),
       ConstantFP::get(*Context, APFloat(Sem, APInt(SizeInBits, 0))));
 
   // First, clear all bits but the sign bit from the second operand (sign).
   CV[0] = ConstantFP::get(*Context,
                           APFloat(Sem, APInt::getHighBitsSet(SizeInBits, 1)));
   Constant *C = ConstantVector::get(CV);
   auto PtrVT = TLI.getPointerTy(DAG.getDataLayout());
   SDValue CPIdx = DAG.getConstantPool(C, PtrVT, 16);
 
   // Perform all logic operations as 16-byte vectors because there are no
   // scalar FP logic instructions in SSE. This allows load folding of the
   // constants into the logic instructions.
-  MVT LogicVT = (VT == MVT::f64) ? MVT::v2f64 : MVT::v4f32;
+  MVT LogicVT = (VT == MVT::f64) ? MVT::v2f64 : (IsF128 ? MVT::f128 : MVT::v4f32);
   SDValue Mask1 =
       DAG.getLoad(LogicVT, dl, DAG.getEntryNode(), CPIdx,
                   MachinePointerInfo::getConstantPool(DAG.getMachineFunction()),
                   false, false, false, 16);
+  if (!IsF128)
     Op1 = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, LogicVT, Op1);
   SDValue SignBit = DAG.getNode(X86ISD::FAND, dl, LogicVT, Op1, Mask1);
 
   // Next, clear the sign bit from the first operand (magnitude).
   // If it's a constant, we can clear it here.
   if (ConstantFPSDNode *Op0CN = dyn_cast<ConstantFPSDNode>(Op0)) {
     APFloat APF = Op0CN->getValueAPF();
     // If the magnitude is a positive zero, the sign bit alone is enough.
     if (APF.isPosZero())
-      return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, SrcVT, SignBit,
-                         DAG.getIntPtrConstant(0, dl));
+      return IsF128 ? SignBit :
+          DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, SrcVT, SignBit,
     APF.clearSign();
     CV[0] = ConstantFP::get(*Context, APF);
   } else {
     CV[0] = ConstantFP::get(
         *Context,
         APFloat(Sem, APInt::getLowBitsSet(SizeInBits, SizeInBits - 1)));
   }
   C = ConstantVector::get(CV);
   CPIdx = DAG.getConstantPool(C, PtrVT, 16);
   SDValue Val =
       DAG.getLoad(LogicVT, dl, DAG.getEntryNode(), CPIdx,
                   MachinePointerInfo::getConstantPool(DAG.getMachineFunction()),
                   false, false, false, 16);
   // If the magnitude operand wasn't a constant, we need to AND out the sign.
   if (!isa<ConstantFPSDNode>(Op0)) {
+    if (!IsF128)
       Op0 = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, LogicVT, Op0);
     Val = DAG.getNode(X86ISD::FAND, dl, LogicVT, Op0, Val);
   }
   // OR the magnitude value with the sign bit.
   Val = DAG.getNode(X86ISD::FOR, dl, LogicVT, Val, SignBit);
-  return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, SrcVT, Val,
-                     DAG.getIntPtrConstant(0, dl));
+  return IsF128 ? Val :
+      DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, SrcVT, Val,
 }
 
 static SDValue LowerFGETSIGN(SDValue Op, SelectionDAG &DAG) {
   SDValue N0 = Op.getOperand(0);
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
 
   // Lower ISD::FGETSIGN to (AND (X86ISD::FGETSIGNx86 ...) 1).
if (NewSetCC.getNode()) {
         return DAG.getNode(ISD::TRUNCATE, dl, MVT::i1, NewSetCC);
       return NewSetCC;
     }
   }
 
   // Look for X == 0, X == 1, X != 0, or X != 1.  We can simplify some forms of
   // these.
   if (Op1.getOpcode() == ISD::Constant &&
+      Op1.getValueType() != MVT::i128 &&  // getZExtValue() works up to i64 only.
       (cast<ConstantSDNode>(Op1)->getZExtValue() == 1 ||
        cast<ConstantSDNode>(Op1)->isNullValue()) &&
       (CC == ISD::SETEQ || CC == ISD::SETNE)) {
 
     // If the input is a setcc, then reuse the input setcc or use a new one with
     // the inverted condition.
     if (Op0.getOpcode() == X86ISD::SETCC) {
       X86::CondCode CCode = (X86::CondCode)Op0.getConstantOperandVal(0);
X86TargetLowering::EmitInstrWithCustomInserter(MachineInstr *MI,
   case X86::SEG_ALLOCA_32:
   case X86::SEG_ALLOCA_64:
     return EmitLoweredSegAlloca(MI, BB);
   case X86::TLSCall_32:
   case X86::TLSCall_64:
     return EmitLoweredTLSCall(MI, BB);
   case X86::CMOV_FR32:
   case X86::CMOV_FR64:
+  case X86::CMOV_FR128:
   case X86::CMOV_GR8:
   case X86::CMOV_GR16:
   case X86::CMOV_GR32:
   case X86::CMOV_RFP32:
   case X86::CMOV_RFP64:
   case X86::CMOV_RFP80:
   case X86::CMOV_V2F64:
   case X86::CMOV_V2I64:
static SDValue PerformSELECTCombine(SDNode *N, SelectionDAG &DAG,
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   // If we have SSE[12] support, try to form min/max nodes. SSE min/max
   // instructions match the semantics of the common C idiom x<y?x:y but not
   // x<=y?x:y, because of how they handle negative zero (which can be
   // ignored in unsafe-math mode).
   // We also try to create v2f32 min/max nodes, which we later widen to v4f32.
   if (Cond.getOpcode() == ISD::SETCC && VT.isFloatingPoint() &&
-      VT != MVT::f80 && (TLI.isTypeLegal(VT) || VT == MVT::v2f32) &&
+      VT != MVT::f80 && VT != MVT::f128 &&
+      (TLI.isTypeLegal(VT) || VT == MVT::v2f32) &&
       (Subtarget->hasSSE2() ||
        (Subtarget->hasSSE1() && VT.getScalarType() == MVT::f32))) {
     ISD::CondCode CC = cast<CondCodeSDNode>(Cond.getOperand(2))->get();
 
     unsigned Opcode = 0;
     // Check for x CC y ? x : y.
     if (DAG.isEqualTo(LHS, Cond.getOperand(0)) &&
         DAG.isEqualTo(RHS, Cond.getOperand(1))) {
case 'x':   // SSE_REGS if SSE1 allowed or AVX_REGS if AVX allowed
       default: break;
       // Scalar SSE types.
       case MVT::f32:
       case MVT::i32:
         return std::make_pair(0U, &X86::FR32RegClass);
       case MVT::f64:
       case MVT::i64:
         return std::make_pair(0U, &X86::FR64RegClass);
+      // TODO: handle f128 and i128 in FR128RegClass.
       // Vector types.
       case MVT::v16i8:
       case MVT::v8i16:
       case MVT::v4i32:
       case MVT::v2i64:
       case MVT::v4f32:
       case MVT::v2f64:
         return std::make_pair(0U, &X86::VR128RegClass);
if (DestReg > 0) {
              Class == &X86::FR32XRegClass || Class == &X86::FR64XRegClass ||
              Class == &X86::VR128XRegClass || Class == &X86::VR256XRegClass ||
              Class == &X86::VR512RegClass) {
     // Handle references to XMM physical registers that got mapped into the
     // wrong class.  This can happen with constraints like {xmm0} where the
     // target independent register mapper will just pick the first match it can
     // find, ignoring the required type.
 
+    // TODO: handle f128 and i128 in FR128RegClass.
     if (VT == MVT::f32 || VT == MVT::i32)
       Res.second = &X86::FR32RegClass;
     else if (VT == MVT::f64 || VT == MVT::i64)
       Res.second = &X86::FR64RegClass;
     else if (X86::VR128RegClass.hasType(VT))
       Res.second = &X86::VR128RegClass;
     else if (X86::VR256RegClass.hasType(VT))
       Res.second = &X86::VR256RegClass;

lib/Target/X86/X86InstrCompiler.td

let usesCustomInserter = 1, Uses = [EFLAGS] in {
 
   let Predicates = [FPStackf64] in
     defm _RFP64 : CMOVrr_PSEUDO<RFP64, f64>;
 
   defm _RFP80 : CMOVrr_PSEUDO<RFP80, f80>;
 
   defm _FR32   : CMOVrr_PSEUDO<FR32, f32>;
   defm _FR64   : CMOVrr_PSEUDO<FR64, f64>;
+  defm _FR128  : CMOVrr_PSEUDO<FR128, f128>;
   defm _V4F32  : CMOVrr_PSEUDO<VR128, v4f32>;
   defm _V2F64  : CMOVrr_PSEUDO<VR128, v2f64>;
   defm _V2I64  : CMOVrr_PSEUDO<VR128, v2i64>;
   defm _V8F32  : CMOVrr_PSEUDO<VR256, v8f32>;
   defm _V4F64  : CMOVrr_PSEUDO<VR256, v4f64>;
   defm _V4I64  : CMOVrr_PSEUDO<VR256, v4i64>;
   defm _V8I64  : CMOVrr_PSEUDO<VR512, v8i64>;
   defm _V8F64  : CMOVrr_PSEUDO<VR512, v8f64>;

lib/Target/X86/X86InstrInfo.td

def loadi32 : PatFrag<(ops node:$ptr), (i32 (unindexedload node:$ptr)), [{
   ISD::LoadExtType ExtType = LD->getExtensionType();
   if (ExtType == ISD::NON_EXTLOAD)
     return true;
   if (ExtType == ISD::EXTLOAD)
     return LD->getAlignment() >= 4 && !LD->isVolatile();
   return false;
 }]>;
 
-def loadi8  : PatFrag<(ops node:$ptr), (i8  (load node:$ptr))>;
-def loadi64 : PatFrag<(ops node:$ptr), (i64 (load node:$ptr))>;
-def loadf32 : PatFrag<(ops node:$ptr), (f32 (load node:$ptr))>;
-def loadf64 : PatFrag<(ops node:$ptr), (f64 (load node:$ptr))>;
-def loadf80 : PatFrag<(ops node:$ptr), (f80 (load node:$ptr))>;
+def loadi8   : PatFrag<(ops node:$ptr), (i8  (load node:$ptr))>;
+def loadi64  : PatFrag<(ops node:$ptr), (i64 (load node:$ptr))>;
+def loadf32  : PatFrag<(ops node:$ptr), (f32 (load node:$ptr))>;
+def loadf64  : PatFrag<(ops node:$ptr), (f64 (load node:$ptr))>;
+def loadf80  : PatFrag<(ops node:$ptr), (f80 (load node:$ptr))>;
+def loadf128 : PatFrag<(ops node:$ptr), (f128 (load node:$ptr))>;
 
 def sextloadi16i8  : PatFrag<(ops node:$ptr), (i16 (sextloadi8 node:$ptr))>;
 def sextloadi32i8  : PatFrag<(ops node:$ptr), (i32 (sextloadi8 node:$ptr))>;
 def sextloadi32i16 : PatFrag<(ops node:$ptr), (i32 (sextloadi16 node:$ptr))>;
 def sextloadi64i8  : PatFrag<(ops node:$ptr), (i64 (sextloadi8 node:$ptr))>;
 def sextloadi64i16 : PatFrag<(ops node:$ptr), (i64 (sextloadi16 node:$ptr))>;
 def sextloadi64i32 : PatFrag<(ops node:$ptr), (i64 (sextloadi32 node:$ptr))>;
 
 def zextloadi8i1   : PatFrag<(ops node:$ptr), (i8  (zextloadi1 node:$ptr))>;

lib/Target/X86/X86InstrSSE.td

let Predicates = [HasSSE2] in {
   def : Pat<(v4f32 (bitconvert (v8i16 VR128:$src))), (v4f32 VR128:$src)>;
   def : Pat<(v4f32 (bitconvert (v16i8 VR128:$src))), (v4f32 VR128:$src)>;
   def : Pat<(v4f32 (bitconvert (v2f64 VR128:$src))), (v4f32 VR128:$src)>;
   def : Pat<(v2f64 (bitconvert (v2i64 VR128:$src))), (v2f64 VR128:$src)>;
   def : Pat<(v2f64 (bitconvert (v4i32 VR128:$src))), (v2f64 VR128:$src)>;
   def : Pat<(v2f64 (bitconvert (v8i16 VR128:$src))), (v2f64 VR128:$src)>;
   def : Pat<(v2f64 (bitconvert (v16i8 VR128:$src))), (v2f64 VR128:$src)>;
   def : Pat<(v2f64 (bitconvert (v4f32 VR128:$src))), (v2f64 VR128:$src)>;
+  def : Pat<(f128  (bitconvert (i128  FR128:$src))), (f128  FR128:$src)>;
+  def : Pat<(i128  (bitconvert (f128  FR128:$src))), (i128  FR128:$src)>;
 }
 
 // Bitcasts between 256-bit vector types. Return the original type since
 // no instruction is needed for the conversion
 let Predicates = [HasAVX] in {
   def : Pat<(v4f64  (bitconvert (v8f32 VR256:$src))),  (v4f64 VR256:$src)>;
   def : Pat<(v4f64  (bitconvert (v8i32 VR256:$src))),  (v4f64 VR256:$src)>;
   def : Pat<(v4f64  (bitconvert (v4i64 VR256:$src))),  (v4f64 VR256:$src)>;
let ExeDomain = SSEPackedDouble in {
     defm VGATHERQPD : avx2_gather<0x93, "vgatherqpd", VR256, vx64mem, vy64mem>, VEX_W;
   }
 
   let ExeDomain = SSEPackedSingle in {
     defm VGATHERDPS : avx2_gather<0x92, "vgatherdps", VR256, vx32mem, vy32mem>;
     defm VGATHERQPS : avx2_gather<0x93, "vgatherqps", VR128, vx32mem, vy32mem>;
   }
 }
+
+//===----------------------------------------------------------------------===//
+// Extra selection patterns for FR128, f128, f128mem
+
+def : Pat<(store (f128 FR128:$src), addr:$dst),
+          (MOVAPSmr addr:$dst, (COPY_TO_REGCLASS (f128 FR128:$src), VR128))>;
+// When the data is used as floating point, "movaps" should be faster and shorter
+// than "movdqa". "movaps" is in SSE and movdqa is in SSE2.
+
+def : Pat<(loadf128 addr:$src),
+          (COPY_TO_REGCLASS (MOVAPSrm addr:$src), FR128)>;
+
+// andps is faster and shorter than andpd, andps is SSE and andpd is SSE2
+def : Pat<(X86fand FR128:$src1, (loadf128 addr:$src2)),
+          (COPY_TO_REGCLASS (ANDPSrm (COPY_TO_REGCLASS FR128:$src1, VR128), f128mem:$src2), FR128)>;
+
+def : Pat<(X86fand FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (ANDPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                     (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;
+
+def : Pat<(and FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (ANDPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                     (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;
+
+def : Pat<(X86for FR128:$src1, (loadf128 addr:$src2)),
+          (COPY_TO_REGCLASS (ORPSrm (COPY_TO_REGCLASS FR128:$src1, VR128), f128mem:$src2), FR128)>;
+
+def : Pat<(X86for FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (ORPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                    (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;
+
+def : Pat<(or FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (ORPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                    (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;
+
+def : Pat<(X86fxor FR128:$src1, (loadf128 addr:$src2)),
+          (COPY_TO_REGCLASS (XORPSrm (COPY_TO_REGCLASS FR128:$src1, VR128), f128mem:$src2), FR128)>;
+
+def : Pat<(X86fxor FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (XORPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                     (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;
+
+def : Pat<(xor FR128:$src1, FR128:$src2),
+          (COPY_TO_REGCLASS (XORPSrr (COPY_TO_REGCLASS FR128:$src1, VR128),
+                                     (COPY_TO_REGCLASS FR128:$src2, VR128)), FR128)>;

lib/Target/X86/X86MCInstLower.cpp

if (auto *C = getConstantFromPool(*MI, MI->getOperand(4))) {
         CS << "<";
         for (int i = 0, NumOperands = CV->getNumOperands(); i < NumOperands; ++i) {
           if (i != 0)
             CS << ",";
           Constant *COp = CV->getOperand(i);
           if (isa<UndefValue>(COp)) {
             CS << "u";
           } else if (auto *CI = dyn_cast<ConstantInt>(COp)) {
+            if (CI->getBitWidth() <= 64) {
               CS << CI->getZExtValue();
+            } else {
+              // print multi-word constant as (w0,w1)
+              auto Val = CI->getValue();
+              CS << "(";
+              for (int i = 0, N = Val.getNumWords(); i < N; ++i) {
+                if (i > 0)
+                  CS << ",";
+                CS << Val.getRawData()[i];
+              }
+              CS << ")";
+            }
           } else if (auto *CF = dyn_cast<ConstantFP>(COp)) {
             SmallString<32> Str;
             CF->getValueAPF().toString(Str);
             CS << Str;
           } else {
             CS << "?";
           }
         }

lib/Target/X86/X86RegisterInfo.td

 // A class to support the 'A' assembler constraint: EAX then EDX.
 def GR32_AD : RegisterClass<"X86", [i32], 32, (add EAX, EDX)>;
 
 // Scalar SSE2 floating point registers.
 def FR32 : RegisterClass<"X86", [f32], 32, (sequence "XMM%u", 0, 15)>;
 
 def FR64 : RegisterClass<"X86", [f64], 64, (add FR32)>;
 
+def FR128 : RegisterClass<"X86", [i128, f128], 128, (add FR32)>;
+
 
 // FIXME: This sets up the floating point register files as though they are f64
 // values, though they really are f80 values.  This will cause us to spill
 // values as 64-bit quantities instead of 80-bit quantities, which is much much
 // faster on common hardware.  In reality, this should be controlled by a
 // command line option or something.
 
 def RFP32 : RegisterClass<"X86",[f32], 32, (sequence "FP%u", 0, 6)>;

 def VK2WM   : RegisterClass<"X86", [v2i1],  8,  (sub VK2, K0)> {let Size = 8;}
 def VK4WM   : RegisterClass<"X86", [v4i1],  8,  (sub VK4, K0)> {let Size = 8;}
 def VK8WM   : RegisterClass<"X86", [v8i1],  8,  (sub VK8, K0)> {let Size = 8;}
 def VK16WM  : RegisterClass<"X86", [v16i1], 16, (add VK8WM)>   {let Size = 16;}
 def VK32WM  : RegisterClass<"X86", [v32i1], 32, (add VK16WM)> {let Size = 32;}
 def VK64WM  : RegisterClass<"X86", [v64i1], 64, (add VK32WM)> {let Size = 64;}
 
 // Bound registers
-def BNDR : RegisterClass<"X86", [v2i64], 128, (sequence "BND%u", 0, 3)>;
+def BNDR : RegisterClass<"X86", [v2i64], 128, (sequence "BND%u", 0, 3)>;
 

test/CodeGen/X86/fp128-calling-conv.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; double myD = 1.0;
+@myD = global double 1.000000e+00, align 8
+
+; long double myFP80 = 1.0L;  // x86_64-linux-gnu
+@myFP80 = global x86_fp80 0xK3FFF8000000000000000, align 16
+
+; long double myFP128 = 1.0L;  // x86_64-linux-android
+@myFP128 = global fp128 0xL00000000000000003FFF000000000000, align 16
+
+; The first few parameters are passed in registers and the other are on stack.
+
+define i64 @TestParam_L_0(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d0
+; CHECK-LABEL: TestParam_L_0:
+; CHECK:       movq %rdi, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_1(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d1
+; CHECK-LABEL: TestParam_L_1:
+; CHECK:       movq %rsi, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_2(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d2
+; CHECK-LABEL: TestParam_L_2:
+; CHECK:       movq %rdx, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_3(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d3
+; CHECK-LABEL: TestParam_L_3:
+; CHECK:       movq %rcx, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_4(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d4
+; CHECK-LABEL: TestParam_L_4:
+; CHECK:       movq %r8, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_5(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d5
+; CHECK-LABEL: TestParam_L_5:
+; CHECK:       movq %r9, %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_6(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d6
+; CHECK-LABEL: TestParam_L_6:
+; CHECK:       movq 8(%rsp), %rax
+; CHECK-NEXT:  retq
+}
+
+define i64 @TestParam_L_7(i64 %d0, i64 %d1, i64 %d2, i64 %d3, i64 %d4, i64 %d5, i64 %d6, i64 %d7, i64 %d8, i64 %d9, i64 %d10, i64 %d11, i64 %d12, i64 %d13, i64 %d14, i64 %d15, i64 %d16, i64 %d17, i64 %d18, i64 %d19) {
+entry:
+  ret i64 %d7
+; CHECK-LABEL: TestParam_L_7:
+; CHECK:       movq 16(%rsp), %rax
+; CHECK-NEXT:  retq
+}
+
+define float @TestParam_F_0(float %d0, float %d1, float %d2, float %d3, float %d4, float %d5, float %d6, float %d7, float %d8, float %d9, float %d10, float %d11, float %d12, float %d13, float %d14, float %d15, float %d16, float %d17, float %d18, float %d19) {
+entry:
+  ret float %d0
+; CHECK-LABEL: TestParam_F_0:
+; CHECK-NOT:   mov
+; CHECK:       retq
+}
+
+define float @TestParam_F_1(float %d0, float %d1, float %d2, float %d3, float %d4, float %d5, float %d6, float %d7, float %d8, float %d9, float %d10, float %d11, float %d12, float %d13, float %d14, float %d15, float %d16, float %d17, float %d18, float %d19) {
+entry:
+  ret float %d1
+; CHECK-LABEL: TestParam_F_1:
+; CHECK:       movaps  %xmm1, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define float @TestParam_F_7(float %d0, float %d1, float %d2, float %d3, float %d4, float %d5, float %d6, float %d7, float %d8, float %d9, float %d10, float %d11, float %d12, float %d13, float %d14, float %d15, float %d16, float %d17, float %d18, float %d19) {
+entry:
+  ret float %d7
+; CHECK-LABEL: TestParam_F_7:
+; CHECK:       movaps  %xmm7, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define float @TestParam_F_8(float %d0, float %d1, float %d2, float %d3, float %d4, float %d5, float %d6, float %d7, float %d8, float %d9, float %d10, float %d11, float %d12, float %d13, float %d14, float %d15, float %d16, float %d17, float %d18, float %d19) {
+entry:
+  ret float %d8
+; CHECK-LABEL: TestParam_F_8:
+; CHECK:       movss 8(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define float @TestParam_F_9(float %d0, float %d1, float %d2, float %d3, float %d4, float %d5, float %d6, float %d7, float %d8, float %d9, float %d10, float %d11, float %d12, float %d13, float %d14, float %d15, float %d16, float %d17, float %d18, float %d19) {
+entry:
+  ret float %d9
+; CHECK-LABEL: TestParam_F_9:
+; CHECK:       movss 16(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define double @TestParam_D_0(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10, double %d11, double %d12, double %d13, double %d14, double %d15, double %d16, double %d17, double %d18, double %d19) {
+entry:
+  ret double %d0
+; CHECK-LABEL: TestParam_D_0:
+; CHECK-NOT:   mov
+; CHECK:       retq
+}
+
+define double @TestParam_D_1(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10, double %d11, double %d12, double %d13, double %d14, double %d15, double %d16, double %d17, double %d18, double %d19) {
+entry:
+  ret double %d1
+; CHECK-LABEL: TestParam_D_1:
+; CHECK:       movaps  %xmm1, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define double @TestParam_D_7(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10, double %d11, double %d12, double %d13, double %d14, double %d15, double %d16, double %d17, double %d18, double %d19) {
+entry:
+  ret double %d7
+; CHECK-LABEL: TestParam_D_7:
+; CHECK:       movaps  %xmm7, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define double @TestParam_D_8(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10, double %d11, double %d12, double %d13, double %d14, double %d15, double %d16, double %d17, double %d18, double %d19) {
+entry:
+  ret double %d8
+; CHECK-LABEL: TestParam_D_8:
+; CHECK:       movsd 8(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define double @TestParam_D_9(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10, double %d11, double %d12, double %d13, double %d14, double %d15, double %d16, double %d17, double %d18, double %d19) {
+entry:
+  ret double %d9
+; CHECK-LABEL: TestParam_D_9:
+; CHECK:       movsd 16(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestParam_FP128_0(fp128 %d0, fp128 %d1, fp128 %d2, fp128 %d3, fp128 %d4, fp128 %d5, fp128 %d6, fp128 %d7, fp128 %d8, fp128 %d9, fp128 %d10, fp128 %d11, fp128 %d12, fp128 %d13, fp128 %d14, fp128 %d15, fp128 %d16, fp128 %d17, fp128 %d18, fp128 %d19) {
+entry:
+  ret fp128 %d0
+; CHECK-LABEL: TestParam_FP128_0:
+; CHECK-NOT:   mov
+; CHECK:       retq
+}
+
+define fp128 @TestParam_FP128_1(fp128 %d0, fp128 %d1, fp128 %d2, fp128 %d3, fp128 %d4, fp128 %d5, fp128 %d6, fp128 %d7, fp128 %d8, fp128 %d9, fp128 %d10, fp128 %d11, fp128 %d12, fp128 %d13, fp128 %d14, fp128 %d15, fp128 %d16, fp128 %d17, fp128 %d18, fp128 %d19) {
+entry:
+  ret fp128 %d1
+; CHECK-LABEL: TestParam_FP128_1:
+; CHECK:       movaps  %xmm1, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestParam_FP128_7(fp128 %d0, fp128 %d1, fp128 %d2, fp128 %d3, fp128 %d4, fp128 %d5, fp128 %d6, fp128 %d7, fp128 %d8, fp128 %d9, fp128 %d10, fp128 %d11, fp128 %d12, fp128 %d13, fp128 %d14, fp128 %d15, fp128 %d16, fp128 %d17, fp128 %d18, fp128 %d19) {
+entry:
+  ret fp128 %d7
+; CHECK-LABEL: TestParam_FP128_7:
+; CHECK:       movaps  %xmm7, %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestParam_FP128_8(fp128 %d0, fp128 %d1, fp128 %d2, fp128 %d3, fp128 %d4, fp128 %d5, fp128 %d6, fp128 %d7, fp128 %d8, fp128 %d9, fp128 %d10, fp128 %d11, fp128 %d12, fp128 %d13, fp128 %d14, fp128 %d15, fp128 %d16, fp128 %d17, fp128 %d18, fp128 %d19) {
+entry:
+  ret fp128 %d8
+; CHECK-LABEL: TestParam_FP128_8:
+; CHECK:       movaps 8(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestParam_FP128_9(fp128 %d0, fp128 %d1, fp128 %d2, fp128 %d3, fp128 %d4, fp128 %d5, fp128 %d6, fp128 %d7, fp128 %d8, fp128 %d9, fp128 %d10, fp128 %d11, fp128 %d12, fp128 %d13, fp128 %d14, fp128 %d15, fp128 %d16, fp128 %d17, fp128 %d18, fp128 %d19) {
+entry:
+  ret fp128 %d9
+; CHECK-LABEL: TestParam_FP128_9:
+; CHECK:       movaps 24(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}

test/CodeGen/X86/fp128-cast.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; Check soft floating point conversion function calls.
+
+@vi32 = common global i32 0, align 4
+@vi64 = common global i64 0, align 8
+@vf32 = common global float 0.000000e+00, align 4
+@vf64 = common global double 0.000000e+00, align 8
+@vf128 = common global fp128 0xL00000000000000000000000000000000, align 16
+
+define void @TestCastF32_I32() {
+entry:
+  %0 = load float, float* @vf32, align 4
+  %conv = fptosi float %0 to i32
+  store i32 %conv, i32* @vi32, align 4
+  ret void
+; CHECK-LABEL: TestCastF32_I32:
+; CHECK:       cvttss2si  vf32(%rip), %eax
+; CHECK-NEXT:  movl       %eax, vi32(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF32_I64() {
+entry:
+  %0 = load float, float* @vf32, align 4
+  %conv = fptosi float %0 to i32
+  %conv1 = sext i32 %conv to i64
+  store i64 %conv1, i64* @vi64, align 8
+  ret void
+; CHECK-LABEL: TestCastF32_I64:
+; CHECK:       cvttss2si  vf32(%rip), %eax
+; CHECK-NEXT:  cltq
+; CHECK-NEXT:  movq       %rax, vi64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF32_F64() {
+entry:
+  %0 = load float, float* @vf32, align 4
+  %conv = fpext float %0 to double
+  store double %conv, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: TestCastF32_F64:
+; CHECK:       movss      vf32(%rip), %xmm0
+; CHECK-NEXT:  cvtss2sd   %xmm0, %xmm0
+; CHECK-NEXT:  movsd      %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF32_F128() {
+entry:
+  %0 = load float, float* @vf32, align 4
+  %conv = fpext float %0 to fp128
+  store fp128 %conv, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: TestCastF32_F128:
+; CHECK:       movss      vf32(%rip), %xmm0
+; CHECK-NEXT:  callq      __extendsftf2
+; CHECK-NEXT:  movaps     %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastF64_I32() {
+entry:
+  %0 = load double, double* @vf64, align 8
+  %conv = fptosi double %0 to i32
+  store i32 %conv, i32* @vi32, align 4
+  ret void
+; CHECK-LABEL: TestCastF64_I32:
+; CHECK:       cvttsd2si  vf64(%rip), %eax
+; CHECK-NEXT:  movl       %eax, vi32(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF64_I64() {
+entry:
+  %0 = load double, double* @vf64, align 8
+  %conv = fptosi double %0 to i32
+  %conv1 = sext i32 %conv to i64
+  store i64 %conv1, i64* @vi64, align 8
+  ret void
+; CHECK-LABEL: TestCastF64_I64:
+; CHECK:       cvttsd2si  vf64(%rip), %eax
+; CHECK-NEXT:  cltq
+; CHECK-NEXT:  movq       %rax, vi64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF64_F32() {
+entry:
+  %0 = load double, double* @vf64, align 8
+  %conv = fptrunc double %0 to float
+  store float %conv, float* @vf32, align 4
+  ret void
+; CHECK-LABEL: TestCastF64_F32:
+; CHECK:       movsd      vf64(%rip), %xmm0
+; CHECK-NEXT:  cvtsd2ss   %xmm0, %xmm0
+; CHECK-NEXT:  movss      %xmm0, vf32(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastF64_F128() {
+entry:
+  %0 = load double, double* @vf64, align 8
+  %conv = fpext double %0 to fp128
+  store fp128 %conv, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: TestCastF64_F128:
+; CHECK:       movsd      vf64(%rip), %xmm0
+; CHECK-NEXT:  callq      __extenddftf2
+; CHECK-NEXT:  movapd     %xmm0, vf128(%rip)
+; CHECK:       ret
+}
+
+define void @TestCastF128_I32() {
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %conv = fptosi fp128 %0 to i32
+  store i32 %conv, i32* @vi32, align 4
+  ret void
+; CHECK-LABEL: TestCastF128_I32:
+; CHECK:        movaps     vf128(%rip), %xmm0
+; CHECK-NEXT:   callq      __fixtfsi
+; CHECK-NEXT:   movl       %eax, vi32(%rip)
+; CHECK:        retq
+}
+
+define void @TestCastF128_I64() {
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %conv = fptosi fp128 %0 to i32
+  %conv1 = sext i32 %conv to i64
+  store i64 %conv1, i64* @vi64, align 8
+  ret void
+; CHECK-LABEL: TestCastF128_I64:
+; CHECK:       movaps      vf128(%rip), %xmm0
+; CHECK-NEXT:  callq       __fixtfsi
+; CHECK-NEXT:  cltq
+; CHECK-NEXT:  movq        %rax, vi64(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastF128_F32() {
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %conv = fptrunc fp128 %0 to float
+  store float %conv, float* @vf32, align 4
+  ret void
+; CHECK-LABEL: TestCastF128_F32:
+; CHECK:       movaps      vf128(%rip), %xmm0
+; CHECK-NEXT:  callq       __trunctfsf2
+; CHECK-NEXT:  movss       %xmm0, vf32(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastF128_F64() {
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %conv = fptrunc fp128 %0 to double
+  store double %conv, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: TestCastF128_F64:
+; CHECK:       movapd      vf128(%rip), %xmm0
+; CHECK-NEXT:  callq       __trunctfdf2
+; CHECK-NEXT:  movsd       %xmm0, vf64(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastI32_I64() {
+entry:
+  %0 = load i32, i32* @vi32, align 4
+  %conv = sitofp i32 %0 to float
+  %conv1 = fptosi float %conv to i64
+  store i64 %conv1, i64* @vi64, align 8
+  ret void
+; CHECK-LABEL: TestCastI32_I64:
+; CHECK:       cvtsi2ssl  vi32(%rip), %xmm0
+; CHECK-NEXT:  cvttss2si  %xmm0, %rax
+; CHECK-NEXT:  movq       %rax, vi64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastI32_F32() {
+entry:
+  %0 = load i32, i32* @vi32, align 4
+  %conv = sitofp i32 %0 to float
+  store float %conv, float* @vf32, align 4
+  ret void
+; CHECK-LABEL: TestCastI32_F32:
+; CHECK:       cvtsi2ssl  vi32(%rip), %xmm0
+; CHECK-NEXT:  movss      %xmm0, vf32(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastI32_F64() {
+entry:
+  %0 = load i32, i32* @vi32, align 4
+  %conv = sitofp i32 %0 to double
+  store double %conv, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: TestCastI32_F64:
+; CHECK:       cvtsi2sdl  vi32(%rip), %xmm0
+; CHECK-NEXT:  movsd      %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastI32_F128() {
+entry:
+  %0 = load i32, i32* @vi32, align 4
+  %conv = sitofp i32 %0 to fp128
+  store fp128 %conv, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: TestCastI32_F128:
+; CHECK:       movl       vi32(%rip), %edi
+; CHECK-NEXT:  callq      __floatsitf
+; CHECK-NEXT:  movaps     %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastI64_I32(){
+entry:
+  %0 = load i64, i64* @vi64, align 8
+  %conv = sitofp i64 %0 to float
+  %conv1 = fptosi float %conv to i32
+  store i32 %conv1, i32* @vi32, align 4
+  ret void
+; CHECK-LABEL: TestCastI64_I32:
+; CHECK:       cvtsi2ssq  vi64(%rip), %xmm0
+; CHECK:       cvttss2si  %xmm0, %eax
+; CHECK:       movl       %eax, vi32(%rip)
+; CHECK:       retq
+}
+
+define void @TestCastI64_F32(){
+entry:
+  %0 = load i64, i64* @vi64, align 8
+  %conv = sitofp i64 %0 to float
+  store float %conv, float* @vf32, align 4
+  ret void
+; CHECK-LABEL: TestCastI64_F32:
+; CHECK:       cvtsi2ssq  vi64(%rip), %xmm0
+; CHECK-NEXT:  movss      %xmm0, vf32(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastI64_F64(){
+entry:
+  %0 = load i64, i64* @vi64, align 8
+  %conv = sitofp i64 %0 to double
+  store double %conv, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: TestCastI64_F64:
+; CHECK:       cvtsi2sdq  vi64(%rip), %xmm0
+; CHECK-NEXT:  movsd      %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @TestCastI64_F128(){
+entry:
+  %0 = load i64, i64* @vi64, align 8
+  %conv = sitofp i64 %0 to fp128
+  store fp128 %conv, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: TestCastI64_F128:
+; CHECK:       movq       vi64(%rip), %rdi
+; CHECK-NEXT:  callq      __floatditf
+; CHECK-NEXT:  movaps     %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define i32 @TestConst32(float %v) {
+entry:
+  %cmp = fcmp ogt float %v, 1.000000e+00
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestConst32:
+; CHECK:       ucomiss {{.*}}, %xmm0
+; CHECK-NEXT:  seta %al
+; CHECK:       retq
+}
+
+define i32 @TestConst64(double %v) {
+entry:
+  %cmp = fcmp ogt double %v, 1.000000e+00
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestConst64:
+; CHECK:       ucomisd {{.*}}, %xmm0
+; CHECK-NEXT:  seta %al
+; CHECK:       retq
+}
+
+define i32 @TestConst128(fp128 %v) {
+entry:
+  %cmp = fcmp ogt fp128 %v, 0xL00000000000000003FFF000000000000
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestConst128:
+; CHECK:       movaps {{.*}}, %xmm1
+; CHECK-NEXT:  callq __gttf2
+; CHECK-NEXT:  testl %eax, %eax
+; CHECK-NEXT:  setg %al
+; CHECK:       retq
+}
+
+define i32 @TestBits128(fp128 %ld) {
+entry:
+  %mul = fmul fp128 %ld, %ld
+  %0 = bitcast fp128 %mul to i128
+  %u.sroa.0.4.extract.shift = lshr i128 %0, 32
+  %or5 = or i128 %u.sroa.0.4.extract.shift, %0
+  %or = trunc i128 %or5 to i32
+  %cmp = icmp eq i32 %or, 0
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestBits128:
+; CHECK:       movaps %xmm0, %xmm1
+; CHECK-NEXT:  callq __multf3
+; CHECK-NEXT:  movaps %xmm0, (%rsp)
+; CHECK-NEXT:  movq (%rsp),
+; CHECK-NEXT:  movq %
+; CHECK-NEXT:  shrq $32,
+; CHECK:       orl
+; CHECK-NEXT:  sete %al
+; CHECK-NEXT:  movzbl %al, %eax
+; CHECK:       retq
+}
+
+define fp128 @TestPair128(i64 %a, i64 %b) {
+entry:
+  %conv = zext i64 %a to i128
+  %shl = shl nuw i128 %conv, 64
+  %conv1 = zext i64 %b to i128
+  %or = or i128 %shl, %conv1
+  %add = add i128 %or, 3
+  %0 = bitcast i128 %add to fp128
+  ret fp128 %0
+; CHECK-LABEL: TestPair128:
+; CHECK:       addq $3, %rsi
+; CHECK-NEXT:  movq %rsi, -24(%rsp)
+; CHECK-NEXT:  adcq $0, %rdi
+; CHECK-NEXT:  movq %rdi, -16(%rsp)
+; CHECK-NEXT:  movaps -24(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestTruncCopysign(fp128 %x, i32 %n) {
+entry:
+  %cmp = icmp sgt i32 %n, 50000
+  br i1 %cmp, label %if.then, label %cleanup
+
+if.then:                                          ; preds = %entry
+  %conv = fptrunc fp128 %x to double
+  %call = tail call double @copysign(double 0x7FF0000000000000, double %conv) #2
+  %conv1 = fpext double %call to fp128
+  br label %cleanup
+
+cleanup:                                          ; preds = %entry, %if.then
+  %retval.0 = phi fp128 [ %conv1, %if.then ], [ %x, %entry ]
+  ret fp128 %retval.0
+; CHECK-LABEL: TestTruncCopysign:
+; CHECK:       callq __trunctfdf2
+; CHECK-NEXT:  andpd {{.*}}, %xmm0
+; CHECK-NEXT:  orpd {{.*}}, %xmm0
+; CHECK-NEXT:  callq __extenddftf2
+; CHECK:       retq
+}
+
+declare double @copysign(double, double) #1
+
+attributes #2 = { nounwind readnone }

test/CodeGen/X86/fp128-compare.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+define i32 @TestComp32GT(float %d1, float %d2) {
+entry:
+  %cmp = fcmp ogt float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32GT:
+; CHECK:       ucomiss %xmm1, %xmm0
+; CHECK-NEXT:  seta %al
+}
+
+define i32 @TestComp64GT(double %d1, double %d2) {
+entry:
+  %cmp = fcmp ogt double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64GT:
+; CHECK:       ucomisd %xmm1, %xmm0
+; CHECK-NEXT:  seta %al
+}
+
+define i32 @TestComp128GT(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp ogt fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128GT:
+; CHECK:       callq __gttf2
+; CHECK:       setg  %al
+; CHECK:       retq
+}
+
+define i32 @TestComp32GE(float %d1, float %d2) {
+entry:
+  %cmp = fcmp oge float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32GE:
+; CHECK:       ucomiss %xmm1, %xmm0
+; CHECK-NEXT:  setae %al
+}
+
+define i32 @TestComp64GE(double %d1, double %d2) {
+entry:
+  %cmp = fcmp oge double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64GE:
+; CHECK:       ucomisd %xmm1, %xmm0
+; CHECK-NEXT:  setae %al
+}
+
+define i32 @TestComp128GE(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp oge fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128GE:
+; CHECK:       callq __getf2
+; CHECK:       testl %eax, %eax
+; CHECK:       retq
+}
+
+define i32 @TestComp32LT(float %d1, float %d2) {
+entry:
+  %cmp = fcmp olt float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32LT:
+; CHECK:       ucomiss %xmm0, %xmm1
+; CHECK-NEXT:  seta %al
+}
+
+define i32 @TestComp64LT(double %d1, double %d2) {
+entry:
+  %cmp = fcmp olt double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64LT:
+; CHECK:       ucomisd %xmm0, %xmm1
+; CHECK-NEXT:  seta %al
+}
+
+define i32 @TestComp128LT(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp olt fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128LT:
+; CHECK:       callq __lttf2
+; CHECK-NEXT:  shrl $31, %eax
+; CHECK:       retq
+}
+
+define i32 @TestComp32LE(float %d1, float %d2) {
+entry:
+  %cmp = fcmp ole float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32LE:
+; CHECK:       ucomiss %xmm0, %xmm1
+; CHECK-NEXT:  setae %al
+}
+
+define i32 @TestComp64LE(double %d1, double %d2) {
+entry:
+  %cmp = fcmp ole double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64LE:
+; CHECK:       ucomisd %xmm0, %xmm1
+; CHECK-NEXT:  setae %al
+}
+
+define i32 @TestComp128LE(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp ole fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128LE:
+; CHECK:       callq __letf2
+; CHECK-NEXT:  testl %eax, %eax
+; CHECK:       retq
+}
+
+define i32 @TestComp32EQ(float %d1, float %d2) {
+entry:
+  %cmp = fcmp oeq float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32EQ:
+; CHECK:       cmpeqss %xmm1, %xmm0
+; CHECK-NEXT:  movd %xmm0, %eax
+; CHECK-NEXT:  andl $1, %eax
+; CHECK-NEXT:  retq
+}
+
+define i32 @TestComp64EQ(double %d1, double %d2) {
+entry:
+  %cmp = fcmp oeq double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64EQ:
+; CHECK:       cmpeqsd %xmm1, %xmm0
+; CHECK-NEXT:  movd %xmm0, %rax
+; CHECK-NEXT:  andl $1, %eax
+; CHECK-NEXT:  retq
+}
+
+define i32 @TestComp128EQ(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp oeq fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128EQ:
+; CHECK:       callq __eqtf2
+; CHECK-NEXT:  testl %eax, %eax
+; CHECK:       retq
+}
+
+define i32 @TestComp32NE(float %d1, float %d2) {
+entry:
+  %cmp = fcmp une float %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp32NE:
+; CHECK:       cmpneqss %xmm1, %xmm0
+; CHECK-NEXT:  movd %xmm0, %eax
+; CHECK-NEXT:  andl $1, %eax
+; CHECK-NEXT:  retq
+}
+
+define i32 @TestComp64NE(double %d1, double %d2) {
+entry:
+  %cmp = fcmp une double %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp64NE:
+; CHECK:       cmpneqsd %xmm1, %xmm0
+; CHECK-NEXT:  movd %xmm0, %rax
+; CHECK-NEXT:  andl $1, %eax
+; CHECK-NEXT:  retq
+}
+
+define i32 @TestComp128NE(fp128 %d1, fp128 %d2) {
+entry:
+  %cmp = fcmp une fp128 %d1, %d2
+  %conv = zext i1 %cmp to i32
+  ret i32 %conv
+; CHECK-LABEL: TestComp128NE:
+; CHECK:       callq __netf2
+; CHECK-NEXT:  testl %eax, %eax
+; CHECK:       retq
+}
+
+define fp128 @TestMax(fp128 %x, fp128 %y) {
+entry:
+  %cmp = fcmp ogt fp128 %x, %y
+  %cond = select i1 %cmp, fp128 %x, fp128 %y
+  ret fp128 %cond
+; CHECK-LABEL: TestMax:
+; CHECK: movaps %xmm1
+; CHECK: movaps %xmm0
+; CHECK: callq __gttf2
+; CHECK: movaps {{.*}}, %xmm0
+; CHECK: testl %eax, %eax
+; CHECK: movaps {{.*}}, %xmm0
+; CHECK: retq
+}

test/CodeGen/X86/fp128-i128.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; Check some i128 instruction patterns triggered by fp128.
+
+define void @TestUnionLD1(fp128 %s, i64 %n) #0 {
+entry:
+  %0 = bitcast fp128 %s to i128
+  %1 = zext i64 %n to i128
+  %bf.value = shl nuw i128 %1, 64
+  %bf.shl = and i128 %bf.value, 5192296858534809181786422619668480
+  %bf.clear = and i128 %0, -5192296858534809181786422619668481
+  %bf.set = or i128 %bf.shl, %bf.clear
+  %2 = bitcast i128 %bf.set to fp128
+  tail call void @foo(fp128 %2) #2
+  ret void
+; CHECK-LABEL: TestUnionLD1:
+; CHECK:       movaps %xmm0, -24(%rsp)
+; CHECK-NEXT:  movq -24(%rsp), %rax
+; CHECK-NEXT:  movabsq $281474976710655, %rcx
+; CHECK-NEXT:  andq %rdi, %rcx
+; CHECK-NEXT:  movabsq $-281474976710656, %rdx
+; CHECK-NEXT:  andq -16(%rsp), %rdx
+; CHECK-NEXT:  movq %rax, -40(%rsp)
+; CHECK-NEXT:  orq %rcx, %rdx
+; CHECK-NEXT:  movq %rdx, -32(%rsp)
+; CHECK-NEXT:  movaps -40(%rsp), %xmm0
+; CHECK-NEXT:  jmp foo
+}
+
+define fp128 @TestUnionLD2(fp128 %s) #0 {
+entry:
+  %0 = bitcast fp128 %s to i128
+  %bf.clear = and i128 %0, -18446744073709551616
+  %1 = bitcast i128 %bf.clear to fp128
+  ret fp128 %1
+; CHECK-LABEL: TestUnionLD2:
+; CHECK:       movaps %xmm0, -24(%rsp)
+; CHECK-NEXT:  movq -16(%rsp), %rax
+; CHECK-NEXT:  movq %rax, -32(%rsp)
+; CHECK-NEXT:  movq $0, -40(%rsp)
+; CHECK-NEXT:  movaps -40(%rsp), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define fp128 @TestI128_1(fp128 %x) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %bf.clear = and i128 %0, 170141183460469231731687303715884105727
+  %1 = bitcast i128 %bf.clear to fp128
+  %cmp = fcmp olt fp128 %1, 0xL999999999999999A3FFB999999999999
+  %cond = select i1 %cmp, fp128 0xL00000000000000003FFF000000000000, fp128 0xL00000000000000004000000000000000
+  ret fp128 %cond
+; CHECK-LABEL: TestI128_1:
+; CHECK:       movaps %xmm0,
+; CHECK:       movabsq $9223372036854775807,
+; CHECK:       callq __lttf2
+; CHECK:       testl %eax, %eax
+; CHECK:       movaps {{.*}}, %xmm0
+; CHECK:       retq
+}
+
+define fp128 @TestI128_2(fp128 %x, fp128 %y) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %cmp = icmp sgt i128 %0, -1
+  %cond = select i1 %cmp, fp128 %x, fp128 %y
+  ret fp128 %cond
+; CHECK-LABEL: TestI128_2:
+; CHECK:       movaps %xmm0, -24(%rsp)
+; CHECK-NEXT:  cmpq $0, -16(%rsp)
+; CHECK-NEXT:  jns
+; CHECK:       movaps %xmm1, %xmm0
+; CHECK:       retq
+}
+
+define fp128 @TestI128_3(fp128 %x, i32* nocapture readnone %ex) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %bf.cast = and i128 %0, 170135991163610696904058773219554885632
+  %cmp = icmp eq i128 %bf.cast, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %mul = fmul fp128 %x, 0xL00000000000000004201000000000000
+  %1 = bitcast fp128 %mul to i128
+  %bf.clear4 = and i128 %1, -170135991163610696904058773219554885633
+  %bf.set = or i128 %bf.clear4, 85060207136517546210586590865283612672
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %u.sroa.0.0 = phi i128 [ %bf.set, %if.then ], [ %0, %entry ]
+  %2 = bitcast i128 %u.sroa.0.0 to fp128
+  ret fp128 %2
+; CHECK-LABEL: TestI128_3:
+; CHECK:       movaps %xmm0,
+; CHECK:       movabsq $9223090561878065152,
+; CHECK:       testq
+; CHECK:       callq __multf3
+; CHECK-NEXT:  movaps %xmm0
+; CHECK:       movabsq $-9223090561878065153,
+; CHECK:       movabsq $4611123068473966592,
+; CHECK:       retq
+}
+
+define fp128 @TestI128_4(fp128 %x) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %bf.clear = and i128 %0, -18446744073709551616
+  %1 = bitcast i128 %bf.clear to fp128
+  %add = fadd fp128 %1, %x
+  ret fp128 %add
+; CHECK-LABEL: TestI128_4:
+; CHECK:       movaps %xmm0, %xmm1
+; CHECK-NEXT:  movaps %xmm1, 16(%rsp)
+; CHECK-NEXT:  movq 24(%rsp), %rax
+; CHECK-NEXT:  movq %rax, 8(%rsp)
+; CHECK-NEXT:  movq $0, (%rsp)
+; CHECK-NEXT:  movaps (%rsp), %xmm0
+; CHECK-NEXT:  callq __addtf3
+; CHECK:       retq
+}
+
+define { i64, i64 } @TestShift128(i64 %x.coerce0, i64 %x.coerce1) #0 {
+entry:
+  %.fca.1.insert = insertvalue { i64, i64 } { i64 0, i64 undef }, i64 %x.coerce0, 1
+  ret { i64, i64 } %.fca.1.insert
+; CHECK-LABEL: TestShift128:
+; CHECK:       xorl %eax, %eax
+; CHECK-NEXT:  movq %rdi, %rdx
+; CHECK-NEXT:  retq
+}
+
+@v128 = common global i128 0, align 16
+@v128_2 = common global i128 0, align 16
+
+define void @TestShift128_2() #2 {
+entry:
+  %0 = load i128, i128* @v128, align 16
+  %shl = shl i128 %0, 96
+  %1 = load i128, i128* @v128_2, align 16
+  %or = or i128 %shl, %1
+  store i128 %or, i128* @v128, align 16
+  ret void
+; CHECK-LABEL: TestShift128_2:
+; CHECK:       movq v128(%rip), %rax
+; CHECK-NEXT:  shlq $32, %rax
+; CHECK-NEXT:  movq v128_2(%rip), %rcx
+; CHECK-NEXT:  orq v128_2+8(%rip), %rax
+; CHECK-NEXT:  movq %rcx, v128(%rip)
+; CHECK-NEXT:  movq %rax, v128+8(%rip)
+; CHECK-NEXT:  retq
+}
+
+define fp128 @acosl(fp128 %x) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %bf.clear = and i128 %0, -18446744073709551616
+  %1 = bitcast i128 %bf.clear to fp128
+  %add = fadd fp128 %1, %x
+  ret fp128 %add
+; CHECK-LABEL: acosl:
+; CHECK:       movaps %xmm0, %xmm1
+; CHECK-NEXT:  movaps %xmm1, 16(%rsp)
+; CHECK-NEXT:  movq 24(%rsp), %rax
+; CHECK-NEXT:  movq %rax, 8(%rsp)
+; CHECK-NEXT:  movq $0, (%rsp)
+; CHECK-NEXT:  movaps (%rsp), %xmm0
+; CHECK-NEXT:  callq __addtf3
+; CHECK:       retq
+}
+
+; Compare i128 values and check i128 constants.
+define fp128 @TestComp(fp128 %x, fp128 %y) #0 {
+entry:
+  %0 = bitcast fp128 %x to i128
+  %cmp = icmp sgt i128 %0, -1
+  %cond = select i1 %cmp, fp128 %x, fp128 %y
+  ret fp128 %cond
+; CHECK-LABEL: TestComp:
+; CHECK:       movaps %xmm0, -24(%rsp)
+; CHECK-NEXT:  cmpq $0, -16(%rsp)
+; CHECK-NEXT:  jns
+; CHECK:       movaps %xmm1, %xmm0
+; CHECK:       retq
+}
+
+declare void @foo(fp128) #1
+
+; Test logical operations on fp128 values.
+define fp128 @TestFABS_LD(fp128 %x) #0 {
+entry:
+  %call = tail call fp128 @fabsl(fp128 %x) #2
+  ret fp128 %call
+; CHECK-LABEL: TestFABS_LD
+; CHECK:       andps {{.*}}, %xmm0
+; CHECK-NEXT:  retq
+}
+
+declare fp128 @fabsl(fp128) #1
+
+declare fp128 @copysignl(fp128, fp128) #1
+
+; Test more complicated logical operations generated from copysignl.
+define void @TestCopySign({ fp128, fp128 }* noalias nocapture sret %agg.result, { fp128, fp128 }* byval nocapture readonly align 16 %z) #0 {
+entry:
+  %z.realp = getelementptr inbounds { fp128, fp128 }, { fp128, fp128 }* %z, i64 0, i32 0
+  %z.real = load fp128, fp128* %z.realp, align 16
+  %z.imagp = getelementptr inbounds { fp128, fp128 }, { fp128, fp128 }* %z, i64 0, i32 1
+  %z.imag4 = load fp128, fp128* %z.imagp, align 16
+  %cmp = fcmp ogt fp128 %z.real, %z.imag4
+  %sub = fsub fp128 %z.imag4, %z.imag4
+  br i1 %cmp, label %if.then, label %cleanup
+
+if.then:                                          ; preds = %entry
+  %call = tail call fp128 @fabsl(fp128 %sub) #2
+  br label %cleanup
+
+cleanup:                                          ; preds = %entry, %if.then
+  %z.real.sink = phi fp128 [ %z.real, %if.then ], [ %sub, %entry ]
+  %call.sink = phi fp128 [ %call, %if.then ], [ %z.real, %entry ]
+  %call5 = tail call fp128 @copysignl(fp128 %z.real.sink, fp128 %z.imag4) #2
+  %0 = getelementptr inbounds { fp128, fp128 }, { fp128, fp128 }* %agg.result, i64 0, i32 0
+  %1 = getelementptr inbounds { fp128, fp128 }, { fp128, fp128 }* %agg.result, i64 0, i32 1
+  store fp128 %call.sink, fp128* %0, align 16
+  store fp128 %call5, fp128* %1, align 16
+  ret void
+; CHECK-LABEL: TestCopySign
+; CHECK-NOT:   call
+; CHECK:       callq __subtf3
+; CHECK-NOT:   call
+; CHECK:       callq __gttf2
+; CHECK-NOT:   call
+; CHECK:       andps {{.*}}, %xmm0
+; CHECK:       retq
+}
+
+
+attributes #0 = { nounwind uwtable "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+ssse3,+sse3,+popcnt,+sse,+sse2,+sse4.1,+sse4.2" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+ssse3,+sse3,+popcnt,+sse,+sse2,+sse4.1,+sse4.2" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #2 = { nounwind readnone }

test/CodeGen/X86/fp128-libcalls.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; Check all soft floating point library function calls.
+
+@vf64 = common global double 0.000000e+00, align 8
+@vf128 = common global fp128 0xL00000000000000000000000000000000, align 16
+
+define void @Test64Add(double %d1, double %d2) {
+entry:
+  %add = fadd double %d1, %d2
+  store double %add, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64Add:
+; CHECK:       addsd %xmm1, %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test64_1Add(double %d1) {
+entry:
+  %0 = load double, double* @vf64, align 8
+  %add = fadd double %0, %d1
+  store double %add, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64_1Add:
+; CHECK:       addsd vf64(%rip), %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test128Add(fp128 %d1, fp128 %d2) {
+entry:
+  %add = fadd fp128 %d1, %d2
+  store fp128 %add, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128Add:
+; CHECK:       callq __addtf3
+; CHECK-NEXT:  movaps %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test128_1Add(fp128 %d1){
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %add = fadd fp128 %0, %d1
+  store fp128 %add, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128_1Add:
+; CHECK:       movaps  %xmm0, %xmm1
+; CHECK-NEXT:  movaps  vf128(%rip), %xmm0
+; CHECK-NEXT:  callq   __addtf3
+; CHECK-NEXT:  movaps  %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test64Sub(double %d1, double %d2){
+entry:
+  %sub = fsub double %d1, %d2
+  store double %sub, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64Sub:
+; CHECK:       subsd %xmm1, %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test64_1Sub(double %d1){
+entry:
+  %0 = load double, double* @vf64, align 8
+  %sub = fsub double %0, %d1
+  store double %sub, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64_1Sub:
+; CHECK:       movsd vf64(%rip), %xmm1
+; CHECK-NEXT:  subsd   %xmm0, %xmm1
+; CHECK-NEXT:  movsd %xmm1, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test128Sub(fp128 %d1, fp128 %d2){
+entry:
+  %sub = fsub fp128 %d1, %d2
+  store fp128 %sub, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128Sub:
+; CHECK:       callq __subtf3
+; CHECK-NEXT:  movaps %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test128_1Sub(fp128 %d1){
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %sub = fsub fp128 %0, %d1
+  store fp128 %sub, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128_1Sub:
+; CHECK:       movaps  %xmm0, %xmm1
+; CHECK-NEXT:  movaps  vf128(%rip), %xmm0
+; CHECK-NEXT:  callq   __subtf3
+; CHECK-NEXT:  movaps  %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test64Mul(double %d1, double %d2){
+entry:
+  %mul = fmul double %d1, %d2
+  store double %mul, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64Mul:
+; CHECK:       mulsd %xmm1, %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test64_1Mul(double %d1){
+entry:
+  %0 = load double, double* @vf64, align 8
+  %mul = fmul double %0, %d1
+  store double %mul, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64_1Mul:
+; CHECK:       mulsd vf64(%rip), %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test128Mul(fp128 %d1, fp128 %d2){
+entry:
+  %mul = fmul fp128 %d1, %d2
+  store fp128 %mul, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128Mul:
+; CHECK:       callq __multf3
+; CHECK-NEXT:  movaps %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test128_1Mul(fp128 %d1){
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %mul = fmul fp128 %0, %d1
+  store fp128 %mul, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128_1Mul:
+; CHECK:       movaps  %xmm0, %xmm1
+; CHECK-NEXT:  movaps  vf128(%rip), %xmm0
+; CHECK-NEXT:  callq   __multf3
+; CHECK-NEXT:  movaps  %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test64Div(double %d1, double %d2){
+entry:
+  %div = fdiv double %d1, %d2
+  store double %div, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64Div:
+; CHECK:       divsd %xmm1, %xmm0
+; CHECK-NEXT:  movsd %xmm0, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test64_1Div(double %d1){
+entry:
+  %0 = load double, double* @vf64, align 8
+  %div = fdiv double %0, %d1
+  store double %div, double* @vf64, align 8
+  ret void
+; CHECK-LABEL: Test64_1Div:
+; CHECK:       movsd vf64(%rip), %xmm1
+; CHECK-NEXT:  divsd %xmm0, %xmm1
+; CHECK-NEXT:  movsd %xmm1, vf64(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @Test128Div(fp128 %d1, fp128 %d2){
+entry:
+  %div = fdiv fp128 %d1, %d2
+  store fp128 %div, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128Div:
+; CHECK:       callq __divtf3
+; CHECK-NEXT:  movaps %xmm0, vf128(%rip)
+; CHECK:       retq
+}
+
+define void @Test128_1Div(fp128 %d1){
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %div = fdiv fp128 %0, %d1
+  store fp128 %div, fp128* @vf128, align 16
+  ret void
+; CHECK-LABEL: Test128_1Div:
+; CHECK:       movaps  %xmm0, %xmm1
+; CHECK-NEXT:  movaps  vf128(%rip), %xmm0
+; CHECK-NEXT:  callq   __divtf3
+; CHECK-NEXT:  movaps  %xmm0, vf128(%rip)
+; CHECK:       retq
+}

test/CodeGen/X86/fp128-load.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; double myD = 1.0;
+@my_double = global double 1.000000e+00, align 8
+
+; long double myFP80 = 1.0L;  // x86_64-linux-gnu
+@my_fp80 = global x86_fp80 0xK3FFF8000000000000000, align 16
+
+; long double myFP128 = 1.0L;  // x86_64-linux-android
+@my_fp128 = global fp128 0xL00000000000000003FFF000000000000, align 16
+
+define double @get_double() {
+entry:
+  %0 = load double, double* @my_double, align 8
+  ret double %0
+; CHECK-LABEL: get_double:
+; CHECK:       movsd   my_double(%rip), %xmm0
+; CHECK-NEXT:  retq
+}
+
+define x86_fp80 @get_fp80() {
+entry:
+  %0 = load x86_fp80, x86_fp80* @my_fp80, align 16
+  ret x86_fp80 %0
+; CHECK-LABEL: get_fp80:
+; CHECK:       fldt    my_fp80(%rip)
+; CHECK-NEXT:  retq
+}
+
+define fp128 @get_fp128() {
+entry:
+  %0 = load fp128, fp128* @my_fp128, align 16
+  ret fp128 %0
+; CHECK-LABEL: get_fp128:
+; CHECK:       movaps my_fp128(%rip), %xmm0
+; CHECK-NEXT:  retq
+}
+
[e-mail] = internal unnamed_addr constant [2 x float] [float 0x3FB99999A0000000, float 0x3FC99999A0000000], align 4
+
+define fp128 @TestLoadExtend(fp128 %x, i32 %n) {
+entry:
+  %idxprom = sext i32 %n to i64
+  %arrayidx = getelementptr inbounds [2 x float], [2 x float]* @TestLoadExtend.data, i64 0, i64 %idxprom
+  %0 = load float, float* %arrayidx, align 4
+  %conv = fpext float %0 to fp128
+  ret fp128 %conv
+; CHECK-LABEL: TestLoadExtend:
+; CHECK:       movslq  %edi, %rax
+; CHECK-NEXT:  movss   TestLoadExtend.data(,%rax,4), %xmm0
+; CHECK-NEXT:  callq   __extendsftf2
+; CHECK:       retq
+}
+
+; CHECK-LABEL:  my_double:
+; CHECK-NEXT:  .quad   4607182418800017408
+; CHECK-NEXT:  .size   my_double, 8
+
+; CHECK-LABEL:  my_fp80:
+; CHECK-NEXT:  .quad   -9223372036854775808
+; CHECK-NEXT:  .short  16383
+; CHECK-NEXT:  .zero   6
+; CHECK-NEXT:  .size   my_fp80, 16
+
+; CHECK-LABEL:  my_fp128:
+; CHECK-NEXT:  .quad   0
+; CHECK-NEXT:  .quad   4611404543450677248
+; CHECK-NEXT:  .size   my_fp128, 16

test/CodeGen/X86/fp128-store.ll

+; RUN: llc < %s -O2 -mtriple=x86_64-linux-android | FileCheck %s
+; RUN: llc < %s -O2 -mtriple=x86_64-linux-gnu | FileCheck %s
+
+; double myD = 1.0;
+@myD = global double 1.000000e+00, align 8
+
+; long double myFP80 = 1.0L;  // x86_64-linux-gnu
+@myFP80 = global x86_fp80 0xK3FFF8000000000000000, align 16
+
+; long double myFP128 = 1.0L;  // x86_64-linux-android
+@myFP128 = global fp128 0xL00000000000000003FFF000000000000, align 16
+
+define void @set_D(double %x) {
+entry:
+  store double %x, double* @myD, align 8
+  ret void
+; CHECK-LABEL: set_D:
+; CHECK:       movsd   %xmm0, myD(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @set_FP80(x86_fp80 %x) {
+entry:
+  store x86_fp80 %x, x86_fp80* @myFP80, align 16
+  ret void
+; CHECK-LABEL: set_FP80:
+; CHECK:       fldt    8(%rsp)
+; CHECK-NEXT:  fstpt   myFP80(%rip)
+; CHECK-NEXT:  retq
+}
+
+define void @set_FP128(fp128 %x) {
+entry:
+  store fp128 %x, fp128* @myFP128, align 16
+  ret void
+; CHECK-LABEL: set_FP128:
+; CHECK:       movaps  %xmm0, myFP128(%rip)
+; CHECK-NEXT:  retq
+}

test/CodeGen/X86/soft-fp.ll

-; RUN: llc < %s -march=x86    -mattr=+sse2,+soft-float | FileCheck %s
-; RUN: llc < %s -march=x86-64 -mattr=+sse2,+soft-float | FileCheck %s
+; RUN: llc < %s -march=x86    -mattr=+sse,+soft-float \
+; RUN:     | FileCheck %s --check-prefix=SOFT1 --check-prefix=CHECK
+; RUN: llc < %s -march=x86-64 -mattr=+sse2,+soft-float \
+; RUN:     | FileCheck %s --check-prefix=SOFT2 --check-prefix=CHECK
+; RUN: llc < %s -march=x86-64 -mattr=+sse \
+; RUN:     | FileCheck %s --check-prefix=SSE1 --check-prefix=CHECK
+; RUN: llc < %s -march=x86-64 -mattr=+sse2 \
+; RUN:     | FileCheck %s --check-prefix=SSE2 --check-prefix=CHECK
 ; RUN: llc < %s -mtriple=x86_64-gnux32 -mattr=+sse2,+soft-float | FileCheck %s
 
-; CHECK-NOT: xmm{[0-9]+}
+; CHECK-NOT: xmm{{[0-9]+}}
 
 %struct.__va_list_tag = type { i32, i32, i8*, i8* }
 
 define i32 @t1(i32 %a, ...) nounwind {
 entry:
   %va = alloca [1 x %struct.__va_list_tag], align 8    ; <[1 x %struct.__va_list_tag]*> [#uses=2]
   %va12 = bitcast [1 x %struct.__va_list_tag]* %va to i8*    ; <i8*> [#uses=2]
   call void @llvm.va_start(i8* %va12)
   %va3 = getelementptr [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %va, i64 0, i64 0    ; <%struct.__va_list_tag*> [#uses=1]
   call void @bar(%struct.__va_list_tag* %va3) nounwind
   call void @llvm.va_end(i8* %va12)
   ret i32 undef
+; CHECK-LABEL: t1:
+; CHECK:       ret{{[lq]}}
 }
 
 declare void @llvm.va_start(i8*) nounwind
 
 declare void @bar(%struct.__va_list_tag*)
 
 declare void @llvm.va_end(i8*) nounwind
 
 define float @t2(float %a, float %b) nounwind readnone {
 entry:
   %0 = fadd float %a, %b    ; <float> [#uses=1]
   ret float %0
+; CHECK-LABEL: t2:
+; SOFT1-NOT:   xmm{{[0-9]+}}
+; SOFT2-NOT:   xmm{{[0-9]+}}
+; SSE1:        xmm{{[0-9]+}}
+; SSE2:        xmm{{[0-9]+}}
+; CHECK:       ret{{[lq]}}
+}
+
+; soft-float means no SSE instruction and passing fp128 as pair of i64.
+define fp128 @t3(fp128 %a, fp128 %b) nounwind readnone {
+entry:
+  %0 = fadd fp128 %b, %a
+  ret fp128 %0
+; CHECK-LABEL: t3:
+; SOFT1-NOT:   xmm{{[0-9]+}}
+; SOFT2-NOT:   xmm{{[0-9]+}}
+; SSE1:        xmm{{[0-9]+}}
+; SSE2:        xmm{{[0-9]+}}
+; CHECK:       ret{{[lq]}}
 }

utils/TableGen/X86RecognizableInstr.cpp

OperandType RecognizableInstr::typeFromString(const std::string &s,
   TYPE("u8imm",               TYPE_UIMM8)
   TYPE("i32u8imm",            TYPE_UIMM8)
   TYPE("GR8",                 TYPE_R8)
   TYPE("VR128",               TYPE_XMM128)
   TYPE("VR128X",              TYPE_XMM128)
   TYPE("f128mem",             TYPE_M128)
   TYPE("f256mem",             TYPE_M256)
   TYPE("f512mem",             TYPE_M512)
+  TYPE("FR128",               TYPE_XMM128)
   TYPE("FR64",                TYPE_XMM64)
   TYPE("FR64X",               TYPE_XMM64)
   TYPE("f64mem",              TYPE_M64FP)
   TYPE("sdmem",               TYPE_M64FP)
   TYPE("FR32",                TYPE_XMM32)
   TYPE("FR32X",               TYPE_XMM32)
   TYPE("f32mem",              TYPE_M32FP)
   TYPE("ssmem",               TYPE_M32FP)
RecognizableInstr::immediateEncodingFromString(const std::string &s,
   ENCODING("i64i8imm",        ENCODING_IB)
   ENCODING("i8imm",           ENCODING_IB)
   ENCODING("u8imm",           ENCODING_IB)
   ENCODING("i32u8imm",        ENCODING_IB)
   // This is not a typo.  Instructions like BLENDVPD put
   // register IDs in 8-bit immediates nowadays.
   ENCODING("FR32",            ENCODING_IB)
   ENCODING("FR64",            ENCODING_IB)
+  ENCODING("FR128",           ENCODING_IB)
   ENCODING("VR128",           ENCODING_IB)
   ENCODING("VR256",           ENCODING_IB)
   ENCODING("FR32X",           ENCODING_IB)
   ENCODING("FR64X",           ENCODING_IB)
   ENCODING("VR128X",          ENCODING_IB)
   ENCODING("VR256X",          ENCODING_IB)
   ENCODING("VR512",           ENCODING_IB)
   errs() << "Unhandled immediate encoding " << s << "\n";
   llvm_unreachable("Unhandled immediate encoding");
 }
 
 OperandEncoding
 RecognizableInstr::rmRegisterEncodingFromString(const std::string &s,
                                                 uint8_t OpSize) {
   ENCODING("RST",             ENCODING_FP)
   ENCODING("GR16",            ENCODING_RM)
   ENCODING("GR32",            ENCODING_RM)
   ENCODING("GR32orGR64",      ENCODING_RM)
   ENCODING("GR64",            ENCODING_RM)
   ENCODING("GR8",             ENCODING_RM)
   ENCODING("VR128",           ENCODING_RM)
   ENCODING("VR128X",          ENCODING_RM)
+  ENCODING("FR128",           ENCODING_RM)
   ENCODING("FR64",            ENCODING_RM)
   ENCODING("FR32",            ENCODING_RM)
   ENCODING("FR64X",           ENCODING_RM)
   ENCODING("FR32X",           ENCODING_RM)
   ENCODING("VR64",            ENCODING_RM)
   ENCODING("VR256",           ENCODING_RM)
   ENCODING("VR256X",          ENCODING_RM)
   ENCODING("VR512",           ENCODING_RM)

 RecognizableInstr::roRegisterEncodingFromString(const std::string &s,
                                                 uint8_t OpSize) {
   ENCODING("GR16",            ENCODING_REG)
   ENCODING("GR32",            ENCODING_REG)
   ENCODING("GR32orGR64",      ENCODING_REG)
   ENCODING("GR64",            ENCODING_REG)
   ENCODING("GR8",             ENCODING_REG)
   ENCODING("VR128",           ENCODING_REG)
+  ENCODING("FR128",           ENCODING_REG)
   ENCODING("FR64",            ENCODING_REG)
   ENCODING("FR32",            ENCODING_REG)
   ENCODING("VR64",            ENCODING_REG)
   ENCODING("SEGMENT_REG",     ENCODING_REG)
   ENCODING("DEBUG_REG",       ENCODING_REG)
   ENCODING("CONTROL_REG",     ENCODING_REG)
   ENCODING("VR256",           ENCODING_REG)
   ENCODING("VR256X",          ENCODING_REG)

 }
 
 OperandEncoding
 RecognizableInstr::vvvvRegisterEncodingFromString(const std::string &s,
                                                   uint8_t OpSize) {
   ENCODING("GR32",            ENCODING_VVVV)
   ENCODING("GR64",            ENCODING_VVVV)
   ENCODING("FR32",            ENCODING_VVVV)
+  ENCODING("FR128",           ENCODING_VVVV)
   ENCODING("FR64",            ENCODING_VVVV)
   ENCODING("VR128",           ENCODING_VVVV)
   ENCODING("VR256",           ENCODING_VVVV)
   ENCODING("FR32X",           ENCODING_VVVV)
   ENCODING("FR64X",           ENCODING_VVVV)
   ENCODING("VR128X",          ENCODING_VVVV)
   ENCODING("VR256X",          ENCODING_VVVV)
   ENCODING("VR512",           ENCODING_VVVV)