[X86] AVX512FP16 instructions enabling 1/6

[pengfei]

Summary

1. Enable FP16 type support and basic declarations used by following patches.
2. Enable new instructions VMOVW and VMOVSH.

Ref.: https://software.intel.com/content/www/us/en/develop/download/intel-avx512-fp16-architecture-specification.html

[tschuett]

Could you add a link to a reference?

[pengfei]

In D105263#2852295, @tschuett wrote:

Could you add a link to a reference?

Done. Thanks for reminding.

[pengfei]

Minor fixups on comments.

[craig.topper]

clang/lib/Headers/avx512fp16intrin.h

51

I think this should be _mm256_undefined_ph(void)

61

I think this should be _mm_undefined_ph(void)

65

I think this should be _mm512_undefined_ph(void)

clang/test/CodeGen/X86/avx512fp16-complex.c

1

Can we split _Complex out of this patch? This affects other targets that have _Float16 right? So probably needs a different set of reviewers.

clang/test/Sema/Float16.c

13

It's odd to change behavior and then have a FIXME asking if the old behavior was correct.

llvm/lib/Support/X86TargetParser.cpp

204

I think FeaturesICLServer should still be at the beginning of the list. FeatureAVX512FP16 should be alphabetized with the other AVX512 features. Looks like FeatureAVXNNI was already incorrectly alphabetized.

llvm/lib/Target/X86/X86ISelLowering.cpp

18961

I think this comment should mention movsh now.

19110

movsh

23042

This should probably include EltVT==MVT::f16 for the FP16 override?

llvm/lib/Target/X86/X86InstrFragmentsSIMD.td

410

Add a blank line above this to match the original formatting

996

This should be with fp32imm0 and friends.

[pengfei]

Address review comments. Thanks Craig!

clang/test/CodeGen/X86/avx512fp16-complex.c

1

Sure. Split to D105331. Do you know someone who is familiar with or may be interested in it?

llvm/lib/Target/X86/X86ISelLowering.cpp

23042

Maybe we can only check EltVT == MVT::f16 like this?

[pengfei]

Remove complex handing code.

[pengfei]

Remove complex test.

[pengfei]

Update doc and add X86 to the target list that supports _Float16.

[RKSimon]

clang/lib/Headers/avx512fp16intrin.h

39

I realize its a lot of work, but is there any chance that we could get doxygen comments to document these intrinsics?

llvm/lib/Target/X86/X86Subtarget.h

748

I'm a little worried this might get confused with hasF16C - am I just being over cautious?

[pengfei]

clang/lib/Headers/avx512fp16intrin.h

39

I'm hesitating not only for the work but also the effect. We have about 1K new intrinsics and more than 5K LOC in total in the two header files. Adding the doxygen comments will make the readability worse and increase the difficulty in review. It's also a burden in maintaining the correctness.
Do you think it's feasible to only add a link to intrinsic guide? We have decided to only using link that points intrinsic guide in our product compiler. Using one source is friendly to maintainess. And I think intrinsic guide is also easy to use that doxygen.

llvm/lib/Target/X86/X86Subtarget.h

748

Make sense. How about hasAVX512FP16? I can update the name as a followup patch once these patches merged.

[craig.topper]

llvm/lib/Target/X86/X86Subtarget.h

748

That sounds good to me. We should maybe go back and update some of the others. Especially VNNI since we also have AVXVNNI.

[RKSimon]

clang/lib/Headers/avx512fp16intrin.h

39

I completely understand where you're coming from. What we do lose is the ability for code editors to display the doxygen when using the intrinsic (or mouseover the code). Are there any particular intrinsics that we could do with having comments closer at hand - ones that take rounding modes that its tricky to remember the enum/defines for or implicit load/store alignments come to mind?

I'm not sure about the idea of linking to external docs for specs - do we have a style guide policy on this?

[pengfei]

Add a few doxygen comments.

[pengfei]

clang/lib/Headers/avx512fp16intrin.h

39

Are there any particular intrinsics that we could do with having comments closer at hand

I only found 3 ones from avx512fintrin.h, anyway, I copied here.

ones that take rounding modes that its tricky to remember the enum/defines for or implicit load/store alignments come to mind

Unfortunately, we didn't add doc for them when enabling avx512 intrinsics.

I'm not sure about the idea of linking to external docs for specs - do we have a style guide policy on this?

I was thinking some thing like "See https://llvm.org/LICENSE.txt for license information." in most source files. But I agree doxygen helps for code editors. I didn't think of them simply because I never used them :)
I had some thought about writing a tool to help transporting intrinsic guide info to doxygen, but haven't yet found time to do it.

Anyway, I guess this is not the block issue for this series patches, right?

[craig.topper]

clang/lib/Headers/avx512fp16intrin.h

255

256-bit

clang/lib/Headers/avx512vlfp16intrin.h

75

Why do we use _mm256_set1_epi32 instead of _mm256_set1_epi16?

79

Same question

llvm/include/llvm/IR/RuntimeLibcalls.def

290

Is this tested in this patch?

llvm/lib/Target/X86/X86FastISel.cpp

58

AVX here should maybe be AVX512, but maybe this is pointing out that this name is bad. Would X86ScalarXMMf* be better?

llvm/lib/Target/X86/X86ISelLowering.cpp

13537

Drop curly braces on these.

[pengfei]

1. Address Craig's comments.
2. Add more patterns for i16 lowering.

clang/lib/Headers/avx512vlfp16intrin.h

75

There's no difference in assembly for immediate value. https://godbolt.org/z/sMbrM611d. But the latency of vpbroadcastd is better than vpbroadcastw in Skylake according to intrinsic guide. Here the only effect is consist with _mm256_and_epi32. Do you think it's better to use _mm256_set1_epi16?

llvm/include/llvm/IR/RuntimeLibcalls.def

290

No. I'll move it to the 3rd patch and test it there.

llvm/lib/Target/X86/X86FastISel.cpp

58

Maybe we can use X86ScalarSSEf16, here SSE means SSE registers? Especially GCC community proposing to support FP16 since SSE2.

[pengfei]

Fix another regression caused by last update.

[pengfei]

Rebased.

[LuoYuanke]

clang/lib/CodeGen/TargetInfo.cpp

3413

float -> half?

clang/lib/Headers/avx512fp16intrin.h

293

Just be curious, why not directly use __W?

320

What is may_alias used for?

351

I see in _mm_mask_load_sh(), we create a __m128h with upper bits zero, not sure we also need it in store intrinsic.

420

Why not return __a[0] directly?

clang/test/CodeGen/X86/avx512fp16-abi.c

90

Any false test case that have padding between a and b?

llvm/include/llvm/IR/Intrinsics.td

315

Not sure about the legacy comments, should it be _Float16 now?

llvm/include/llvm/Target/TargetSelectionDAG.td

1055

I notice it is true for other extload. Is it same to "true"?

llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp

341

This is the same to ((byte1 & 0x8) == 0x0)?

[LuoYuanke]

llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp

801

Add comments for map5 and map6?

llvm/lib/Target/X86/X86.td

189

customize?

llvm/lib/Target/X86/X86FastISel.cpp

2291

Also add it in isCMOVPseudo()?

llvm/lib/Target/X86/X86ISelLowering.cpp

1944

Drop the brace.

10523–10524

Need check Subtarget.hasFP16()?

10525

Why handle i16? Isn't it handled by movw?

10716

Why exclude f16? Is there better choice for fp16?

18962–18963

movss/movsh

[LuoYuanke]

llvm/lib/Target/X86/X86InstrAVX512.td

82

indent

3878

Not sure this can be merged to 512 version load/store pattern with muticlass by abstract type info.

4159

Why there is no OptForSize for vmovsh?

4478

Sorry, I forgot what REV stand for. Do you know it?
Is this just encoding difference for register operand compared with VMOVSHZrr? What is it used for?

llvm/lib/Target/X86/X86RegisterInfo.td

570

Given there is only EVEX instructions for fp16, is it necessary to add f16 type to it?

572

Ditto.

[LuoYuanke]

llvm/test/CodeGen/X86/vector-reduce-fmax-nnan.ll

374

Why this test case changes? Shall we add -mattr=+avx512fp16 to run?

llvm/test/CodeGen/X86/vector-reduce-fmin-nnan.ll

373

Ditto.

[pengfei]

Thanks Yuanke.

clang/lib/Headers/avx512fp16intrin.h

293

First, this is a simple mimic of _mm_mask_load_ss.
I think the reason is the intrinsic requests dst[MAX:16] := 0, while the builtin returns with src[MAX:16].
So we need to explicitly clear the upper bits.

320

This is used for preventing type-based alias analysis.
https://gcc.gnu.org/onlinedocs/gcc/Common-Type-Attributes.html#Common-Type-Attributes

"In the context of section 6.5 paragraph 7 of the C99 standard, an lvalue expression dereferencing such a pointer is treated like having a character type."
"This extension exists to support some vector APIs, in which pointers to one vector type are permitted to alias pointers to a different vector type."

351

Both load and store intrinsics only access 16bit memory, the different is the load intrinsic needs to set up the high bits of the XMM register (because we do return a 128 bits result). We don't need to do that for a store.

420

Because __m128i is defined as <2 x i64>. __a[0] is correct only for i64 type.

clang/test/CodeGen/X86/avx512fp16-abi.c

90

This is the one with padding, since _Float16 aligns to 2 bytes while float aligns to 4.

llvm/include/llvm/IR/Intrinsics.td

315

LLVM IR serves for not only one type. __fp16 is still usable in Clang. Besides, OpenCL half type also use half in IR. And maybe we have other FE types too. So I'd like to keep it as is unless we have a better way to cover all other FE types.

llvm/include/llvm/Target/TargetSelectionDAG.td

1055

Good catch. I noticed it too, but forgot to change it.

llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp

341

Yes, but I'm not sure if this is intentional. Maybe it keeps the shape in & X == X?

llvm/lib/Target/X86/X86.td

189

customise seems correct too. Anyway, I can change it.

llvm/lib/Target/X86/X86ISelLowering.cpp

10523–10524

No, f16 is legal here, so it implies the feature.

10525

No, we don't have a movw instruction.

10716

We prefer to using shuffle vector rather than insert_vector_elt here, because we don't have a insert instruction for half type.

llvm/lib/Target/X86/X86InstrAVX512.td

3878

I think it is probably feasible. We may add a codegen only opcode to reuse VMOVDQU instruction defination.
But that may need careful tune, so I think we can do it as a followup.

4159

Good catch. I think we should add it here.

4478

I think REV is short for revert. Which allows a different encoding when operands order are reverted.
Yes. It's used for a different encoding.

llvm/lib/Target/X86/X86RegisterInfo.td

570

I think so. For example, we may use some i16 instructions which may be or may finally turn into AVX2 ones. Adding to it is useful for them since VR128 is subset of VR128X.

llvm/test/CodeGen/X86/vector-reduce-fmax-nnan.ll

374

Because we allowed one combine after X86ISelLowering.cpp:41180 without check the feature.
Although it seems the code here is correct and better, I'll add the check for feature in case any confusing.
We do have the test for avx512fp16 in D105264.

[pengfei]

Address Yuanke's comments.

[craig.topper]

llvm/lib/Target/X86/X86InstrAVX512.td

4478

It is short for "reverse". Meaing the operands are in the reversed order. There are two valid encodings moving from one register to another. This happens because there are separate opcodes for moving register to memory(Store) and moving memory to register(load). The memory operand for both of those opcodes can be a register as well. The assembler and isel always uses the register to register version of the load opcode. The reversed version is only used by the disassembler

There is an exception to that. For VEX encoded AVX/AVX2 instructions, X86MCInstLowering will use an _REV move if it allows a 2 byte VEX prefix instead of a 3 byte VEX prefix. This doesn't apply to any AVX512 instructions though.

[pengfei]

llvm/lib/Target/X86/X86InstrAVX512.td

4478

Thanks Craig for the information.

[LuoYuanke]

llvm/lib/Target/X86/X86InstrAVX512.td

4478

It is short for "reverse". Meaing the operands are in the reversed order. There are two valid encodings moving from one register to another. This happens because there are separate opcodes for moving register to memory(Store) and moving memory to register(load). The memory operand for both of those opcodes can be a register as well. The assembler and isel always uses the register to register version of the load opcode. The reversed version is only used by the disassembler

There is an exception to that. For VEX encoded AVX/AVX2 instructions, X86MCInstLowering will use an _REV move if it allows a 2 byte VEX prefix instead of a 3 byte VEX prefix. This doesn't apply to any AVX512 instructions though.

I understand now. Thanks, Craig and Pengfei.

[pengfei]

Add missing changes from Yuanke's comments.

[pengfei]

Fix ABI incompatibility issue when a structure has three half or a float and half.

[LuoYuanke]

clang/lib/CodeGen/TargetInfo.cpp

3487

For 2 float, return <2xfloat> to be compatible to previous ABI?

[pengfei]

clang/lib/CodeGen/TargetInfo.cpp

3487

It is already handled in line 3456.

[pengfei]

Fix a Lint warning.

[pengfei]

Add override for <3 x half>.

[LuoYuanke]

LGTM, but may wait 1 or 2 days for the comments from others.

[craig.topper]

clang/docs/LanguageExtensions.rst

599 ↗

(On Diff #365070)

Might be worth mentioning that it requires AVX512FP16 here

clang/lib/CodeGen/TargetInfo.cpp

2825

Merge with the previous if?

2956

Merge with the FloatTy if?

[pengfei]

Address review comments. Thanks Craig.

[craig.topper]

LGTM

[pengfei]

llvm/lib/Target/X86/X86InstrAVX512.td

4159

Sorry, I think we should not add OptForSize here.
This predicate is used to force to select blend instead of mov due to performance consideration.
E.g.: https://godbolt.org/z/W4v38K6va

Since we don't have a blendph instruction, I think we can always select it to movsh. Not sure if using pblendw is beneficial.
I'll change it back in next patch.

[vtjnash]

I was tracking back a recent ABI break (also failing now in gcc 12, so maybe this irregularity is intentional), and was concerned that this commit is observed to cause the platform ABI to change depending on the feature flags of the current compilation unit. Prior to this change, f16 was always treated as i16 for the purpose of the calling-convention (e.g. returned in %ax). But after this change, the ABI of the value is now inconsistent between compile units. I made a small change to one of the existing tests to show this. Note how the callq result was in %ax without this mattr flag, and in %xmm0 with this mattr flag added. But the function known as "identity.half" is external, and did not change between those two calls to the llvm.

diff --git a/llvm/test/CodeGen/X86/half.ll b/llvm/test/CodeGen/X86/half.ll
index 46179e7d9113..8c1b8c4b76ff 100644
--- a/llvm/test/CodeGen/X86/half.ll
+++ b/llvm/test/CodeGen/X86/half.ll
@@ -5,6 +5,8 @@
 ; RUN:   | FileCheck %s -check-prefixes=CHECK,CHECK-LIBCALL,BWOFF
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+f16c -fixup-byte-word-insts=1 \
 ; RUN:    | FileCheck %s -check-prefixes=CHECK,BWON,BWON-F16C
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512fp16 -fixup-byte-word-insts=0 \
+; RUN:    | FileCheck %s -check-prefixes=CHECK-CC
 ; RUN: llc < %s -mtriple=i686-unknown-linux-gnu -mattr +sse2 -fixup-byte-word-insts=0  \
 ; RUN:    | FileCheck %s -check-prefixes=CHECK-I686

@@ -163,16 +199,31 @@ define void @test_trunc32(float %in, half* %addr) #0 {
   ret void
 }
 
+declare half @identity.half(half)
+
 define void @test_trunc64(double %in, half* %addr) #0 {
 ; CHECK-LABEL: test_trunc64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    movq %rdi, %rbx
 ; CHECK-NEXT:    callq __truncdfhf2@PLT
+; CHECK-NEXT:    # kill: def $ax killed $ax def $eax
+; CHECK-NEXT:    movl %eax, %edi
+; CHECK-NEXT:    callq identity.half@PLT
 ; CHECK-NEXT:    movw %ax, (%rbx)
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    retq
 ;
+; CHECK-CC-LABEL: test_trunc64:
+; CHECK-CC:       # %bb.0:
+; CHECK-CC-NEXT:    pushq %rbx
+; CHECK-CC-NEXT:    movq %rdi, %rbx
+; CHECK-CC-NEXT:    vcvtsd2sh %xmm0, %xmm0, %xmm0
+; CHECK-CC-NEXT:    callq identity.half@PLT
+; CHECK-CC-NEXT:    vmovsh %xmm0, (%rbx)
+; CHECK-CC-NEXT:    popq %rbx
+; CHECK-CC-NEXT:    retq
+;
 ; CHECK-I686-LABEL: test_trunc64:
 ; CHECK-I686:       # %bb.0:
 ; CHECK-I686-NEXT:    pushl %esi
@@ -181,12 +232,16 @@ define void @test_trunc64(double %in, half* %addr) #0 {
 ; CHECK-I686-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; CHECK-I686-NEXT:    movsd %xmm0, (%esp)
 ; CHECK-I686-NEXT:    calll __truncdfhf2
+; CHECK-I686-NEXT:    # kill: def $ax killed $ax def $eax
+; CHECK-I686-NEXT:    movl %eax, (%esp)
+; CHECK-I686-NEXT:    calll identity.half@PLT
 ; CHECK-I686-NEXT:    movw %ax, (%esi)
 ; CHECK-I686-NEXT:    addl $8, %esp
 ; CHECK-I686-NEXT:    popl %esi
 ; CHECK-I686-NEXT:    retl
   %val16 = fptrunc double %in to half
-  store half %val16, half* %addr
+  %val16b = call half @identity.half(half %val16)
+  store half %val16b, half* %addr
   ret void
 }

Is this intentional? We do already have code to handle the ABI dependency on vector-sizes, and could add this to the list of flags that change the ABI (i.e. we disable it if it will break the ABI), but wanted to confirm first if that was the intent here.

discovered from https://github.com/JuliaLang/julia/issues/44829

[pengfei]

Thanks @vtjnash for the information! Comments on https://github.com/JuliaLang/julia/issues/44829

clang/docs/ClangCommandLineReference.rst

 .. option:: -mavx512cd, -mno-avx512cd
 
 .. option:: -mavx512dq, -mno-avx512dq
 
 .. option:: -mavx512er, -mno-avx512er
 
 .. option:: -mavx512f, -mno-avx512f
 
+.. option:: -mavx512fp16, -mno-avx512fp16
+
 .. option:: -mavx512ifma, -mno-avx512ifma
 
 .. option:: -mavx512pf, -mno-avx512pf
 
 .. option:: -mavx512vbmi, -mno-avx512vbmi
 
 .. option:: -mavx512vbmi2, -mno-avx512vbmi2
 

clang/docs/ReleaseNotes.rst

 CUDA Support in Clang
 ---------------------
 
 - ...
 
 X86 Support in Clang
 --------------------
 
-- ...
+- Support for ``AVX512-FP16`` instructions has been added.
 
 Internal API Changes
 --------------------
 
 These are major API changes that have happened since the 12.0.0 release of
 Clang. If upgrading an external codebase that uses Clang as a library,
 this section should help get you past the largest hurdles of upgrading.
 

clang/include/clang/Basic/Builtins.def

 
 // The second value provided to the macro specifies the type of the function
 // (result value, then each argument) as follows:
 //  v -> void
 //  b -> boolean
 //  c -> char
 //  s -> short
 //  i -> int
-//  h -> half
+//  h -> half (__fp16, OpenCL)
+//  x -> half (_Float16)
 //  f -> float
 //  d -> double
 //  z -> size_t
 //  w -> wchar_t
 //  F -> constant CFString
 //  G -> id
 //  H -> SEL
 //  M -> struct objc_super

clang/include/clang/Basic/BuiltinsX86.def

 
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_q_512, "vV8OiV8OiUc*Uc*", "nV:512:", "avx512vp2intersect")
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_q_256, "vV4OiV4OiUc*Uc*", "nV:256:", "avx512vp2intersect,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_q_128, "vV2OiV2OiUc*Uc*", "nV:128:", "avx512vp2intersect,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_d_512, "vV16iV16iUs*Us*", "nV:512:", "avx512vp2intersect")
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_d_256, "vV8iV8iUc*Uc*", "nV:256:", "avx512vp2intersect,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_vp2intersect_d_128, "vV4iV4iUc*Uc*", "nV:128:", "avx512vp2intersect,avx512vl")
 
+// AVX512 fp16 intrinsics
+TARGET_BUILTIN(__builtin_ia32_loadsh128_mask, "V8xV8x*V8xUc", "nV:128:", "avx512fp16")
+TARGET_BUILTIN(__builtin_ia32_storesh128_mask, "vV8x*V8xUc", "nV:128:", "avx512fp16")
+
 // generic select intrinsics
 TARGET_BUILTIN(__builtin_ia32_selectb_128, "V16cUsV16cV16c", "ncV:128:", "avx512bw,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectb_256, "V32cUiV32cV32c", "ncV:256:", "avx512bw,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectb_512, "V64cUOiV64cV64c", "ncV:512:", "avx512bw")
 TARGET_BUILTIN(__builtin_ia32_selectw_128, "V8sUcV8sV8s", "ncV:128:", "avx512bw,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectw_256, "V16sUsV16sV16s", "ncV:256:", "avx512bw,avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectw_512, "V32sUiV32sV32s", "ncV:512:", "avx512bw")
 TARGET_BUILTIN(__builtin_ia32_selectd_128, "V4iUcV4iV4i", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectd_256, "V8iUcV8iV8i", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectd_512, "V16iUsV16iV16i", "ncV:512:", "avx512f")
+TARGET_BUILTIN(__builtin_ia32_selectph_128, "V8xUcV8xV8x", "ncV:128:", "avx512fp16,avx512vl")
+TARGET_BUILTIN(__builtin_ia32_selectph_256, "V16xUsV16xV16x", "ncV:256:", "avx512fp16,avx512vl")
+TARGET_BUILTIN(__builtin_ia32_selectph_512, "V32xUiV32xV32x", "ncV:512:", "avx512fp16")
 TARGET_BUILTIN(__builtin_ia32_selectq_128, "V2OiUcV2OiV2Oi", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectq_256, "V4OiUcV4OiV4Oi", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectq_512, "V8OiUcV8OiV8Oi", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_selectps_128, "V4fUcV4fV4f", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectps_256, "V8fUcV8fV8f", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectps_512, "V16fUsV16fV16f", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_selectpd_128, "V2dUcV2dV2d", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectpd_256, "V4dUcV4dV4d", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_selectpd_512, "V8dUcV8dV8d", "ncV:512:", "avx512f")
+TARGET_BUILTIN(__builtin_ia32_selectsh_128, "V8xUcV8xV8x", "ncV:128:", "avx512fp16")
 TARGET_BUILTIN(__builtin_ia32_selectss_128, "V4fUcV4fV4f", "ncV:128:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_selectsd_128, "V2dUcV2dV2d", "ncV:128:", "avx512f")
 
 // generic reduction intrinsics
 TARGET_BUILTIN(__builtin_ia32_reduce_add_d512, "iV16i", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_reduce_add_q512, "OiV8Oi", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_reduce_and_d512, "iV16i", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_reduce_and_q512, "OiV8Oi", "ncV:512:", "avx512f")

clang/include/clang/Basic/IdentifierTable.h

 //===- IdentifierTable.h - Hash table for identifier lookup -----*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 /// A simple pair of identifier info and location.
 using IdentifierLocPair = std::pair<IdentifierInfo *, SourceLocation>;
 
 /// IdentifierInfo and other related classes are aligned to
 /// 8 bytes so that DeclarationName can use the lower 3 bits
 /// of a pointer to one of these classes.
 enum { IdentifierInfoAlignment = 8 };
 
-static constexpr int ObjCOrBuiltinIDBits = 15;
+static constexpr int ObjCOrBuiltinIDBits = 16;
 
 /// One of these records is kept for each identifier that
 /// is lexed.  This contains information about whether the token was \#define'd,
 /// is a language keyword, or if it is a front-end token of some sort (e.g. a
 /// variable or function name).  The preprocessor keeps this information in a
 /// set, and all tok::identifier tokens have a pointer to one of these.
 /// It is aligned to 8 bytes because DeclarationName needs the lower 3 bits.
 class alignas(IdentifierInfoAlignment) IdentifierInfo {

clang/include/clang/Driver/Options.td

 def mavx512bw : Flag<["-"], "mavx512bw">, Group<m_x86_Features_Group>;
 def mno_avx512bw : Flag<["-"], "mno-avx512bw">, Group<m_x86_Features_Group>;
 def mavx512cd : Flag<["-"], "mavx512cd">, Group<m_x86_Features_Group>;
 def mno_avx512cd : Flag<["-"], "mno-avx512cd">, Group<m_x86_Features_Group>;
 def mavx512dq : Flag<["-"], "mavx512dq">, Group<m_x86_Features_Group>;
 def mno_avx512dq : Flag<["-"], "mno-avx512dq">, Group<m_x86_Features_Group>;
 def mavx512er : Flag<["-"], "mavx512er">, Group<m_x86_Features_Group>;
 def mno_avx512er : Flag<["-"], "mno-avx512er">, Group<m_x86_Features_Group>;
+def mavx512fp16 : Flag<["-"], "mavx512fp16">, Group<m_x86_Features_Group>;
+def mno_avx512fp16 : Flag<["-"], "mno-avx512fp16">, Group<m_x86_Features_Group>;
 def mavx512ifma : Flag<["-"], "mavx512ifma">, Group<m_x86_Features_Group>;
 def mno_avx512ifma : Flag<["-"], "mno-avx512ifma">, Group<m_x86_Features_Group>;
 def mavx512pf : Flag<["-"], "mavx512pf">, Group<m_x86_Features_Group>;
 def mno_avx512pf : Flag<["-"], "mno-avx512pf">, Group<m_x86_Features_Group>;
 def mavx512vbmi : Flag<["-"], "mavx512vbmi">, Group<m_x86_Features_Group>;
 def mno_avx512vbmi : Flag<["-"], "mno-avx512vbmi">, Group<m_x86_Features_Group>;
 def mavx512vbmi2 : Flag<["-"], "mavx512vbmi2">, Group<m_x86_Features_Group>;
 def mno_avx512vbmi2 : Flag<["-"], "mno-avx512vbmi2">, Group<m_x86_Features_Group>;

clang/lib/AST/ASTContext.cpp

 //===- ASTContext.cpp - Context to hold long-lived AST nodes --------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
while (!Done) {
     }
   }
 
   QualType Type;
 
   // Read the base type.
   switch (*Str++) {
   default: llvm_unreachable("Unknown builtin type letter!");
+  case 'x':
+    assert(HowLong == 0 && !Signed && !Unsigned &&
+           "Bad modifiers used with 'x'!");
+    Type = Context.Float16Ty;
+    break;
   case 'y':
     assert(HowLong == 0 && !Signed && !Unsigned &&
            "Bad modifiers used with 'y'!");
     Type = Context.BFloat16Ty;
     break;
   case 'v':
     assert(HowLong == 0 && !Signed && !Unsigned &&
            "Bad modifiers used with 'v'!");

clang/lib/Basic/Targets/X86.h

 //===--- X86.h - Declare X86 target feature support -------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
   bool HasADX = false;
   bool HasTBM = false;
   bool HasLWP = false;
   bool HasFMA = false;
   bool HasF16C = false;
   bool HasAVX512CD = false;
   bool HasAVX512VPOPCNTDQ = false;
   bool HasAVX512VNNI = false;
+  bool HasAVX512FP16 = false;
   bool HasAVX512BF16 = false;
   bool HasAVX512ER = false;
   bool HasAVX512PF = false;
   bool HasAVX512DQ = false;
   bool HasAVX512BITALG = false;
   bool HasAVX512BW = false;
   bool HasAVX512VL = false;
   bool HasAVX512VBMI = false;

clang/lib/Basic/Targets/X86.cpp

 //===--- X86.cpp - Implement X86 target feature support -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
for (const auto &Feature : Features) {
     } else if (Feature == "+avx512vpopcntdq") {
       HasAVX512VPOPCNTDQ = true;
     } else if (Feature == "+avx512vnni") {
       HasAVX512VNNI = true;
     } else if (Feature == "+avx512bf16") {
       HasAVX512BF16 = true;
     } else if (Feature == "+avx512er") {
       HasAVX512ER = true;
+    } else if (Feature == "+avx512fp16") {
+      HasAVX512FP16 = true;
+      HasFloat16 = true;
     } else if (Feature == "+avx512pf") {
       HasAVX512PF = true;
     } else if (Feature == "+avx512dq") {
       HasAVX512DQ = true;
     } else if (Feature == "+avx512bitalg") {
       HasAVX512BITALG = true;
     } else if (Feature == "+avx512bw") {
       HasAVX512BW = true;
void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
   if (HasAVX512VPOPCNTDQ)
     Builder.defineMacro("__AVX512VPOPCNTDQ__");
   if (HasAVX512VNNI)
     Builder.defineMacro("__AVX512VNNI__");
   if (HasAVX512BF16)
     Builder.defineMacro("__AVX512BF16__");
   if (HasAVX512ER)
     Builder.defineMacro("__AVX512ER__");
+  if (HasAVX512FP16)
+    Builder.defineMacro("__AVX512FP16__");
   if (HasAVX512PF)
     Builder.defineMacro("__AVX512PF__");
   if (HasAVX512DQ)
     Builder.defineMacro("__AVX512DQ__");
   if (HasAVX512BITALG)
     Builder.defineMacro("__AVX512BITALG__");
   if (HasAVX512BW)
     Builder.defineMacro("__AVX512BW__");
return llvm::StringSwitch<bool>(Name)
       .Case("avx", true)
       .Case("avx2", true)
       .Case("avx512f", true)
       .Case("avx512cd", true)
       .Case("avx512vpopcntdq", true)
       .Case("avx512vnni", true)
       .Case("avx512bf16", true)
       .Case("avx512er", true)
+      .Case("avx512fp16", true)
       .Case("avx512pf", true)
       .Case("avx512dq", true)
       .Case("avx512bitalg", true)
       .Case("avx512bw", true)
       .Case("avx512vl", true)
       .Case("avx512vbmi", true)
       .Case("avx512vbmi2", true)
       .Case("avx512ifma", true)
return llvm::StringSwitch<bool>(Feature)
       .Case("avx", SSELevel >= AVX)
       .Case("avx2", SSELevel >= AVX2)
       .Case("avx512f", SSELevel >= AVX512F)
       .Case("avx512cd", HasAVX512CD)
       .Case("avx512vpopcntdq", HasAVX512VPOPCNTDQ)
       .Case("avx512vnni", HasAVX512VNNI)
       .Case("avx512bf16", HasAVX512BF16)
       .Case("avx512er", HasAVX512ER)
+      .Case("avx512fp16", HasAVX512FP16)
       .Case("avx512pf", HasAVX512PF)
       .Case("avx512dq", HasAVX512DQ)
       .Case("avx512bitalg", HasAVX512BITALG)
       .Case("avx512bw", HasAVX512BW)
       .Case("avx512vl", HasAVX512VL)
       .Case("avx512vbmi", HasAVX512VBMI)
       .Case("avx512vbmi2", HasAVX512VBMI2)
       .Case("avx512ifma", HasAVX512IFMA)

clang/lib/CodeGen/CGBuiltin.cpp

 //===---- CGBuiltin.cpp - Emit LLVM Code for builtins ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
#undef INTRINSIC_X86_XSAVE_ID
   case X86::BI__builtin_ia32_storedqudi512_mask:
   case X86::BI__builtin_ia32_storedqusi512_mask:
   case X86::BI__builtin_ia32_storedquhi512_mask:
   case X86::BI__builtin_ia32_storedquqi512_mask:
   case X86::BI__builtin_ia32_storeupd512_mask:
   case X86::BI__builtin_ia32_storeups512_mask:
     return EmitX86MaskedStore(*this, Ops, Align(1));
 
+  case X86::BI__builtin_ia32_storesh128_mask:
   case X86::BI__builtin_ia32_storess128_mask:
   case X86::BI__builtin_ia32_storesd128_mask:
     return EmitX86MaskedStore(*this, Ops, Align(1));
 
   case X86::BI__builtin_ia32_vpopcntb_128:
   case X86::BI__builtin_ia32_vpopcntd_128:
   case X86::BI__builtin_ia32_vpopcntq_128:
   case X86::BI__builtin_ia32_vpopcntw_128:
#undef INTRINSIC_X86_XSAVE_ID
   case X86::BI__builtin_ia32_loaddqusi128_mask:
   case X86::BI__builtin_ia32_loaddqusi256_mask:
   case X86::BI__builtin_ia32_loaddqusi512_mask:
   case X86::BI__builtin_ia32_loaddqudi128_mask:
   case X86::BI__builtin_ia32_loaddqudi256_mask:
   case X86::BI__builtin_ia32_loaddqudi512_mask:
     return EmitX86MaskedLoad(*this, Ops, Align(1));
 
+  case X86::BI__builtin_ia32_loadsh128_mask:
   case X86::BI__builtin_ia32_loadss128_mask:
   case X86::BI__builtin_ia32_loadsd128_mask:
     return EmitX86MaskedLoad(*this, Ops, Align(1));
 
   case X86::BI__builtin_ia32_loadaps128_mask:
   case X86::BI__builtin_ia32_loadaps256_mask:
   case X86::BI__builtin_ia32_loadaps512_mask:
   case X86::BI__builtin_ia32_loadapd128_mask:

   case X86::BI__builtin_ia32_selectw_256:
   case X86::BI__builtin_ia32_selectw_512:
   case X86::BI__builtin_ia32_selectd_128:
   case X86::BI__builtin_ia32_selectd_256:
   case X86::BI__builtin_ia32_selectd_512:
   case X86::BI__builtin_ia32_selectq_128:
   case X86::BI__builtin_ia32_selectq_256:
   case X86::BI__builtin_ia32_selectq_512:
+  case X86::BI__builtin_ia32_selectph_128:
+  case X86::BI__builtin_ia32_selectph_256:
+  case X86::BI__builtin_ia32_selectph_512:
   case X86::BI__builtin_ia32_selectps_128:
   case X86::BI__builtin_ia32_selectps_256:
   case X86::BI__builtin_ia32_selectps_512:
   case X86::BI__builtin_ia32_selectpd_128:
   case X86::BI__builtin_ia32_selectpd_256:
   case X86::BI__builtin_ia32_selectpd_512:
     return EmitX86Select(*this, Ops[0], Ops[1], Ops[2]);
+  case X86::BI__builtin_ia32_selectsh_128:
   case X86::BI__builtin_ia32_selectss_128:
   case X86::BI__builtin_ia32_selectsd_128: {
     Value *A = Builder.CreateExtractElement(Ops[1], (uint64_t)0);
     Value *B = Builder.CreateExtractElement(Ops[2], (uint64_t)0);
     A = EmitX86ScalarSelect(*this, Ops[0], A, B);
     return Builder.CreateInsertElement(Ops[1], A, (uint64_t)0);
   }
   case X86::BI__builtin_ia32_cmpb128_mask:

clang/lib/CodeGen/TargetInfo.cpp

 //===---- TargetInfo.cpp - Encapsulate target details -----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (isAggregateTypeForABI(RetTy)) {
     // If specified, structs and unions are always indirect.
     if (!IsRetSmallStructInRegABI && !RetTy->isAnyComplexType())
       return getIndirectReturnResult(RetTy, State);
 
     // Ignore empty structs/unions.
     if (isEmptyRecord(getContext(), RetTy, true))
       return ABIArgInfo::getIgnore();
 
+    // Return complex of _Float16 as <2 x half> so the backend will use xmm0.
+    if (const ComplexType *CT = RetTy->getAs<ComplexType>()) {
+      QualType ET = getContext().getCanonicalType(CT->getElementType());
+      if (ET->isFloat16Type())
+        return ABIArgInfo::getDirect(llvm::FixedVectorType::get(
+            llvm::Type::getHalfTy(getVMContext()), 2));
+    }
+
     // Small structures which are register sized are generally returned
     // in a register.
     if (shouldReturnTypeInRegister(RetTy, getContext())) {
       uint64_t Size = getContext().getTypeSize(RetTy);
 
       // As a special-case, if the struct is a "single-element" struct, and
       // the field is of type "float" or "double", return it in a
       // floating-point register. (MSVC does not apply this special case.)
if (k == BuiltinType::Void) {
       Current = NoClass;
     } else if (k == BuiltinType::Int128 || k == BuiltinType::UInt128) {
       Lo = Integer;
       Hi = Integer;
     } else if (k >= BuiltinType::Bool && k <= BuiltinType::LongLong) {
       Current = Integer;
     } else if (k == BuiltinType::Float || k == BuiltinType::Double) {
       Current = SSE;
+    } else if (k == BuiltinType::Float16) {
+      Current = SSE;
     } else if (k == BuiltinType::LongDouble) {
       const llvm::fltSemantics *LDF = &getTarget().getLongDoubleFormat();
       if (LDF == &llvm::APFloat::IEEEquad()) {
         Lo = SSE;
         Hi = SSEUp;
       } else if (LDF == &llvm::APFloat::x87DoubleExtended()) {
         Lo = X87;
         Hi = X87Up;
if (const ComplexType *CT = Ty->getAs<ComplexType>()) {
     QualType ET = getContext().getCanonicalType(CT->getElementType());
 
     uint64_t Size = getContext().getTypeSize(Ty);
     if (ET->isIntegralOrEnumerationType()) {
       if (Size <= 64)
         Current = Integer;
       else if (Size <= 128)
         Lo = Hi = Integer;
+    } else if (ET->isFloat16Type()) {
+      Current = SSE;
     } else if (ET == getContext().FloatTy) {
       Current = SSE;
     } else if (ET == getContext().DoubleTy) {
       Lo = Hi = SSE;
     } else if (ET == getContext().LongDoubleTy) {
       const llvm::fltSemantics *LDF = &getTarget().getLongDoubleFormat();
       if (LDF == &llvm::APFloat::IEEEquad())
         Current = Memory;
if (llvm::ArrayType *ATy = dyn_cast<llvm::ArrayType>(IRType)) {
     unsigned EltSize = TD.getTypeAllocSize(EltTy);
     IROffset -= IROffset/EltSize*EltSize;
     return ContainsFloatAtOffset(EltTy, IROffset, TD);
   }
 
   return false;
 }
 
+/// ContainsHalfAtOffset - Return true if the specified LLVM IR type has a
+/// half member at the specified offset.  For example, {int,{half}} has a
+/// float at offset 4.  It is conservatively correct for this routine to return
+/// false.
+/// FIXME: Merge with ContainsFloatAtOffset
+static bool ContainsHalfAtOffset(llvm::Type *IRType, unsigned IROffset,
+                                 const llvm::DataLayout &TD) {
+  // Base case if we find a float.
+  if (IROffset == 0 && IRType->isHalfTy())
+    return true;
+
+  // If this is a struct, recurse into the field at the specified offset.
+  if (llvm::StructType *STy = dyn_cast<llvm::StructType>(IRType)) {
+    const llvm::StructLayout *SL = TD.getStructLayout(STy);
+    unsigned Elt = SL->getElementContainingOffset(IROffset);
+    IROffset -= SL->getElementOffset(Elt);
+    return ContainsHalfAtOffset(STy->getElementType(Elt), IROffset, TD);
+  }
+
+  // If this is an array, recurse into the field at the specified offset.
+  if (llvm::ArrayType *ATy = dyn_cast<llvm::ArrayType>(IRType)) {
+    llvm::Type *EltTy = ATy->getElementType();
+    unsigned EltSize = TD.getTypeAllocSize(EltTy);
+    IROffset -= IROffset / EltSize * EltSize;
+    return ContainsHalfAtOffset(EltTy, IROffset, TD);
+  }
+
+  return false;
+}
 
 /// GetSSETypeAtOffset - Return a type that will be passed by the backend in the
 /// low 8 bytes of an XMM register, corresponding to the SSE class.
 llvm::Type *X86_64ABIInfo::
 GetSSETypeAtOffset(llvm::Type *IRType, unsigned IROffset,
                    QualType SourceTy, unsigned SourceOffset) const {
-  // The only three choices we have are either double, <2 x float>, or float. We
-  // pass as float if the last 4 bytes is just padding.  This happens for
-  // structs that contain 3 floats.
-  if (BitsContainNoUserData(SourceTy, SourceOffset*8+32,
-                            SourceOffset*8+64, getContext()))
+  // If only 16 bits are used, pass in half.
+  if (BitsContainNoUserData(SourceTy, SourceOffset * 8 + 16,
+                            SourceOffset * 8 + 64, getContext()))
+    return llvm::Type::getHalfTy(getVMContext());
+
+  // If only 32 bits are used, we have two choices. Single float or two halfs.
+  if (BitsContainNoUserData(SourceTy, SourceOffset * 8 + 32,
+                            SourceOffset * 8 + 64, getContext())) {
+    if (ContainsHalfAtOffset(IRType, IROffset, getDataLayout()) &&
+        ContainsHalfAtOffset(IRType, IROffset + 2, getDataLayout()))
+      return llvm::FixedVectorType::get(llvm::Type::getHalfTy(getVMContext()),
+                                        2);
     return llvm::Type::getFloatTy(getVMContext());
+  }
+
+  // If 48 bits are used, we pass as <3 x half>.
+  if (BitsContainNoUserData(SourceTy, SourceOffset * 8 + 48,
+                            SourceOffset * 8 + 64, getContext())) {
+    return llvm::FixedVectorType::get(llvm::Type::getHalfTy(getVMContext()), 3);
+  }
 
   // We want to pass as <2 x float> if the LLVM IR type contains a float at
   // offset+0 and offset+4.  Walk the LLVM IR type to find out if this is the
   // case.
   if (ContainsFloatAtOffset(IRType, IROffset, getDataLayout()) &&
       ContainsFloatAtOffset(IRType, IROffset+4, getDataLayout()))
     return llvm::FixedVectorType::get(llvm::Type::getFloatTy(getVMContext()),
                                       2);
 
+  // We want to pass as <4 x half> if the LLVM IR type contains a half at
+  // offset+0, +2, +4, +6.  Walk the LLVM IR type to find out if this is the
+  // case.
+  if (ContainsHalfAtOffset(IRType, IROffset, getDataLayout()) &&
+      ContainsHalfAtOffset(IRType, IROffset + 2, getDataLayout()) &&
+      ContainsHalfAtOffset(IRType, IROffset + 4, getDataLayout()) &&
+      ContainsHalfAtOffset(IRType, IROffset + 6, getDataLayout()))
+    return llvm::FixedVectorType::get(llvm::Type::getHalfTy(getVMContext()), 4);
+
+  // TODO: What about mixes of float and half?
+
   return llvm::Type::getDoubleTy(getVMContext());
 }
 
 
 /// GetINTEGERTypeAtOffset - The ABI specifies that a value should be passed in
 /// an 8-byte GPR.  This means that we either have a scalar or we are talking
 /// about the high or low part of an up-to-16-byte struct.  This routine picks
 /// the best LLVM IR type to represent this, which may be i64 or may be anything
 /// else that the backend will pass in a GPR that works better (e.g. i8, %foo*,
 /// etc).

clang/lib/Headers/CMakeLists.txt

set(files
   avx512bwintrin.h
   avx512bitalgintrin.h
   avx512vlbitalgintrin.h
   avx512cdintrin.h
   avx512vpopcntdqintrin.h
   avx512dqintrin.h
   avx512erintrin.h
   avx512fintrin.h
+  avx512fp16intrin.h
   avx512ifmaintrin.h
   avx512ifmavlintrin.h
   avx512pfintrin.h
   avx512vbmiintrin.h
   avx512vbmivlintrin.h
   avx512vbmi2intrin.h
   avx512vlvbmi2intrin.h
   avx512vlbf16intrin.h
   avx512vlbwintrin.h
   avx512vlcdintrin.h
   avx512vldqintrin.h
+  avx512vlfp16intrin.h
   avx512vlintrin.h
   avx512vp2intersectintrin.h
   avx512vlvp2intersectintrin.h
   avx512vpopcntdqvlintrin.h
   avx512vnniintrin.h
   avx512vlvnniintrin.h
   avxintrin.h
   avxvnniintrin.h

clang/lib/Headers/avx512fp16intrin.h

+/*===----------- avx512fp16intrin.h - AVX512-FP16 intrinsics ---------------===
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===-----------------------------------------------------------------------===
+ */
+#ifndef __IMMINTRIN_H
+#error "Never use <avx512fp16intrin.h> directly; include <immintrin.h> instead."
+#endif
+
+#ifndef __AVX512FP16INTRIN_H
+#define __AVX512FP16INTRIN_H
+
+/* Define the default attributes for the functions in this file. */
+typedef _Float16 __v32hf __attribute__((__vector_size__(64), __aligned__(64)));
+typedef _Float16 __m512h __attribute__((__vector_size__(64), __aligned__(64)));
+typedef _Float16 __m512h_u __attribute__((__vector_size__(64), __aligned__(1)));
+typedef _Float16 __v8hf __attribute__((__vector_size__(16), __aligned__(16)));
+typedef _Float16 __m128h __attribute__((__vector_size__(16), __aligned__(16)));
+typedef _Float16 __m128h_u __attribute__((__vector_size__(16), __aligned__(1)));
+typedef _Float16 __v16hf __attribute__((__vector_size__(32), __aligned__(32)));
+typedef _Float16 __m256h __attribute__((__vector_size__(32), __aligned__(32)));
+typedef _Float16 __m256h_u __attribute__((__vector_size__(32), __aligned__(1)));
+
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS512                                                  \
+  __attribute__((__always_inline__, __nodebug__, __target__("avx512fp16"),     \
+                 __min_vector_width__(512)))
+#define __DEFAULT_FN_ATTRS256                                                  \
+  __attribute__((__always_inline__, __nodebug__, __target__("avx512fp16"),     \
+                 __min_vector_width__(256)))
+#define __DEFAULT_FN_ATTRS128                                                  \
+  __attribute__((__always_inline__, __nodebug__, __target__("avx512fp16"),     \
+                 __min_vector_width__(128)))
+
+static __inline__ _Float16 __DEFAULT_FN_ATTRS512 _mm512_cvtsh_h(__m512h __a) {
+  return __a[0];
+}
+
+static __inline __m128h __DEFAULT_FN_ATTRS128 _mm_setzero_ph(void) {
+  return (__m128h){0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0};
+}
+
+static __inline __m256h __DEFAULT_FN_ATTRS256 _mm256_setzero_ph(void) {
+  return (__m256h){0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
+                   0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0};
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256 _mm256_undefined_ph() {
+  return (__m256h)__builtin_ia32_undef256();
+}
+
+static __inline __m512h __DEFAULT_FN_ATTRS512 _mm512_setzero_ph(void) {
+  return (__m512h){0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
+                   0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
+                   0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0};
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_undefined_ph() {
+  return (__m128h)__builtin_ia32_undef128();
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512 _mm512_undefined_ph() {
+  return (__m512h)__builtin_ia32_undef512();
+}
+
+static __inline __m512h __DEFAULT_FN_ATTRS512 _mm512_set1_ph(_Float16 __h) {
+  return (__m512h)(__v32hf){__h, __h, __h, __h, __h, __h, __h, __h,
+                            __h, __h, __h, __h, __h, __h, __h, __h,
+                            __h, __h, __h, __h, __h, __h, __h, __h,
+                            __h, __h, __h, __h, __h, __h, __h, __h};
+}
+
+static __inline __m512h __DEFAULT_FN_ATTRS512
+_mm512_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+              _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+              _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+              _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16,
+              _Float16 __h17, _Float16 __h18, _Float16 __h19, _Float16 __h20,
+              _Float16 __h21, _Float16 __h22, _Float16 __h23, _Float16 __h24,
+              _Float16 __h25, _Float16 __h26, _Float16 __h27, _Float16 __h28,
+              _Float16 __h29, _Float16 __h30, _Float16 __h31, _Float16 __h32) {
+  return (__m512h)(__v32hf){__h1,  __h2,  __h3,  __h4,  __h5,  __h6,  __h7,
+                            __h8,  __h9,  __h10, __h11, __h12, __h13, __h14,
+                            __h15, __h16, __h17, __h18, __h19, __h20, __h21,
+                            __h22, __h23, __h24, __h25, __h26, __h27, __h28,
+                            __h29, __h30, __h31, __h32};
+}
+
+#define _mm512_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8, __h9,   \
+                       __h10, __h11, __h12, __h13, __h14, __h15, __h16, __h17, \
+                       __h18, __h19, __h20, __h21, __h22, __h23, __h24, __h25, \
+                       __h26, __h27, __h28, __h29, __h30, __h31, __h32)        \
+  _mm512_set_ph((__h32), (__h31), (__h30), (__h29), (__h28), (__h27), (__h26), \
+                (__h25), (__h24), (__h23), (__h22), (__h21), (__h20), (__h19), \
+                (__h18), (__h17), (__h16), (__h15), (__h14), (__h13), (__h12), \
+                (__h11), (__h10), (__h9), (__h8), (__h7), (__h6), (__h5),      \
+                (__h4), (__h3), (__h2), (__h1))
+
+static __inline__ __m128 __DEFAULT_FN_ATTRS128 _mm_castph_ps(__m128h __a) {
+  return (__m128)__a;
+}
+
+static __inline__ __m256 __DEFAULT_FN_ATTRS256 _mm256_castph_ps(__m256h __a) {
+  return (__m256)__a;
+}
+
+static __inline__ __m512 __DEFAULT_FN_ATTRS512 _mm512_castph_ps(__m512h __a) {
+  return (__m512)__a;
+}
+
+static __inline__ __m128d __DEFAULT_FN_ATTRS128 _mm_castph_pd(__m128h __a) {
+  return (__m128d)__a;
+}
+
+static __inline__ __m256d __DEFAULT_FN_ATTRS256 _mm256_castph_pd(__m256h __a) {
+  return (__m256d)__a;
+}
+
+static __inline__ __m512d __DEFAULT_FN_ATTRS512 _mm512_castph_pd(__m512h __a) {
+  return (__m512d)__a;
+}
+
+static __inline__ __m128i __DEFAULT_FN_ATTRS128 _mm_castph_si128(__m128h __a) {
+  return (__m128i)__a;
+}
+
+static __inline__ __m256i __DEFAULT_FN_ATTRS256
+_mm256_castph_si256(__m256h __a) {
+  return (__m256i)__a;
+}
+
+static __inline__ __m512i __DEFAULT_FN_ATTRS512
+_mm512_castph_si512(__m512h __a) {
+  return (__m512i)__a;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_castps_ph(__m128 __a) {
+  return (__m128h)__a;
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256 _mm256_castps_ph(__m256 __a) {
+  return (__m256h)__a;
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512 _mm512_castps_ph(__m512 __a) {
+  return (__m512h)__a;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_castpd_ph(__m128d __a) {
+  return (__m128h)__a;
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256 _mm256_castpd_ph(__m256d __a) {
+  return (__m256h)__a;
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512 _mm512_castpd_ph(__m512d __a) {
+  return (__m512h)__a;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_castsi128_ph(__m128i __a) {
+  return (__m128h)__a;
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_castsi256_ph(__m256i __a) {
+  return (__m256h)__a;
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_castsi512_ph(__m512i __a) {
+  return (__m512h)__a;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS256
+_mm256_castph256_ph128(__m256h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS512
+_mm512_castph512_ph128(__m512h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS512
+_mm512_castph512_ph256(__m512h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
+                                 12, 13, 14, 15);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_castph128_ph256(__m128h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7, -1, -1, -1,
+                                 -1, -1, -1, -1, -1);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_castph128_ph512(__m128h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7, -1, -1, -1,
+                                 -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+                                 -1, -1, -1, -1, -1, -1, -1, -1, -1);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_castph256_ph512(__m256h __a) {
+  return __builtin_shufflevector(__a, __a, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
+                                 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1,
+                                 -1, -1, -1, -1, -1, -1, -1, -1);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_zextph128_ph256(__m128h __a) {
+  return __builtin_shufflevector(__a, (__v8hf)_mm_setzero_ph(), 0, 1, 2, 3, 4,
+                                 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_zextph128_ph512(__m128h __a) {
+  return __builtin_shufflevector(
+      __a, (__v8hf)_mm_setzero_ph(), 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
+      13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15, 8, 9, 10, 11, 12, 13, 14, 15);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_zextph256_ph512(__m256h __a) {
+  return __builtin_shufflevector(__a, (__v16hf)_mm256_setzero_ph(), 0, 1, 2, 3,
+                                 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
+                                 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
+                                 29, 30, 31);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512 _mm512_abs_ph(__m512h __A) {
+  return (__m512h)_mm512_and_epi32(_mm512_set1_epi32(0x7FFF7FFF), (__m512i)__A);
+}
+
+// loads with vmovsh:
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_load_sh(void const *__dp) {
+  struct __mm_load_sh_struct {
+    _Float16 __u;
+  } __attribute__((__packed__, __may_alias__));
+  _Float16 __u = ((struct __mm_load_sh_struct *)__dp)->__u;
+  return (__m128h){__u, 0, 0, 0, 0, 0, 0, 0};
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128
+_mm_mask_load_sh(__m128h __W, __mmask8 __U, const void *__A) {
+  __m128h src = (__v8hf)__builtin_shufflevector(
+      (__v8hf)__W, (__v8hf)_mm_setzero_ph(), 0, 8, 8, 8, 8, 8, 8, 8);
+
+  return (__m128h)__builtin_ia32_loadsh128_mask((__v8hf *)__A, src, __U & 1);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128
+_mm_maskz_load_sh(__mmask8 __U, const void *__A) {
+  return (__m128h)__builtin_ia32_loadsh128_mask(
+      (__v8hf *)__A, (__v8hf)_mm_setzero_ph(), __U & 1);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_load_ph(void const *__p) {
+  return *(const __m512h *)__p;
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_load_ph(void const *__p) {
+  return *(const __m256h *)__p;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_load_ph(void const *__p) {
+  return *(const __m128h *)__p;
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_loadu_ph(void const *__p) {
+  struct __loadu_ph {
+    __m512h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  return ((const struct __loadu_ph *)__p)->__v;
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_loadu_ph(void const *__p) {
+  struct __loadu_ph {
+    __m256h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  return ((const struct __loadu_ph *)__p)->__v;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_loadu_ph(void const *__p) {
+  struct __loadu_ph {
+    __m128h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  return ((const struct __loadu_ph *)__p)->__v;
+}
+
+// stores with vmovsh:
+static __inline__ void __DEFAULT_FN_ATTRS128 _mm_store_sh(void *__dp,
+                                                          __m128h __a) {
+  struct __mm_store_sh_struct {
+    _Float16 __u;
+  } __attribute__((__packed__, __may_alias__));
+  ((struct __mm_store_sh_struct *)__dp)->__u = __a[0];
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS128 _mm_mask_store_sh(void *__W,
+                                                               __mmask8 __U,
+                                                               __m128h __A) {
+  __builtin_ia32_storesh128_mask((__v8hf *)__W, __A, __U & 1);
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS512 _mm512_store_ph(void *__P,
+                                                             __m512h __A) {
+  *(__m512h *)__P = __A;
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS256 _mm256_store_ph(void *__P,
+                                                             __m256h __A) {
+  *(__m256h *)__P = __A;
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS128 _mm_store_ph(void *__P,
+                                                          __m128h __A) {
+  *(__m128h *)__P = __A;
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS512 _mm512_storeu_ph(void *__P,
+                                                              __m512h __A) {
+  struct __storeu_ph {
+    __m512h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  ((struct __storeu_ph *)__P)->__v = __A;
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS256 _mm256_storeu_ph(void *__P,
+                                                              __m256h __A) {
+  struct __storeu_ph {
+    __m256h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  ((struct __storeu_ph *)__P)->__v = __A;
+}
+
+static __inline__ void __DEFAULT_FN_ATTRS128 _mm_storeu_ph(void *__P,
+                                                           __m128h __A) {
+  struct __storeu_ph {
+    __m128h_u __v;
+  } __attribute__((__packed__, __may_alias__));
+  ((struct __storeu_ph *)__P)->__v = __A;
+}
+
+// moves with vmovsh:
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_move_sh(__m128h __a,
+                                                            __m128h __b) {
+  __a[0] = __b[0];
+  return __a;
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_mask_move_sh(__m128h __W,
+                                                                 __mmask8 __U,
+                                                                 __m128h __A,
+                                                                 __m128h __B) {
+  return __builtin_ia32_selectsh_128(__U, _mm_move_sh(__A, __B), __W);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_maskz_move_sh(__mmask8 __U,
+                                                                  __m128h __A,
+                                                                  __m128h __B) {
+  return __builtin_ia32_selectsh_128(__U, _mm_move_sh(__A, __B),
+                                     _mm_setzero_ph());
+}
+
+// vmovw:
+static __inline__ __m128i __DEFAULT_FN_ATTRS128 _mm_cvtsi16_si128(short __a) {
+  return (__m128i)(__v8hi){__a, 0, 0, 0, 0, 0, 0, 0};
+}
+
+static __inline__ short __DEFAULT_FN_ATTRS128 _mm_cvtsi128_si16(__m128i __a) {
+  __v8hi __b = (__v8hi)__a;
+  return __b[0];
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_mask_blend_ph(__mmask32 __U, __m512h __A, __m512h __W) {
+  return (__m512h)__builtin_ia32_selectph_512((__mmask32)__U, (__v32hf)__W,
+                                              (__v32hf)__A);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_permutex2var_ph(__m512h __A, __m512i __I, __m512h __B) {
+  return (__m512h)__builtin_ia32_vpermi2varhi512((__v32hi)__A, (__v32hi)__I,
+                                                 (__v32hi)__B);
+}
+
+static __inline__ __m512h __DEFAULT_FN_ATTRS512
+_mm512_permutexvar_ph(__m512i __A, __m512h __B) {
+  return (__m512h)__builtin_ia32_permvarhi512((__v32hi)__B, (__v32hi)__A);
+}
+
+#undef __DEFAULT_FN_ATTRS128
+#undef __DEFAULT_FN_ATTRS256
+#undef __DEFAULT_FN_ATTRS512
+
+#endif

clang/lib/Headers/avx512vlfp16intrin.h

+/*===---------- avx512vlfp16intrin.h - AVX512-FP16 intrinsics --------------===
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===-----------------------------------------------------------------------===
+ */
+#ifndef __IMMINTRIN_H
+#error                                                                         \
+    "Never use <avx512vlfp16intrin.h> directly; include <immintrin.h> instead."
+#endif
+
+#ifndef __AVX512VLFP16INTRIN_H
+#define __AVX512VLFP16INTRIN_H
+
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS256                                                  \
+  __attribute__((__always_inline__, __nodebug__,                               \
+                 __target__("avx512fp16, avx512vl"),                           \
+                 __min_vector_width__(256)))
+#define __DEFAULT_FN_ATTRS128                                                  \
+  __attribute__((__always_inline__, __nodebug__,                               \
+                 __target__("avx512fp16, avx512vl"),                           \
+                 __min_vector_width__(128)))
+
+static __inline__ _Float16 __DEFAULT_FN_ATTRS128 _mm_cvtsh_h(__m128h __a) {
+  return __a[0];
+}
+
+static __inline__ _Float16 __DEFAULT_FN_ATTRS256 _mm256_cvtsh_h(__m256h __a) {
+  return __a[0];
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_set_sh(_Float16 __h) {
+  return __extension__(__m128h){__h, 0, 0, 0, 0, 0, 0, 0};
+}
+
+static __inline __m128h __DEFAULT_FN_ATTRS128 _mm_set1_ph(_Float16 __h) {
+  return (__m128h)(__v8hf){__h, __h, __h, __h, __h, __h, __h, __h};
+}
+
+static __inline __m256h __DEFAULT_FN_ATTRS256 _mm256_set1_ph(_Float16 __h) {
+  return (__m256h)(__v16hf){__h, __h, __h, __h, __h, __h, __h, __h,
+                            __h, __h, __h, __h, __h, __h, __h, __h};
+}
+
+static __inline __m128h __DEFAULT_FN_ATTRS128
+_mm_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+           _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8) {
+  return (__m128h)(__v8hf){__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8};
+}
+
+static __inline __m256h __DEFAULT_FN_ATTRS256
+_mm256_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+              _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+              _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+              _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16) {
+  return (__m256h)(__v16hf){__h1,  __h2,  __h3,  __h4,  __h5,  __h6,
+                            __h7,  __h8,  __h9,  __h10, __h11, __h12,
+                            __h13, __h14, __h15, __h16};
+}
+
+#define _mm_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8)            \
+  _mm_set_ph((__h8), (__h7), (__h6), (__h5), (__h4), (__h3), (__h2), (__h1))
+
+#define _mm256_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8, __h9,   \
+                       __h10, __h11, __h12, __h13, __h14, __h15, __h16)        \
+  _mm256_set_ph((__h16), (__h15), (__h14), (__h13), (__h12), (__h11), (__h10), \
+                (__h9), (__h8), (__h7), (__h6), (__h5), (__h4), (__h3),        \
+                (__h2), (__h1))
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256 _mm256_abs_ph(__m256h __A) {
+  return (__m256h)_mm256_and_epi32(_mm256_set1_epi32(0x7FFF7FFF), (__m256i)__A);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_abs_ph(__m128h __A) {
+  return (__m128h)_mm_and_epi32(_mm_set1_epi32(0x7FFF7FFF), (__m128i)__A);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128 _mm_mask_blend_ph(__mmask8 __U,
+                                                                  __m128h __A,
+                                                                  __m128h __W) {
+  return (__m128h)__builtin_ia32_selectph_128((__mmask8)__U, (__v8hf)__W,
+                                              (__v8hf)__A);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_mask_blend_ph(__mmask16 __U, __m256h __A, __m256h __W) {
+  return (__m256h)__builtin_ia32_selectph_256((__mmask16)__U, (__v16hf)__W,
+                                              (__v16hf)__A);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128
+_mm_permutex2var_ph(__m128h __A, __m128i __I, __m128h __B) {
+  return (__m128h)__builtin_ia32_vpermi2varhi128((__v8hi)__A, (__v8hi)__I,
+                                                 (__v8hi)__B);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_permutex2var_ph(__m256h __A, __m256i __I, __m256h __B) {
+  return (__m256h)__builtin_ia32_vpermi2varhi256((__v16hi)__A, (__v16hi)__I,
+                                                 (__v16hi)__B);
+}
+
+static __inline__ __m128h __DEFAULT_FN_ATTRS128
+_mm_permutexvar_ph(__m128i __A, __m128h __B) {
+  return (__m128h)__builtin_ia32_permvarhi128((__v8hi)__B, (__v8hi)__A);
+}
+
+static __inline__ __m256h __DEFAULT_FN_ATTRS256
+_mm256_permutexvar_ph(__m256i __A, __m256h __B) {
+  return (__m256h)__builtin_ia32_permvarhi256((__v16hi)__B, (__v16hi)__A);
+}
+
+#undef __DEFAULT_FN_ATTRS128
+#undef __DEFAULT_FN_ATTRS256
+
+#endif

clang/lib/Headers/cpuid.h

 /*===---- cpuid.h - X86 cpu model detection --------------------------------===
  *
  * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
  * See https://llvm.org/LICENSE.txt for license information.
  * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
  *
  *===-----------------------------------------------------------------------===
  */

 #define bit_AVX5124VNNIW  0x00000004
 #define bit_AVX5124FMAPS  0x00000008
 #define bit_UINTR         0x00000020
 #define bit_SERIALIZE     0x00004000
 #define bit_TSXLDTRK      0x00010000
 #define bit_PCONFIG       0x00040000
 #define bit_IBT           0x00100000
 #define bit_AMXBF16       0x00400000
+#define bit_AVX512FP16    0x00800000
 #define bit_AMXTILE       0x01000000
 #define bit_AMXINT8       0x02000000
 
 /* Features in %eax for leaf 7 sub-leaf 1 */
 #define bit_AVXVNNI       0x00000008
 #define bit_AVX512BF16    0x00000020
 #define bit_HRESET        0x00400000
 

clang/lib/Headers/immintrin.h

 /*===---- immintrin.h - Intel intrinsics -----------------------------------===
  *
  * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
  * See https://llvm.org/LICENSE.txt for license information.
  * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
  *
  *===-----------------------------------------------------------------------===
  */

 #include <avx512vlvbmi2intrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AVX512PF__)
 #include <avx512pfintrin.h>
 #endif
 
+/*
+ * FIXME: _Float16 type is legal only when HW support float16 operation.
+ * We use __AVX512FP16__ to identify if float16 is supported or not, so
+ * when float16 is not supported, the related header is not included.
+ *
+ */
+#if defined(__AVX512FP16__)
+#include <avx512fp16intrin.h>
+#endif
+
+#if defined(__AVX512FP16__) && defined(__AVX512VL__)
+#include <avx512vlfp16intrin.h>
+#endif
+
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AVX512BF16__)
 #include <avx512bf16intrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     (defined(__AVX512VL__) && defined(__AVX512BF16__))
 #include <avx512vlbf16intrin.h>

clang/lib/Sema/DeclSpec.cpp

 //===--- DeclSpec.cpp - Declaration Specifier Semantic Analysis -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (TypeSpecType == TST_unspecified) {
                                                  " double");
       TypeSpecType = TST_double;   // _Complex -> _Complex double.
     } else if (TypeSpecType == TST_int || TypeSpecType == TST_char ||
                TypeSpecType == TST_extint) {
       // Note that this intentionally doesn't include _Complex _Bool.
       if (!S.getLangOpts().CPlusPlus)
         S.Diag(TSTLoc, diag::ext_integer_complex);
     } else if (TypeSpecType != TST_float && TypeSpecType != TST_double &&
-               TypeSpecType != TST_float128) {
-      // FIXME: _Float16, __fp16?
+               TypeSpecType != TST_float128 && TypeSpecType != TST_float16) {
+      // FIXME: __fp16?
       S.Diag(TSCLoc, diag::err_invalid_complex_spec)
         << getSpecifierName((TST)TypeSpecType, Policy);
       TypeSpecComplex = TSC_unspecified;
     }
   }
 
   // C11 6.7.1/3, C++11 [dcl.stc]p1, GNU TLS: __thread, thread_local and
   // _Thread_local can only appear with the 'static' and 'extern' storage class

clang/test/CodeGen/X86/avx512fp16-abi.c

+// RUN: %clang_cc1 -triple x86_64-linux -emit-llvm  -target-feature +avx512fp16 < %s | FileCheck %s --check-prefixes=CHECK
+
+struct half1 {
+  _Float16 a;
+};
+
+struct half1 h1(_Float16 a) {
+  // CHECK: define{{.*}}half @h1
+  struct half1 x;
+  x.a = a;
+  return x;
+}
+
+struct half2 {
+  _Float16 a;
+  _Float16 b;
+};
+
+struct half2 h2(_Float16 a, _Float16 b) {
+  // CHECK: define{{.*}}<2 x half> @h2
+  struct half2 x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct half3 {
+  _Float16 a;
+  _Float16 b;
+  _Float16 c;
+};
+
+struct half3 h3(_Float16 a, _Float16 b, _Float16 c) {
+  // CHECK: define{{.*}}<3 x half> @h3
+  struct half3 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct half4 {
+  _Float16 a;
+  _Float16 b;
+  _Float16 c;
+  _Float16 d;
+};
+
+struct half4 h4(_Float16 a, _Float16 b, _Float16 c, _Float16 d) {
+  // CHECK: define{{.*}}<4 x half> @h4
+  struct half4 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  x.d = d;
+  return x;
+}
+
+struct floathalf {
+  float a;
+  _Float16 b;
+};
+
+struct floathalf fh(float a, _Float16 b) {
+  // CHECK: <3 x half> @fh
+  struct floathalf x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct floathalf2 {
+  float a;
+  _Float16 b;
+  _Float16 c;
+};
+
+struct floathalf2 fh2(float a, _Float16 b, _Float16 c) {
+  // CHECK: define{{.*}}double @fh2
+  struct floathalf2 x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}
+
+struct halffloat {
+  _Float16 a;
+  float b;
+};
+
+struct halffloat hf(_Float16 a, float b) {
+  // CHECK: define{{.*}}double @hf
+  struct halffloat x;
+  x.a = a;
+  x.b = b;
+  return x;
+}
+
+struct half2float {
+  _Float16 a;
+  _Float16 b;
+  float c;
+};
+
+struct half2float h2f(_Float16 a, _Float16 b, float c) {
+  // CHECK: define{{.*}}double @h2f
+  struct half2float x;
+  x.a = a;
+  x.b = b;
+  x.c = c;
+  return x;
+}

clang/test/CodeGen/X86/avx512fp16-builtins.c

+// RUN: %clang_cc1 -ffreestanding -flax-vector-conversions=none %s -triple=x86_64-unknown-unknown -target-feature +avx512fp16 -emit-llvm -o - -Wall -Werror | FileCheck %s
+
+#include <immintrin.h>
+
+_Float16 test_mm512_cvtsh_h(__m512h __A) {
+  // CHECK-LABEL: @test_mm512_cvtsh_h
+  // CHECK: extractelement <32 x half> %{{.*}}, i32 0
+  return _mm512_cvtsh_h(__A);
+}
+
+__m128h test_mm_setzero_ph() {
+  // CHECK-LABEL: @test_mm_setzero_ph
+  // CHECK: zeroinitializer
+  return _mm_setzero_ph();
+}
+
+__m256h test_mm256_setzero_ph() {
+  // CHECK-LABEL: @test_mm256_setzero_ph
+  // CHECK: zeroinitializer
+  return _mm256_setzero_ph();
+}
+
+__m256h test_mm256_undefined_ph() {
+  // CHECK-LABEL: @test_mm256_undefined_ph
+  // CHECK: ret <16 x half> zeroinitializer
+  return _mm256_undefined_ph();
+}
+
+__m512h test_mm512_setzero_ph() {
+  // CHECK-LABEL: @test_mm512_setzero_ph
+  // CHECK: zeroinitializer
+  return _mm512_setzero_ph();
+}
+
+__m128h test_mm_undefined_ph() {
+  // CHECK-LABEL: @test_mm_undefined_ph
+  // CHECK: ret <8 x half> zeroinitializer
+  return _mm_undefined_ph();
+}
+
+__m512h test_mm512_undefined_ph() {
+  // CHECK-LABEL: @test_mm512_undefined_ph
+  // CHECK: ret <32 x half> zeroinitializer
+  return _mm512_undefined_ph();
+}
+
+__m512h test_mm512_set1_ph(_Float16 h) {
+  // CHECK-LABEL: @test_mm512_set1_ph
+  // CHECK: insertelement <32 x half> {{.*}}, i32 0
+  // CHECK: insertelement <32 x half> {{.*}}, i32 1
+  // CHECK: insertelement <32 x half> {{.*}}, i32 2
+  // CHECK: insertelement <32 x half> {{.*}}, i32 3
+  // CHECK: insertelement <32 x half> {{.*}}, i32 4
+  // CHECK: insertelement <32 x half> {{.*}}, i32 5
+  // CHECK: insertelement <32 x half> {{.*}}, i32 6
+  // CHECK: insertelement <32 x half> {{.*}}, i32 7
+  // CHECK: insertelement <32 x half> {{.*}}, i32 8
+  // CHECK: insertelement <32 x half> {{.*}}, i32 9
+  // CHECK: insertelement <32 x half> {{.*}}, i32 10
+  // CHECK: insertelement <32 x half> {{.*}}, i32 11
+  // CHECK: insertelement <32 x half> {{.*}}, i32 12
+  // CHECK: insertelement <32 x half> {{.*}}, i32 13
+  // CHECK: insertelement <32 x half> {{.*}}, i32 14
+  // CHECK: insertelement <32 x half> {{.*}}, i32 15
+  // CHECK: insertelement <32 x half> {{.*}}, i32 16
+  // CHECK: insertelement <32 x half> {{.*}}, i32 17
+  // CHECK: insertelement <32 x half> {{.*}}, i32 18
+  // CHECK: insertelement <32 x half> {{.*}}, i32 19
+  // CHECK: insertelement <32 x half> {{.*}}, i32 20
+  // CHECK: insertelement <32 x half> {{.*}}, i32 21
+  // CHECK: insertelement <32 x half> {{.*}}, i32 22
+  // CHECK: insertelement <32 x half> {{.*}}, i32 23
+  // CHECK: insertelement <32 x half> {{.*}}, i32 24
+  // CHECK: insertelement <32 x half> {{.*}}, i32 25
+  // CHECK: insertelement <32 x half> {{.*}}, i32 26
+  // CHECK: insertelement <32 x half> {{.*}}, i32 27
+  // CHECK: insertelement <32 x half> {{.*}}, i32 28
+  // CHECK: insertelement <32 x half> {{.*}}, i32 29
+  // CHECK: insertelement <32 x half> {{.*}}, i32 30
+  // CHECK: insertelement <32 x half> {{.*}}, i32 31
+  return _mm512_set1_ph(h);
+}
+
+__m512h test_mm512_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                          _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+                          _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+                          _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16,
+                          _Float16 __h17, _Float16 __h18, _Float16 __h19, _Float16 __h20,
+                          _Float16 __h21, _Float16 __h22, _Float16 __h23, _Float16 __h24,
+                          _Float16 __h25, _Float16 __h26, _Float16 __h27, _Float16 __h28,
+                          _Float16 __h29, _Float16 __h30, _Float16 __h31, _Float16 __h32) {
+  // CHECK-LABEL: @test_mm512_set_ph
+  // CHECK: insertelement <32 x half> {{.*}}, i32 0
+  // CHECK: insertelement <32 x half> {{.*}}, i32 1
+  // CHECK: insertelement <32 x half> {{.*}}, i32 2
+  // CHECK: insertelement <32 x half> {{.*}}, i32 3
+  // CHECK: insertelement <32 x half> {{.*}}, i32 4
+  // CHECK: insertelement <32 x half> {{.*}}, i32 5
+  // CHECK: insertelement <32 x half> {{.*}}, i32 6
+  // CHECK: insertelement <32 x half> {{.*}}, i32 7
+  // CHECK: insertelement <32 x half> {{.*}}, i32 8
+  // CHECK: insertelement <32 x half> {{.*}}, i32 9
+  // CHECK: insertelement <32 x half> {{.*}}, i32 10
+  // CHECK: insertelement <32 x half> {{.*}}, i32 11
+  // CHECK: insertelement <32 x half> {{.*}}, i32 12
+  // CHECK: insertelement <32 x half> {{.*}}, i32 13
+  // CHECK: insertelement <32 x half> {{.*}}, i32 14
+  // CHECK: insertelement <32 x half> {{.*}}, i32 15
+  // CHECK: insertelement <32 x half> {{.*}}, i32 16
+  // CHECK: insertelement <32 x half> {{.*}}, i32 17
+  // CHECK: insertelement <32 x half> {{.*}}, i32 18
+  // CHECK: insertelement <32 x half> {{.*}}, i32 19
+  // CHECK: insertelement <32 x half> {{.*}}, i32 20
+  // CHECK: insertelement <32 x half> {{.*}}, i32 21
+  // CHECK: insertelement <32 x half> {{.*}}, i32 22
+  // CHECK: insertelement <32 x half> {{.*}}, i32 23
+  // CHECK: insertelement <32 x half> {{.*}}, i32 24
+  // CHECK: insertelement <32 x half> {{.*}}, i32 25
+  // CHECK: insertelement <32 x half> {{.*}}, i32 26
+  // CHECK: insertelement <32 x half> {{.*}}, i32 27
+  // CHECK: insertelement <32 x half> {{.*}}, i32 28
+  // CHECK: insertelement <32 x half> {{.*}}, i32 29
+  // CHECK: insertelement <32 x half> {{.*}}, i32 30
+  // CHECK: insertelement <32 x half> {{.*}}, i32 31
+  return _mm512_set_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8,
+                       __h9, __h10, __h11, __h12, __h13, __h14, __h15, __h16,
+                       __h17, __h18, __h19, __h20, __h21, __h22, __h23, __h24,
+                       __h25, __h26, __h27, __h28, __h29, __h30, __h31, __h32);
+}
+
+__m512h test_mm512_setr_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                           _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+                           _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+                           _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16,
+                           _Float16 __h17, _Float16 __h18, _Float16 __h19, _Float16 __h20,
+                           _Float16 __h21, _Float16 __h22, _Float16 __h23, _Float16 __h24,
+                           _Float16 __h25, _Float16 __h26, _Float16 __h27, _Float16 __h28,
+                           _Float16 __h29, _Float16 __h30, _Float16 __h31, _Float16 __h32) {
+  // CHECK-LABEL: @test_mm512_setr_ph
+  // CHECK: insertelement <32 x half> {{.*}}, i32 0
+  // CHECK: insertelement <32 x half> {{.*}}, i32 1
+  // CHECK: insertelement <32 x half> {{.*}}, i32 2
+  // CHECK: insertelement <32 x half> {{.*}}, i32 3
+  // CHECK: insertelement <32 x half> {{.*}}, i32 4
+  // CHECK: insertelement <32 x half> {{.*}}, i32 5
+  // CHECK: insertelement <32 x half> {{.*}}, i32 6
+  // CHECK: insertelement <32 x half> {{.*}}, i32 7
+  // CHECK: insertelement <32 x half> {{.*}}, i32 8
+  // CHECK: insertelement <32 x half> {{.*}}, i32 9
+  // CHECK: insertelement <32 x half> {{.*}}, i32 10
+  // CHECK: insertelement <32 x half> {{.*}}, i32 11
+  // CHECK: insertelement <32 x half> {{.*}}, i32 12
+  // CHECK: insertelement <32 x half> {{.*}}, i32 13
+  // CHECK: insertelement <32 x half> {{.*}}, i32 14
+  // CHECK: insertelement <32 x half> {{.*}}, i32 15
+  // CHECK: insertelement <32 x half> {{.*}}, i32 16
+  // CHECK: insertelement <32 x half> {{.*}}, i32 17
+  // CHECK: insertelement <32 x half> {{.*}}, i32 18
+  // CHECK: insertelement <32 x half> {{.*}}, i32 19
+  // CHECK: insertelement <32 x half> {{.*}}, i32 20
+  // CHECK: insertelement <32 x half> {{.*}}, i32 21
+  // CHECK: insertelement <32 x half> {{.*}}, i32 22
+  // CHECK: insertelement <32 x half> {{.*}}, i32 23
+  // CHECK: insertelement <32 x half> {{.*}}, i32 24
+  // CHECK: insertelement <32 x half> {{.*}}, i32 25
+  // CHECK: insertelement <32 x half> {{.*}}, i32 26
+  // CHECK: insertelement <32 x half> {{.*}}, i32 27
+  // CHECK: insertelement <32 x half> {{.*}}, i32 28
+  // CHECK: insertelement <32 x half> {{.*}}, i32 29
+  // CHECK: insertelement <32 x half> {{.*}}, i32 30
+  // CHECK: insertelement <32 x half> {{.*}}, i32 31
+  return _mm512_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8,
+                        __h9, __h10, __h11, __h12, __h13, __h14, __h15, __h16,
+                        __h17, __h18, __h19, __h20, __h21, __h22, __h23, __h24,
+                        __h25, __h26, __h27, __h28, __h29, __h30, __h31, __h32);
+}
+
+__m128 test_mm_castph_ps(__m128h A) {
+  // CHECK-LABEL: test_mm_castph_ps
+  // CHECK: bitcast <8 x half> %{{.*}} to <4 x float>
+  return _mm_castph_ps(A);
+}
+
+__m256 test_mm256_castph_ps(__m256h A) {
+  // CHECK-LABEL: test_mm256_castph_ps
+  // CHECK: bitcast <16 x half> %{{.*}} to <8 x float>
+  return _mm256_castph_ps(A);
+}
+
+__m512 test_mm512_castph_ps(__m512h A) {
+  // CHECK-LABEL: test_mm512_castph_ps
+  // CHECK: bitcast <32 x half> %{{.*}} to <16 x float>
+  return _mm512_castph_ps(A);
+}
+
+__m128d test_mm_castph_pd(__m128h A) {
+  // CHECK-LABEL: test_mm_castph_pd
+  // CHECK: bitcast <8 x half> %{{.*}} to <2 x double>
+  return _mm_castph_pd(A);
+}
+
+__m256d test_mm256_castph_pd(__m256h A) {
+  // CHECK-LABEL: test_mm256_castph_pd
+  // CHECK: bitcast <16 x half> %{{.*}} to <4 x double>
+  return _mm256_castph_pd(A);
+}
+
+__m512d test_mm512_castph_pd(__m512h A) {
+  // CHECK-LABEL: test_mm512_castph_pd
+  // CHECK: bitcast <32 x half> %{{.*}} to <8 x double>
+  return _mm512_castph_pd(A);
+}
+
+__m128i test_mm_castph_si128(__m128h A) {
+  // CHECK-LABEL: test_mm_castph_si128
+  // CHECK: bitcast <8 x half> %{{.*}} to <2 x i64>
+  return _mm_castph_si128(A);
+}
+
+__m256i test_mm256_castph_si256(__m256h A) {
+  // CHECK-LABEL: test_mm256_castph_si256
+  // CHECK: bitcast <16 x half> %{{.*}} to <4 x i64>
+  return _mm256_castph_si256(A);
+}
+
+__m512i test_mm512_castph_si512(__m512h A) {
+  // CHECK-LABEL: test_mm512_castph_si512
+  // CHECK: bitcast <32 x half> %{{.*}} to <8 x i64>
+  return _mm512_castph_si512(A);
+}
+
+__m128h test_mm_castps_ph(__m128 A) {
+  // CHECK-LABEL: test_mm_castps_ph
+  // CHECK: bitcast <4 x float> %{{.*}} to <8 x half>
+  return _mm_castps_ph(A);
+}
+
+__m256h test_mm256_castps_ph(__m256 A) {
+  // CHECK-LABEL: test_mm256_castps_ph
+  // CHECK: bitcast <8 x float> %{{.*}} to <16 x half>
+  return _mm256_castps_ph(A);
+}
+
+__m512h test_mm512_castps_ph(__m512 A) {
+  // CHECK-LABEL: test_mm512_castps_ph
+  // CHECK: bitcast <16 x float> %{{.*}} to <32 x half>
+  return _mm512_castps_ph(A);
+}
+
+__m128h test_mm_castpd_ph(__m128d A) {
+  // CHECK-LABEL: test_mm_castpd_ph
+  // CHECK: bitcast <2 x double> %{{.*}} to <8 x half>
+  return _mm_castpd_ph(A);
+}
+
+__m256h test_mm256_castpd_ph(__m256d A) {
+  // CHECK-LABEL: test_mm256_castpd_ph
+  // CHECK: bitcast <4 x double> %{{.*}} to <16 x half>
+  return _mm256_castpd_ph(A);
+}
+
+__m512h test_mm512_castpd_ph(__m512d A) {
+  // CHECK-LABEL: test_mm512_castpd_ph
+  // CHECK: bitcast <8 x double> %{{.*}} to <32 x half>
+  return _mm512_castpd_ph(A);
+}
+
+__m128h test_mm_castsi128_ph(__m128i A) {
+  // CHECK-LABEL: test_mm_castsi128_ph
+  // CHECK: bitcast <2 x i64> %{{.*}} to <8 x half>
+  return _mm_castsi128_ph(A);
+}
+
+__m256h test_mm256_castsi256_ph(__m256i A) {
+  // CHECK-LABEL: test_mm256_castsi256_ph
+  // CHECK: bitcast <4 x i64> %{{.*}} to <16 x half>
+  return _mm256_castsi256_ph(A);
+}
+
+__m512h test_mm512_castsi512_ph(__m512i A) {
+  // CHECK-LABEL: test_mm512_castsi512_ph
+  // CHECK: bitcast <8 x i64> %{{.*}} to <32 x half>
+  return _mm512_castsi512_ph(A);
+}
+
+__m128h test_mm256_castph256_ph128(__m256h __a) {
+  // CHECK-LABEL: test_mm256_castph256_ph128
+  // CHECK: shufflevector <16 x half> %{{.*}}, <16 x half> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  return _mm256_castph256_ph128(__a);
+}
+
+__m128h test_mm512_castph512_ph128(__m512h __a) {
+  // CHECK-LABEL: test_mm512_castph512_ph128
+  // CHECK: shufflevector <32 x half> %{{.*}}, <32 x half> %{{.*}}, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  return _mm512_castph512_ph128(__a);
+}
+
+__m256h test_mm512_castph512_ph256(__m512h __a) {
+  // CHECK-LABEL: test_mm512_castph512_ph256
+  // CHECK: shufflevector <32 x half> %{{.*}}, <32 x half> %{{.*}}, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  return _mm512_castph512_ph256(__a);
+}
+
+__m256h test_mm256_castph128_ph256(__m128h __a) {
+  // CHECK-LABEL: test_mm256_castph128_ph256
+  // CHECK: shufflevector <8 x half> %{{.*}}, <8 x half> %{{.*}}, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  return _mm256_castph128_ph256(__a);
+}
+
+__m512h test_mm512_castph128_ph512(__m128h __a) {
+  // CHECK-LABEL: test_mm512_castph128_ph512
+  // CHECK: shufflevector <8 x half> %{{.*}}, <8 x half> %{{.*}}, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  return _mm512_castph128_ph512(__a);
+}
+
+__m512h test_mm512_castph256_ph512(__m256h __a) {
+  // CHECK-LABEL: test_mm512_castph256_ph512
+  // CHECK: shufflevector <16 x half> %{{.*}}, <16 x half> %{{.*}}, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  return _mm512_castph256_ph512(__a);
+}
+
+__m256h test_mm256_zextph128_ph256(__m128h __a) {
+  // CHECK-LABEL: test_mm256_zextph128_ph256
+  // CHECK: shufflevector <8 x half> %{{.*}}, <8 x half> {{.*}}, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  return _mm256_zextph128_ph256(__a);
+}
+
+__m512h test_mm512_zextph128_ph512(__m128h __a) {
+  // CHECK-LABEL: test_mm512_zextph128_ph512
+  // CHECK: shufflevector <8 x half> %{{.*}}, <8 x half> {{.*}}, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  return _mm512_zextph128_ph512(__a);
+}
+
+__m512h test_mm512_zextph256_ph512(__m256h __a) {
+  // CHECK-LABEL: test_mm512_zextph256_ph512
+  // CHECK: shufflevector <16 x half> %{{.*}}, <16 x half> {{.*}}, <32 x i32>
+  return _mm512_zextph256_ph512(__a);
+}
+
+__m512h test_mm512_abs_ph(__m512h a) {
+  // CHECK-LABEL: @test_mm512_abs_ph
+  // CHECK: and <16 x i32>
+  return _mm512_abs_ph(a);
+}
+
+// VMOVSH
+
+__m128h test_mm_load_sh(void const *A) {
+  // CHECK-LABEL: test_mm_load_sh
+  // CHECK: load half, half* %{{.*}}, align 1{{$}}
+  return _mm_load_sh(A);
+}
+
+__m128h test_mm_mask_load_sh(__m128h __A, __mmask8 __U, const void *__W) {
+  // CHECK-LABEL: @test_mm_mask_load_sh
+  // CHECK: %{{.*}} = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %{{.*}}, i32 1, <8 x i1> %{{.*}}, <8 x half> %{{.*}})
+  return _mm_mask_load_sh(__A, __U, __W);
+}
+
+__m128h test_mm_maskz_load_sh(__mmask8 __U, const void *__W) {
+  // CHECK-LABEL: @test_mm_maskz_load_sh
+  // CHECK: %{{.*}} = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %{{.*}}, i32 1, <8 x i1> %{{.*}}, <8 x half> %{{.*}})
+  return _mm_maskz_load_sh(__U, __W);
+}
+
+__m512h test_mm512_load_ph(void *p) {
+  // CHECK-LABEL: @test_mm512_load_ph
+  // CHECK: load <32 x half>, <32 x half>* %{{.*}}, align 64
+  return _mm512_load_ph(p);
+}
+
+__m256h test_mm256_load_ph(void *p) {
+  // CHECK-LABEL: @test_mm256_load_ph
+  // CHECK: load <16 x half>, <16 x half>* %{{.*}}, align 32
+  return _mm256_load_ph(p);
+}
+
+__m128h test_mm_load_ph(void *p) {
+  // CHECK-LABEL: @test_mm_load_ph
+  // CHECK: load <8 x half>, <8 x half>* %{{.*}}, align 16
+  return _mm_load_ph(p);
+}
+
+__m512h test_mm512_loadu_ph(void *p) {
+  // CHECK-LABEL: @test_mm512_loadu_ph
+  // CHECK: load <32 x half>, <32 x half>* {{.*}}, align 1{{$}}
+  return _mm512_loadu_ph(p);
+}
+
+__m256h test_mm256_loadu_ph(void *p) {
+  // CHECK-LABEL: @test_mm256_loadu_ph
+  // CHECK: load <16 x half>, <16 x half>* {{.*}}, align 1{{$}}
+  return _mm256_loadu_ph(p);
+}
+
+__m128h test_mm_loadu_ph(void *p) {
+  // CHECK-LABEL: @test_mm_loadu_ph
+  // CHECK: load <8 x half>, <8 x half>* {{.*}}, align 1{{$}}
+  return _mm_loadu_ph(p);
+}
+
+void test_mm_store_sh(void *A, __m128h B) {
+  // CHECK-LABEL: test_mm_store_sh
+  // CHECK: extractelement <8 x half> %{{.*}}, i32 0
+  // CHECK: store half %{{.*}}, half* %{{.*}}, align 1{{$}}
+  _mm_store_sh(A, B);
+}
+
+void test_mm_mask_store_sh(void *__P, __mmask8 __U, __m128h __A) {
+  // CHECK-LABEL: @test_mm_mask_store_sh
+  // CHECK: call void @llvm.masked.store.v8f16.p0v8f16(<8 x half> %{{.*}}, <8 x half>* %{{.*}}, i32 1, <8 x i1> %{{.*}})
+  _mm_mask_store_sh(__P, __U, __A);
+}
+
+void test_mm512_store_ph(void *p, __m512h a) {
+  // CHECK-LABEL: @test_mm512_store_ph
+  // CHECK: store <32 x half> %{{.*}}, <32 x half>* %{{.*}}, align 64
+  _mm512_store_ph(p, a);
+}
+
+void test_mm256_store_ph(void *p, __m256h a) {
+  // CHECK-LABEL: @test_mm256_store_ph
+  // CHECK: store <16 x half> %{{.*}}, <16 x half>* %{{.*}}, align 32
+  _mm256_store_ph(p, a);
+}
+
+void test_mm_store_ph(void *p, __m128h a) {
+  // CHECK-LABEL: @test_mm_store_ph
+  // CHECK: store <8 x half> %{{.*}}, <8 x half>* %{{.*}}, align 16
+  _mm_store_ph(p, a);
+}
+
+void test_mm512_storeu_ph(void *p, __m512h a) {
+  // CHECK-LABEL: @test_mm512_storeu_ph
+  // CHECK: store <32 x half> %{{.*}}, <32 x half>* %{{.*}}, align 1{{$}}
+  // CHECK-NEXT: ret void
+  _mm512_storeu_ph(p, a);
+}
+
+void test_mm256_storeu_ph(void *p, __m256h a) {
+  // CHECK-LABEL: @test_mm256_storeu_ph
+  // CHECK: store <16 x half> %{{.*}}, <16 x half>* %{{.*}}, align 1{{$}}
+  // CHECK-NEXT: ret void
+  _mm256_storeu_ph(p, a);
+}
+
+void test_mm_storeu_ph(void *p, __m128h a) {
+  // CHECK-LABEL: @test_mm_storeu_ph
+  // CHECK: store <8 x half> %{{.*}}, <8 x half>* %{{.*}}, align 1{{$}}
+  // CHECK-NEXT: ret void
+  _mm_storeu_ph(p, a);
+}
+
+__m128h test_mm_move_sh(__m128h A, __m128h B) {
+  // CHECK-LABEL: test_mm_move_sh
+  // CHECK: extractelement <8 x half> %{{.*}}, i32 0
+  // CHECK: insertelement <8 x half> %{{.*}}, half %{{.*}}, i32 0
+  return _mm_move_sh(A, B);
+}
+
+__m128h test_mm_mask_move_sh(__m128h __W, __mmask8 __U, __m128h __A, __m128h __B) {
+  // CHECK-LABEL: @test_mm_mask_move_sh
+  // CHECK: [[EXT:%.*]] = extractelement <8 x half> %{{.*}}, i32 0
+  // CHECK: insertelement <8 x half> %{{.*}}, half [[EXT]], i32 0
+  // CHECK: [[A:%.*]] = extractelement <8 x half> [[VEC:%.*]], i64 0
+  // CHECK-NEXT: [[B:%.*]] = extractelement <8 x half> %{{.*}}, i64 0
+  // CHECK-NEXT: bitcast i8 %{{.*}} to <8 x i1>
+  // CHECK-NEXT: extractelement <8 x i1> %{{.*}}, i64 0
+  // CHECK-NEXT: [[SEL:%.*]] = select i1 %{{.*}}, half [[A]], half [[B]]
+  // CHECK-NEXT: insertelement <8 x half> [[VEC]], half [[SEL]], i64 0
+  return _mm_mask_move_sh(__W, __U, __A, __B);
+}
+
+__m128h test_mm_maskz_move_sh(__mmask8 __U, __m128h __A, __m128h __B) {
+  // CHECK-LABEL: @test_mm_maskz_move_sh
+  // CHECK: [[EXT:%.*]] = extractelement <8 x half> %{{.*}}, i32 0
+  // CHECK: insertelement <8 x half> %{{.*}}, half [[EXT]], i32 0
+  // CHECK: [[A:%.*]] = extractelement <8 x half> [[VEC:%.*]], i64 0
+  // CHECK-NEXT: [[B:%.*]] = extractelement <8 x half> %{{.*}}, i64 0
+  // CHECK-NEXT: bitcast i8 %{{.*}} to <8 x i1>
+  // CHECK-NEXT: extractelement <8 x i1> %{{.*}}, i64 0
+  // CHECK-NEXT: [[SEL:%.*]] = select i1 %{{.*}}, half [[A]], half [[B]]
+  // CHECK-NEXT: insertelement <8 x half> [[VEC]], half [[SEL]], i64 0
+  return _mm_maskz_move_sh(__U, __A, __B);
+}
+
+short test_mm_cvtsi128_si16(__m128i A) {
+  // CHECK-LABEL: test_mm_cvtsi128_si16
+  // CHECK: extractelement <8 x i16> %{{.*}}, i32 0
+  return _mm_cvtsi128_si16(A);
+}
+
+__m128i test_mm_cvtsi16_si128(short A) {
+  // CHECK-LABEL: test_mm_cvtsi16_si128
+  // CHECK: insertelement <8 x i16> undef, i16 %{{.*}}, i32 0
+  // CHECK: insertelement <8 x i16> %{{.*}}, i16 0, i32 1
+  // CHECK: insertelement <8 x i16> %{{.*}}, i16 0, i32 2
+  // CHECK: insertelement <8 x i16> %{{.*}}, i16 0, i32 3
+  return _mm_cvtsi16_si128(A);
+}
+
+__m512h test_mm512_mask_blend_ph(__mmask32 __U, __m512h __A, __m512h __W) {
+  // CHECK-LABEL: @test_mm512_mask_blend_ph
+  // CHECK:  %{{.*}} = bitcast i32 %{{.*}} to <32 x i1>
+  // CHECK:  %{{.*}} = select <32 x i1> %{{.*}}, <32 x half> %{{.*}}, <32 x half> %{{.*}}
+  return _mm512_mask_blend_ph(__U, __A, __W);
+}
+
+__m512h test_mm512_permutex2var_ph(__m512h __A, __m512i __I, __m512h __B) {
+  // CHECK-LABEL: @test_mm512_permutex2var_ph
+  // CHECK:  %{{.*}} = bitcast <32 x half> %{{.*}} to <32 x i16>
+  // CHECK:  %{{.*}} = bitcast <8 x i64> %{{.*}} to <32 x i16>
+  // CHECK:  %{{.*}} = bitcast <32 x half> %{{.*}} to <32 x i16>
+  // CHECK:  %{{.*}} = call <32 x i16> @llvm.x86.avx512.vpermi2var.hi.512(<32 x i16> %{{.*}}, <32 x i16> %{{.*}}, <32 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <32 x i16> %{{.*}} to <32 x half>
+  return _mm512_permutex2var_ph(__A, __I, __B);
+}
+
+__m512h test_mm512_permutexvar_epi16(__m512i __A, __m512h __B) {
+  // CHECK-LABEL: @test_mm512_permutexvar_epi16
+  // CHECK:  %{{.*}} = bitcast <32 x half> %{{.*}} to <32 x i16>
+  // CHECK:  %{{.*}} = bitcast <8 x i64> %{{.*}} to <32 x i16>
+  // CHECK:  %{{.*}} = call <32 x i16> @llvm.x86.avx512.permvar.hi.512(<32 x i16> %{{.*}}, <32 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <32 x i16> %{{.*}} to <32 x half>
+  return _mm512_permutexvar_ph(__A, __B);
+}

clang/test/CodeGen/X86/avx512fp16-complex-abi.c

+// RUN: %clang_cc1 -triple i386-unknown-linux-gnu -target-feature +avx512fp16 -emit-llvm -o - %s | FileCheck %s -check-prefix=CHECK
+
+// Return value should be passed in <2 x half> so the backend will use xmm0
+_Complex _Float16 f16(_Complex _Float16 A, _Complex _Float16 B) {
+  // CHECK-LABEL: define{{.*}}<2 x half> @f16({ half, half }* byval({ half, half }) align 4 %{{.*}}, { half, half }* byval({ half, half }) align 4 %{{.*}})
+  return A + B;
+}

clang/test/CodeGen/X86/avx512fp16-complex.c

+// RUN: %clang_cc1 %s -O0 -fno-experimental-new-pass-manager -emit-llvm -triple x86_64-unknown-unknown -target-feature +avx512fp16 -o - | FileCheck %s --check-prefix=X86
+
+_Float16 _Complex add_half_rr(_Float16 a, _Float16 b) {
+  // X86-LABEL: @add_half_rr(
+  // X86: fadd
+  // X86-NOT: fadd
+  // X86: ret
+  return a + b;
+}
+_Float16 _Complex add_half_cr(_Float16 _Complex a, _Float16 b) {
+  // X86-LABEL: @add_half_cr(
+  // X86: fadd
+  // X86-NOT: fadd
+  // X86: ret
+  return a + b;
+}
+_Float16 _Complex add_half_rc(_Float16 a, _Float16 _Complex b) {
+  // X86-LABEL: @add_half_rc(
+  // X86: fadd
+  // X86-NOT: fadd
+  // X86: ret
+  return a + b;
+}
+_Float16 _Complex add_half_cc(_Float16 _Complex a, _Float16 _Complex b) {
+  // X86-LABEL: @add_half_cc(
+  // X86: fadd
+  // X86: fadd
+  // X86-NOT: fadd
+  // X86: ret
+  return a + b;
+}
+
+_Float16 _Complex sub_half_rr(_Float16 a, _Float16 b) {
+  // X86-LABEL: @sub_half_rr(
+  // X86: fsub
+  // X86-NOT: fsub
+  // X86: ret
+  return a - b;
+}
+_Float16 _Complex sub_half_cr(_Float16 _Complex a, _Float16 b) {
+  // X86-LABEL: @sub_half_cr(
+  // X86: fsub
+  // X86-NOT: fsub
+  // X86: ret
+  return a - b;
+}
+_Float16 _Complex sub_half_rc(_Float16 a, _Float16 _Complex b) {
+  // X86-LABEL: @sub_half_rc(
+  // X86: fsub
+  // X86: fneg
+  // X86-NOT: fsub
+  // X86: ret
+  return a - b;
+}
+_Float16 _Complex sub_half_cc(_Float16 _Complex a, _Float16 _Complex b) {
+  // X86-LABEL: @sub_half_cc(
+  // X86: fsub
+  // X86: fsub
+  // X86-NOT: fsub
+  // X86: ret
+  return a - b;
+}
+
+_Float16 _Complex mul_half_rr(_Float16 a, _Float16 b) {
+  // X86-LABEL: @mul_half_rr(
+  // X86: fmul
+  // X86-NOT: fmul
+  // X86: ret
+  return a * b;
+}
+_Float16 _Complex mul_half_cr(_Float16 _Complex a, _Float16 b) {
+  // X86-LABEL: @mul_half_cr(
+  // X86: fmul
+  // X86: fmul
+  // X86-NOT: fmul
+  // X86: ret
+  return a * b;
+}
+_Float16 _Complex mul_half_rc(_Float16 a, _Float16 _Complex b) {
+  // X86-LABEL: @mul_half_rc(
+  // X86: fmul
+  // X86: fmul
+  // X86-NOT: fmul
+  // X86: ret
+  return a * b;
+}
+_Float16 _Complex mul_half_cc(_Float16 _Complex a, _Float16 _Complex b) {
+  // X86-LABEL: @mul_half_cc(
+  // X86: %[[AC:[^ ]+]] = fmul
+  // X86: %[[BD:[^ ]+]] = fmul
+  // X86: %[[AD:[^ ]+]] = fmul
+  // X86: %[[BC:[^ ]+]] = fmul
+  // X86: %[[RR:[^ ]+]] = fsub half %[[AC]], %[[BD]]
+  // X86: %[[RI:[^ ]+]] = fadd half
+  // X86-DAG: %[[AD]]
+  // X86-DAG: ,
+  // X86-DAG: %[[BC]]
+  // X86: fcmp uno half %[[RR]]
+  // X86: fcmp uno half %[[RI]]
+  // X86: call {{.*}} @__mulhc3(
+  // X86: ret
+  return a * b;
+}
+
+_Float16 _Complex div_half_rr(_Float16 a, _Float16 b) {
+  // X86-LABEL: @div_half_rr(
+  // X86: fdiv
+  // X86-NOT: fdiv
+  // X86: ret
+  return a / b;
+}
+_Float16 _Complex div_half_cr(_Float16 _Complex a, _Float16 b) {
+  // X86-LABEL: @div_half_cr(
+  // X86: fdiv
+  // X86: fdiv
+  // X86-NOT: fdiv
+  // X86: ret
+  return a / b;
+}
+_Float16 _Complex div_half_rc(_Float16 a, _Float16 _Complex b) {
+  // X86-LABEL: @div_half_rc(
+  // X86-NOT: fdiv
+  // X86: call {{.*}} @__divhc3(
+  // X86: ret
+  return a / b;
+}
+_Float16 _Complex div_half_cc(_Float16 _Complex a, _Float16 _Complex b) {
+  // X86-LABEL: @div_half_cc(
+  // X86-NOT: fdiv
+  // X86: call {{.*}} @__divhc3(
+  // X86: ret
+  return a / b;
+}

clang/test/CodeGen/X86/avx512vlfp16-builtins.c

+// RUN: %clang_cc1 -flax-vector-conversions=none -ffreestanding %s -triple=x86_64-unknown-unknown -target-feature +avx512vl -target-feature +avx512fp16 -emit-llvm -o - -Wall -Werror | FileCheck %s
+
+#include <immintrin.h>
+
+_Float16 test_mm_cvtsh_h(__m128h __A) {
+  // CHECK-LABEL: @test_mm_cvtsh_h
+  // CHECK: extractelement <8 x half> %{{.*}}, i32 0
+  return _mm_cvtsh_h(__A);
+}
+
+_Float16 test_mm256_cvtsh_h(__m256h __A) {
+  // CHECK-LABEL: @test_mm256_cvtsh_h
+  // CHECK: extractelement <16 x half> %{{.*}}, i32 0
+  return _mm256_cvtsh_h(__A);
+}
+
+__m128h test_mm_set_sh(_Float16 __h) {
+  // CHECK-LABEL: @test_mm_set_sh
+  // CHECK: insertelement <8 x half> {{.*}}, i32 0
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 1
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 2
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 3
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 4
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 5
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 6
+  // CHECK: insertelement <8 x half> %{{.*}}, half 0xH0000, i32 7
+  return _mm_set_sh(__h);
+}
+
+__m128h test_mm_set1_ph(_Float16 h) {
+  // CHECK-LABEL: @test_mm_set1_ph
+  // CHECK: insertelement <8 x half> {{.*}}, i32 0
+  // CHECK: insertelement <8 x half> {{.*}}, i32 1
+  // CHECK: insertelement <8 x half> {{.*}}, i32 2
+  // CHECK: insertelement <8 x half> {{.*}}, i32 3
+  // CHECK: insertelement <8 x half> {{.*}}, i32 4
+  // CHECK: insertelement <8 x half> {{.*}}, i32 5
+  // CHECK: insertelement <8 x half> {{.*}}, i32 6
+  // CHECK: insertelement <8 x half> {{.*}}, i32 7
+  return _mm_set1_ph(h);
+}
+
+__m256h test_mm256_set1_ph(_Float16 h) {
+  // CHECK-LABEL: @test_mm256_set1_ph
+  // CHECK: insertelement <16 x half> {{.*}}, i32 0
+  // CHECK: insertelement <16 x half> {{.*}}, i32 1
+  // CHECK: insertelement <16 x half> {{.*}}, i32 2
+  // CHECK: insertelement <16 x half> {{.*}}, i32 3
+  // CHECK: insertelement <16 x half> {{.*}}, i32 4
+  // CHECK: insertelement <16 x half> {{.*}}, i32 5
+  // CHECK: insertelement <16 x half> {{.*}}, i32 6
+  // CHECK: insertelement <16 x half> {{.*}}, i32 7
+  // CHECK: insertelement <16 x half> {{.*}}, i32 8
+  // CHECK: insertelement <16 x half> {{.*}}, i32 9
+  // CHECK: insertelement <16 x half> {{.*}}, i32 10
+  // CHECK: insertelement <16 x half> {{.*}}, i32 11
+  // CHECK: insertelement <16 x half> {{.*}}, i32 12
+  // CHECK: insertelement <16 x half> {{.*}}, i32 13
+  // CHECK: insertelement <16 x half> {{.*}}, i32 14
+  // CHECK: insertelement <16 x half> {{.*}}, i32 15
+  return _mm256_set1_ph(h);
+}
+
+__m128h test_mm_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                       _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8) {
+  // CHECK-LABEL: @test_mm_set_ph
+  // CHECK: insertelement <8 x half> {{.*}}, i32 0
+  // CHECK: insertelement <8 x half> {{.*}}, i32 1
+  // CHECK: insertelement <8 x half> {{.*}}, i32 2
+  // CHECK: insertelement <8 x half> {{.*}}, i32 3
+  // CHECK: insertelement <8 x half> {{.*}}, i32 4
+  // CHECK: insertelement <8 x half> {{.*}}, i32 5
+  // CHECK: insertelement <8 x half> {{.*}}, i32 6
+  // CHECK: insertelement <8 x half> {{.*}}, i32 7
+  return _mm_set_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8);
+}
+
+__m256h test_mm256_set_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                          _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+                          _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+                          _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16) {
+  // CHECK-LABEL: @test_mm256_set_ph
+  // CHECK: insertelement <16 x half> {{.*}}, i32 0
+  // CHECK: insertelement <16 x half> {{.*}}, i32 1
+  // CHECK: insertelement <16 x half> {{.*}}, i32 2
+  // CHECK: insertelement <16 x half> {{.*}}, i32 3
+  // CHECK: insertelement <16 x half> {{.*}}, i32 4
+  // CHECK: insertelement <16 x half> {{.*}}, i32 5
+  // CHECK: insertelement <16 x half> {{.*}}, i32 6
+  // CHECK: insertelement <16 x half> {{.*}}, i32 7
+  // CHECK: insertelement <16 x half> {{.*}}, i32 8
+  // CHECK: insertelement <16 x half> {{.*}}, i32 9
+  // CHECK: insertelement <16 x half> {{.*}}, i32 10
+  // CHECK: insertelement <16 x half> {{.*}}, i32 11
+  // CHECK: insertelement <16 x half> {{.*}}, i32 12
+  // CHECK: insertelement <16 x half> {{.*}}, i32 13
+  // CHECK: insertelement <16 x half> {{.*}}, i32 14
+  // CHECK: insertelement <16 x half> {{.*}}, i32 15
+  return _mm256_set_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8,
+                       __h9, __h10, __h11, __h12, __h13, __h14, __h15, __h16);
+}
+
+__m128h test_mm_setr_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                        _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8) {
+  // CHECK-LABEL: @test_mm_setr_ph
+  // CHECK: insertelement <8 x half> {{.*}}, i32 0
+  // CHECK: insertelement <8 x half> {{.*}}, i32 1
+  // CHECK: insertelement <8 x half> {{.*}}, i32 2
+  // CHECK: insertelement <8 x half> {{.*}}, i32 3
+  // CHECK: insertelement <8 x half> {{.*}}, i32 4
+  // CHECK: insertelement <8 x half> {{.*}}, i32 5
+  // CHECK: insertelement <8 x half> {{.*}}, i32 6
+  // CHECK: insertelement <8 x half> {{.*}}, i32 7
+  return _mm_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8);
+}
+
+__m256h test_mm256_setr_ph(_Float16 __h1, _Float16 __h2, _Float16 __h3, _Float16 __h4,
+                           _Float16 __h5, _Float16 __h6, _Float16 __h7, _Float16 __h8,
+                           _Float16 __h9, _Float16 __h10, _Float16 __h11, _Float16 __h12,
+                           _Float16 __h13, _Float16 __h14, _Float16 __h15, _Float16 __h16) {
+  // CHECK-LABEL: @test_mm256_setr_ph
+  // CHECK: insertelement <16 x half> {{.*}}, i32 0
+  // CHECK: insertelement <16 x half> {{.*}}, i32 1
+  // CHECK: insertelement <16 x half> {{.*}}, i32 2
+  // CHECK: insertelement <16 x half> {{.*}}, i32 3
+  // CHECK: insertelement <16 x half> {{.*}}, i32 4
+  // CHECK: insertelement <16 x half> {{.*}}, i32 5
+  // CHECK: insertelement <16 x half> {{.*}}, i32 6
+  // CHECK: insertelement <16 x half> {{.*}}, i32 7
+  // CHECK: insertelement <16 x half> {{.*}}, i32 8
+  // CHECK: insertelement <16 x half> {{.*}}, i32 9
+  // CHECK: insertelement <16 x half> {{.*}}, i32 10
+  // CHECK: insertelement <16 x half> {{.*}}, i32 11
+  // CHECK: insertelement <16 x half> {{.*}}, i32 12
+  // CHECK: insertelement <16 x half> {{.*}}, i32 13
+  // CHECK: insertelement <16 x half> {{.*}}, i32 14
+  // CHECK: insertelement <16 x half> {{.*}}, i32 15
+  return _mm256_setr_ph(__h1, __h2, __h3, __h4, __h5, __h6, __h7, __h8,
+                        __h9, __h10, __h11, __h12, __h13, __h14, __h15, __h16);
+}
+
+__m128h test_mm_abs_ph(__m128h a) {
+  // CHECK-LABEL: @test_mm_abs_ph
+  // CHECK: and <4 x i32>
+  return _mm_abs_ph(a);
+}
+
+__m256h test_mm256_abs_ph(__m256h a) {
+  // CHECK-LABEL: @test_mm256_abs_ph
+  // CHECK: and <8 x i32>
+  return _mm256_abs_ph(a);
+}
+
+__m128h test_mm_mask_blend_ph(__mmask8 __U, __m128h __A, __m128h __W) {
+  // CHECK-LABEL: @test_mm_mask_blend_ph
+  // CHECK:  %{{.*}} = bitcast i8 %{{.*}} to <8 x i1>
+  // CHECK:  %{{.*}} = select <8 x i1> %{{.*}}, <8 x half> %{{.*}}, <8 x half> %{{.*}}
+  return _mm_mask_blend_ph(__U, __A, __W);
+}
+
+__m256h test_mm256_mask_blend_ph(__mmask16 __U, __m256h __A, __m256h __W) {
+  // CHECK-LABEL: @test_mm256_mask_blend_ph
+  // CHECK:  %{{.*}} = bitcast i16 %{{.*}} to <16 x i1>
+  // CHECK:  %{{.*}} = select <16 x i1> %{{.*}}, <16 x half> %{{.*}}, <16 x half> %{{.*}}
+  return _mm256_mask_blend_ph(__U, __A, __W);
+}
+
+__m128h test_mm_permutex2var_ph(__m128h __A, __m128i __I, __m128h __B) {
+  // CHECK-LABEL: @test_mm_permutex2var_ph
+  // CHECK:  %{{.*}} = bitcast <8 x half> %{{.*}} to <8 x i16>
+  // CHECK:  %{{.*}} = bitcast <2 x i64> %{{.*}} to <8 x i16>
+  // CHECK:  %{{.*}} = bitcast <8 x half> %{{.*}} to <8 x i16>
+  // CHECK:  %{{.*}} = call <8 x i16> @llvm.x86.avx512.vpermi2var.hi.128(<8 x i16> %{{.*}}, <8 x i16> %{{.*}}, <8 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <8 x i16> %{{.*}} to <8 x half>
+  return _mm_permutex2var_ph(__A, __I, __B);
+}
+
+__m256h test_mm256_permutex2var_ph(__m256h __A, __m256i __I, __m256h __B) {
+  // CHECK-LABEL: @test_mm256_permutex2var_ph
+  // CHECK:  %{{.*}} = bitcast <16 x half> %{{.*}} to <16 x i16>
+  // CHECK:  %{{.*}} = bitcast <4 x i64> %{{.*}} to <16 x i16>
+  // CHECK:  %{{.*}} = bitcast <16 x half> %{{.*}} to <16 x i16>
+  // CHECK:  %{{.*}} = call <16 x i16> @llvm.x86.avx512.vpermi2var.hi.256(<16 x i16> %{{.*}}, <16 x i16> %{{.*}}, <16 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <16 x i16> %{{.*}} to <16 x half>
+  return _mm256_permutex2var_ph(__A, __I, __B);
+}
+
+__m128h test_mm_permutexvar_ph(__m128i __A, __m128h __B) {
+  // CHECK-LABEL: @test_mm_permutexvar_ph
+  // CHECK:  %{{.*}} = bitcast <8 x half> %{{.*}} to <8 x i16>
+  // CHECK:  %{{.*}} = bitcast <2 x i64> %{{.*}} to <8 x i16>
+  // CHECK:  %{{.*}} = call <8 x i16> @llvm.x86.avx512.permvar.hi.128(<8 x i16> %{{.*}}, <8 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <8 x i16> %{{.*}} to <8 x half>
+  return _mm_permutexvar_ph(__A, __B);
+}
+
+__m256h test_mm256_permutexvar_ph(__m256i __A, __m256h __B) {
+  // CHECK-LABEL: @test_mm256_permutexvar_ph
+  // CHECK:  %{{.*}} = bitcast <16 x half> %{{.*}} to <16 x i16>
+  // CHECK:  %{{.*}} = bitcast <4 x i64> %{{.*}} to <16 x i16>
+  // CHECK:  %{{.*}} = call <16 x i16> @llvm.x86.avx512.permvar.hi.256(<16 x i16> %{{.*}}, <16 x i16> %{{.*}})
+  // CHECK:  %{{.*}} = bitcast <16 x i16> %{{.*}} to <16 x half>
+  return _mm256_permutexvar_ph(__A, __B);
+}

clang/test/CodeGen/attr-target-x86.c

 // CHECK: qax{{.*}} #5
 // CHECK: qq{{.*}} #6
 // CHECK: lake{{.*}} #7
 // CHECK: use_before_def{{.*}} #7
 // CHECK: walrus{{.*}} #8
 // CHECK: #0 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87" "tune-cpu"="i686"
 // CHECK: #1 = {{.*}}"target-cpu"="ivybridge" "target-features"="+avx,+cx16,+cx8,+f16c,+fsgsbase,+fxsr,+mmx,+pclmul,+popcnt,+rdrnd,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt"
 // CHECK-NOT: tune-cpu
-// CHECK: #2 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-aes,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
+// CHECK: #2 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-aes,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512fp16,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
 // CHECK: #3 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+mmx,+popcnt,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87" "tune-cpu"="i686"
-// CHECK: #4 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-sse4.1,-sse4.2,-vaes,-vpclmulqdq,-xop" "tune-cpu"="i686"
+// CHECK: #4 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512fp16,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-sse4.1,-sse4.2,-vaes,-vpclmulqdq,-xop" "tune-cpu"="i686"
 // CHECK: #5 = {{.*}}"target-cpu"="ivybridge" "target-features"="+avx,+cx16,+cx8,+f16c,+fsgsbase,+fxsr,+mmx,+pclmul,+popcnt,+rdrnd,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt,-aes,-vaes"
 // CHECK-NOT: tune-cpu
 // CHECK: #6 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-3dnow,-3dnowa,-mmx"
 // CHECK: #7 = {{.*}}"target-cpu"="lakemont" "target-features"="+cx8,+mmx"
 // CHECK-NOT: tune-cpu
 // CHECK: #8 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87" "tune-cpu"="sandybridge"
 
 // CHECK: "target-cpu"="x86-64-v2"
 // CHECK-SAME: "target-features"="+cx16,+cx8,+fxsr,+mmx,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87"
 // CHECK: "target-cpu"="x86-64-v3"
 // CHECK-SAME: "target-features"="+avx,+avx2,+bmi,+bmi2,+cx16,+cx8,+f16c,+fma,+fxsr,+lzcnt,+mmx,+movbe,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"
 // CHECK: "target-cpu"="x86-64-v4"
 // CHECK-SAME: "target-features"="+avx,+avx2,+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl,+bmi,+bmi2,+cx16,+cx8,+f16c,+fma,+fxsr,+lzcnt,+mmx,+movbe,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"

clang/test/Driver/x86-target-features.c

 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-uintr %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-UINTR %s
 // UINTR: "-target-feature" "+uintr"
 // NO-UINTR: "-target-feature" "-uintr"
 
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mavxvnni %s -### -o %t.o 2>&1 | FileCheck --check-prefix=AVX-VNNI %s
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-avxvnni %s -### -o %t.o 2>&1 | FileCheck --check-prefix=NO-AVX-VNNI %s
 // AVX-VNNI: "-target-feature" "+avxvnni"
 // NO-AVX-VNNI: "-target-feature" "-avxvnni"
+
+// RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mavx512fp16 %s -### -o %t.o 2>&1 | FileCheck -check-prefix=AVX512FP16 %s
+// RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-avx512fp16 %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-AVX512FP16 %s
+// AVX512FP16: "-target-feature" "+avx512fp16"
+// NO-AVX512FP16: "-target-feature" "-avx512fp16"

clang/test/Preprocessor/predefined-arch-macros.c

 // CHECK_SPR_M32: #define __AMXINT8__ 1
 // CHECK_SPR_M32: #define __AMXTILE__ 1
 // CHECK_SPR_M32: #define __AVX2__ 1
 // CHECK_SPR_M32: #define __AVX512BF16__ 1
 // CHECK_SPR_M32: #define __AVX512BITALG__ 1
 // CHECK_SPR_M32: #define __AVX512BW__ 1
 // CHECK_SPR_M32: #define __AVX512CD__ 1
 // CHECK_SPR_M32: #define __AVX512DQ__ 1
+// CHECK_SPR_M32: #define __AVX512FP16__ 1
 // CHECK_SPR_M32: #define __AVX512F__ 1
 // CHECK_SPR_M32: #define __AVX512IFMA__ 1
 // CHECK_SPR_M32: #define __AVX512VBMI2__ 1
 // CHECK_SPR_M32: #define __AVX512VBMI__ 1
 // CHECK_SPR_M32: #define __AVX512VL__ 1
 // CHECK_SPR_M32: #define __AVX512VNNI__ 1
 // CHECK_SPR_M32: #define __AVX512VPOPCNTDQ__ 1
 // CHECK_SPR_M32: #define __AVXVNNI__ 1

 // CHECK_SPR_M64: #define __AMXINT8__ 1
 // CHECK_SPR_M64: #define __AMXTILE__ 1
 // CHECK_SPR_M64: #define __AVX2__ 1
 // CHECK_SPR_M64: #define __AVX512BF16__ 1
 // CHECK_SPR_M64: #define __AVX512BITALG__ 1
 // CHECK_SPR_M64: #define __AVX512BW__ 1
 // CHECK_SPR_M64: #define __AVX512CD__ 1
 // CHECK_SPR_M64: #define __AVX512DQ__ 1
+// CHECK_SPR_M64: #define __AVX512FP16__ 1
 // CHECK_SPR_M64: #define __AVX512F__ 1
 // CHECK_SPR_M64: #define __AVX512IFMA__ 1
 // CHECK_SPR_M64: #define __AVX512VBMI2__ 1
 // CHECK_SPR_M64: #define __AVX512VBMI__ 1
 // CHECK_SPR_M64: #define __AVX512VL__ 1
 // CHECK_SPR_M64: #define __AVX512VNNI__ 1
 // CHECK_SPR_M64: #define __AVX512VPOPCNTDQ__ 1
 // CHECK_SPR_M64: #define __AVXVNNI__ 1

clang/test/Preprocessor/x86_target_features.c

 // RUN: %clang -target i386-unknown-unknown -march=atom -mno-avxvnni -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=NOAVXVNNI %s
 
 // NOAVXVNNI-NOT: #define __AVXVNNI__ 1
 
 // RUN: %clang -target i386-unknown-unknown -march=atom -mavxvnni -mno-avx2 -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVXVNNINOAVX2 %s
 
 // AVXVNNINOAVX2-NOT: #define __AVX2__ 1
 // AVXVNNINOAVX2-NOT: #define __AVXVNNI__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavx512fp16 -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVX512FP16 %s
+
+// AVX512FP16: #define __AVX512BW__ 1
+// AVX512FP16: #define __AVX512DQ__ 1
+// AVX512FP16: #define __AVX512FP16__ 1
+// AVX512FP16: #define __AVX512VL__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavx512fp16 -mno-avx512vl -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVX512FP16NOAVX512VL %s
+
+// AVX512FP16NOAVX512VL-NOT: #define __AVX512FP16__ 1
+// AVX512FP16NOAVX512VL-NOT: #define __AVX512VL__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavx512fp16 -mno-avx512bw -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVX512FP16NOAVX512BW %s
+
+// AVX512FP16NOAVX512BW-NOT: #define __AVX512BW__ 1
+// AVX512FP16NOAVX512BW-NOT: #define __AVX512FP16__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavx512fp16 -mno-avx512dq -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVX512FP16NOAVX512DQ %s
+
+// AVX512FP16NOAVX512DQ-NOT: #define __AVX512DQ__ 1
+// AVX512FP16NOAVX512DQ-NOT: #define __AVX512FP16__ 1

clang/test/Sema/Float16.c

 // RUN: %clang_cc1 -fsyntax-only -verify -triple x86_64-linux-pc %s
+// RUN: %clang_cc1 -fsyntax-only -verify -triple x86_64-linux-pc -target-feature +avx512fp16 %s -DHAVE
 // RUN: %clang_cc1 -fsyntax-only -verify -triple spir-unknown-unknown %s -DHAVE
 // RUN: %clang_cc1 -fsyntax-only -verify -triple armv7a-linux-gnu %s -DHAVE
 // RUN: %clang_cc1 -fsyntax-only -verify -triple aarch64-linux-gnu %s -DHAVE
 
 #ifndef HAVE
 // expected-error@+2{{_Float16 is not supported on this target}}
 #endif // HAVE
 _Float16 f;
 
 #ifdef HAVE
-// FIXME: Should this be valid?
-_Complex _Float16 a; // expected-error {{'_Complex _Float16' is invalid}}
+// FIXME: Should this be invalid?
+_Complex _Float16 a;
 void builtin_complex() {
   _Float16 a = 0;
   (void)__builtin_complex(a, a); // expected-error {{'_Complex _Float16' is invalid}}
 }
 #endif

llvm/docs/ReleaseNotes.rst

 
 During this release ...
 
 Changes to the X86 Target
 -------------------------
 
 During this release ...
 
+* Support for ``AVX512-FP16`` instructions has been added.
+
 Changes to the AMDGPU Target
 -----------------------------
 
 During this release ...
 
 Changes to the AVR Target
 -----------------------------
 

llvm/include/llvm/IR/Intrinsics.td

 def llvm_v16i64_ty     : LLVMType<v16i64>;   // 16 x i64
 def llvm_v32i64_ty     : LLVMType<v32i64>;   // 32 x i64
 
 def llvm_v1i128_ty     : LLVMType<v1i128>;   //  1 x i128
 
 def llvm_v2f16_ty      : LLVMType<v2f16>;    //  2 x half (__fp16)
 def llvm_v4f16_ty      : LLVMType<v4f16>;    //  4 x half (__fp16)
 def llvm_v8f16_ty      : LLVMType<v8f16>;    //  8 x half (__fp16)
+def llvm_v16f16_ty     : LLVMType<v16f16>;   // 16 x half (__fp16)
+def llvm_v32f16_ty     : LLVMType<v32f16>;   // 32 x half (__fp16)
 def llvm_v2bf16_ty     : LLVMType<v2bf16>;   //  2 x bfloat (__bf16)
 def llvm_v4bf16_ty     : LLVMType<v4bf16>;   //  4 x bfloat (__bf16)
 def llvm_v8bf16_ty     : LLVMType<v8bf16>;   //  8 x bfloat (__bf16)
 def llvm_v1f32_ty      : LLVMType<v1f32>;    //  1 x float
 def llvm_v2f32_ty      : LLVMType<v2f32>;    //  2 x float
 def llvm_v4f32_ty      : LLVMType<v4f32>;    //  4 x float
 def llvm_v8f32_ty      : LLVMType<v8f32>;    //  8 x float
 def llvm_v16f32_ty     : LLVMType<v16f32>;   // 16 x float

llvm/include/llvm/IR/RuntimeLibcalls.def

 
 // Conversion
 HANDLE_LIBCALL(FPEXT_F32_PPCF128, "__gcc_stoq")
 HANDLE_LIBCALL(FPEXT_F64_PPCF128, "__gcc_dtoq")
 HANDLE_LIBCALL(FPEXT_F80_F128, "__extendxftf2")
 HANDLE_LIBCALL(FPEXT_F64_F128, "__extenddftf2")
 HANDLE_LIBCALL(FPEXT_F32_F128, "__extendsftf2")
 HANDLE_LIBCALL(FPEXT_F16_F128, "__extendhftf2")
+HANDLE_LIBCALL(FPEXT_F16_F80, "__extendhfxf2")
 HANDLE_LIBCALL(FPEXT_F32_F64, "__extendsfdf2")
 HANDLE_LIBCALL(FPEXT_F16_F64, "__extendhfdf2")
 HANDLE_LIBCALL(FPEXT_F16_F32, "__gnu_h2f_ieee")
 HANDLE_LIBCALL(FPROUND_F32_F16, "__gnu_f2h_ieee")
 HANDLE_LIBCALL(FPROUND_F64_F16, "__truncdfhf2")
 HANDLE_LIBCALL(FPROUND_F80_F16, "__truncxfhf2")
 HANDLE_LIBCALL(FPROUND_F128_F16, "__trunctfhf2")
 HANDLE_LIBCALL(FPROUND_PPCF128_F16, "__trunctfhf2")

llvm/include/llvm/Support/X86DisassemblerDecoderCommon.h

 //===-- X86DisassemblerDecoderCommon.h - Disassembler decoder ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #define ONEBYTE_SYM       x86DisassemblerOneByteOpcodes
 #define TWOBYTE_SYM       x86DisassemblerTwoByteOpcodes
 #define THREEBYTE38_SYM   x86DisassemblerThreeByte38Opcodes
 #define THREEBYTE3A_SYM   x86DisassemblerThreeByte3AOpcodes
 #define XOP8_MAP_SYM      x86DisassemblerXOP8Opcodes
 #define XOP9_MAP_SYM      x86DisassemblerXOP9Opcodes
 #define XOPA_MAP_SYM      x86DisassemblerXOPAOpcodes
 #define THREEDNOW_MAP_SYM x86Disassembler3DNowOpcodes
+#define MAP5_SYM          x86DisassemblerMap5Opcodes
+#define MAP6_SYM          x86DisassemblerMap6Opcodes
 
 #define INSTRUCTIONS_STR  "x86DisassemblerInstrSpecifiers"
 #define CONTEXTS_STR      "x86DisassemblerContexts"
 #define ONEBYTE_STR       "x86DisassemblerOneByteOpcodes"
 #define TWOBYTE_STR       "x86DisassemblerTwoByteOpcodes"
 #define THREEBYTE38_STR   "x86DisassemblerThreeByte38Opcodes"
 #define THREEBYTE3A_STR   "x86DisassemblerThreeByte3AOpcodes"
 #define XOP8_MAP_STR      "x86DisassemblerXOP8Opcodes"
 #define XOP9_MAP_STR      "x86DisassemblerXOP9Opcodes"
 #define XOPA_MAP_STR      "x86DisassemblerXOPAOpcodes"
 #define THREEDNOW_MAP_STR "x86Disassembler3DNowOpcodes"
+#define MAP5_STR          "x86DisassemblerMap5Opcodes"
+#define MAP6_STR          "x86DisassemblerMap6Opcodes"
 
 // Attributes of an instruction that must be known before the opcode can be
 // processed correctly.  Most of these indicate the presence of particular
 // prefixes, but ATTR_64BIT is simply an attribute of the decoding context.
 enum attributeBits {
   ATTR_NONE   = 0x00,
   ATTR_64BIT  = 0x1 << 0,
   ATTR_XS     = 0x1 << 1,
enum InstructionContext {
   INSTRUCTION_CONTEXTS
   IC_max
 };
 #undef ENUM_ENTRY
 
 // Opcode types, which determine which decode table to use, both in the Intel
 // manual and also for the decoder.
 enum OpcodeType {
   ONEBYTE       = 0,
   TWOBYTE       = 1,
   THREEBYTE_38  = 2,
   THREEBYTE_3A  = 3,
   XOP8_MAP      = 4,
   XOP9_MAP      = 5,
   XOPA_MAP      = 6,
-  THREEDNOW_MAP = 7
+  THREEDNOW_MAP = 7,
+  MAP5          = 8,
+  MAP6          = 9
 };
 
 // The following structs are used for the hierarchical decode table.  After
 // determining the instruction's class (i.e., which IC_* constant applies to
 // it), the decoder reads the opcode.  Some instructions require specific
 // values of the ModR/M byte, so the ModR/M byte indexes into the final table.
 //
 // If a ModR/M byte is not required, "required" is left unset, and the values

llvm/include/llvm/Support/X86TargetParser.def

 X86_FEATURE       (WAITPKG,         "waitpkg")
 X86_FEATURE       (WBNOINVD,        "wbnoinvd")
 X86_FEATURE       (X87,             "x87")
 X86_FEATURE       (XSAVE,           "xsave")
 X86_FEATURE       (XSAVEC,          "xsavec")
 X86_FEATURE       (XSAVEOPT,        "xsaveopt")
 X86_FEATURE       (XSAVES,          "xsaves")
 X86_FEATURE       (HRESET,          "hreset")
+X86_FEATURE       (AVX512FP16,      "avx512fp16")
 X86_FEATURE       (AVXVNNI,         "avxvnni")
 // These features aren't really CPU features, but the frontend can set them.
 X86_FEATURE       (RETPOLINE_EXTERNAL_THUNK,    "retpoline-external-thunk")
 X86_FEATURE       (RETPOLINE_INDIRECT_BRANCHES, "retpoline-indirect-branches")
 X86_FEATURE       (RETPOLINE_INDIRECT_CALLS,    "retpoline-indirect-calls")
 X86_FEATURE       (LVI_CFI,                     "lvi-cfi")
 X86_FEATURE       (LVI_LOAD_HARDENING,          "lvi-load-hardening")
 #undef X86_FEATURE_COMPAT
 #undef X86_FEATURE

llvm/include/llvm/Target/TargetSelectionDAG.td

 def extloadvi16 : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = i16;
 }
 def extloadvi32 : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = i32;
 }
+def extloadvf16 : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
+  let IsLoad = 1;
+  let ScalarMemoryVT = f16;
+}
 def extloadvf32 : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = f32;
 }
 def extloadvf64 : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = f64;
 }

llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp

 //===-------- LegalizeFloatTypes.cpp - Legalization of float types --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #ifndef NDEBUG
     dbgs() << "SoftenFloatOperand Op #" << OpNo << ": ";
     N->dump(&DAG); dbgs() << "\n";
 #endif
     llvm_unreachable("Do not know how to soften this operator's operand!");
 
   case ISD::BITCAST:     Res = SoftenFloatOp_BITCAST(N); break;
   case ISD::BR_CC:       Res = SoftenFloatOp_BR_CC(N); break;
+  case ISD::STRICT_FP_TO_FP16:
   case ISD::FP_TO_FP16:  // Same as FP_ROUND for softening purposes
   case ISD::STRICT_FP_ROUND:
   case ISD::FP_ROUND:    Res = SoftenFloatOp_FP_ROUND(N); break;
   case ISD::STRICT_FP_TO_SINT:
   case ISD::STRICT_FP_TO_UINT:
   case ISD::FP_TO_SINT:
   case ISD::FP_TO_UINT:  Res = SoftenFloatOp_FP_TO_XINT(N); break;
   case ISD::FP_TO_SINT_SAT:
SDValue DAGTypeLegalizer::SoftenFloatOp_BITCAST(SDNode *N) {
 
   return DAG.getNode(ISD::BITCAST, SDLoc(N), N->getValueType(0), Op0);
 }
 
 SDValue DAGTypeLegalizer::SoftenFloatOp_FP_ROUND(SDNode *N) {
   // We actually deal with the partially-softened FP_TO_FP16 node too, which
   // returns an i16 so doesn't meet the constraints necessary for FP_ROUND.
   assert(N->getOpcode() == ISD::FP_ROUND || N->getOpcode() == ISD::FP_TO_FP16 ||
+         N->getOpcode() == ISD::STRICT_FP_TO_FP16 ||
          N->getOpcode() == ISD::STRICT_FP_ROUND);
 
   bool IsStrict = N->isStrictFPOpcode();
   SDValue Op = N->getOperand(IsStrict ? 1 : 0);
   EVT SVT = Op.getValueType();
   EVT RVT = N->getValueType(0);
-  EVT FloatRVT = N->getOpcode() == ISD::FP_TO_FP16 ? MVT::f16 : RVT;
+  EVT FloatRVT = (N->getOpcode() == ISD::FP_TO_FP16 ||
+                  N->getOpcode() == ISD::STRICT_FP_TO_FP16)
+                     ? MVT::f16
+                     : RVT;
 
   RTLIB::Libcall LC = RTLIB::getFPROUND(SVT, FloatRVT);
   assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unsupported FP_ROUND libcall");
 
   SDValue Chain = IsStrict ? N->getOperand(0) : SDValue();
   Op = GetSoftenedFloat(Op);
   TargetLowering::MakeLibCallOptions CallOptions;
   CallOptions.setTypeListBeforeSoften(SVT, RVT, true);

llvm/lib/CodeGen/TargetLoweringBase.cpp

 //===- TargetLoweringBase.cpp - Implement the TargetLoweringBase class ----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 /// getFPEXT - Return the FPEXT_*_* value for the given types, or
 /// UNKNOWN_LIBCALL if there is none.
 RTLIB::Libcall RTLIB::getFPEXT(EVT OpVT, EVT RetVT) {
   if (OpVT == MVT::f16) {
     if (RetVT == MVT::f32)
       return FPEXT_F16_F32;
     if (RetVT == MVT::f64)
       return FPEXT_F16_F64;
+    if (RetVT == MVT::f80)
+      return FPEXT_F16_F80;
     if (RetVT == MVT::f128)
       return FPEXT_F16_F128;
   } else if (OpVT == MVT::f32) {
     if (RetVT == MVT::f64)
       return FPEXT_F32_F64;
     if (RetVT == MVT::f128)
       return FPEXT_F32_F128;
     if (RetVT == MVT::ppcf128)

llvm/lib/Support/Host.cpp

 //===-- Host.cpp - Implement OS Host Concept --------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
#endif
   // availability of specific pconfig leafs.
   // The target feature here only refers to the the first of these two.
   // Users might need to check for the availability of specific pconfig
   // leaves using cpuid, since that information is ignored while
   // detecting features using the "-march=native" flag.
   // For more info, see X86 ISA docs.
   Features["pconfig"] = HasLeaf7 && ((EDX >> 18) & 1);
   Features["amx-bf16"]   = HasLeaf7 && ((EDX >> 22) & 1) && HasAMXSave;
+  Features["avx512fp16"] = HasLeaf7 && ((EDX >> 23) & 1) && HasAVX512Save;
   Features["amx-tile"]   = HasLeaf7 && ((EDX >> 24) & 1) && HasAMXSave;
   Features["amx-int8"]   = HasLeaf7 && ((EDX >> 25) & 1) && HasAMXSave;
   bool HasLeaf7Subleaf1 =
       MaxLevel >= 7 && !getX86CpuIDAndInfoEx(0x7, 0x1, &EAX, &EBX, &ECX, &EDX);
   Features["avxvnni"]    = HasLeaf7Subleaf1 && ((EAX >> 4) & 1) && HasAVXSave;
   Features["avx512bf16"] = HasLeaf7Subleaf1 && ((EAX >> 5) & 1) && HasAVX512Save;
   Features["hreset"]     = HasLeaf7Subleaf1 && ((EAX >> 22) & 1);
 

llvm/lib/Support/X86TargetParser.cpp

 //===-- X86TargetParser - Parser for X86 features ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
constexpr FeatureBitset FeaturesICLClient =
     FeatureVAES | FeatureVPCLMULQDQ;
 constexpr FeatureBitset FeaturesRocketlake = FeaturesICLClient & ~FeatureSGX;
 constexpr FeatureBitset FeaturesICLServer =
     FeaturesICLClient | FeatureCLWB | FeaturePCONFIG | FeatureWBNOINVD;
 constexpr FeatureBitset FeaturesTigerlake =
     FeaturesICLClient | FeatureAVX512VP2INTERSECT | FeatureMOVDIR64B |
     FeatureCLWB | FeatureMOVDIRI | FeatureSHSTK | FeatureKL | FeatureWIDEKL;
 constexpr FeatureBitset FeaturesSapphireRapids =
-    FeaturesICLServer | FeatureAMX_TILE | FeatureAMX_INT8 | FeatureAMX_BF16 |
-    FeatureAVX512BF16 | FeatureAVX512VP2INTERSECT | FeatureCLDEMOTE |
-    FeatureENQCMD | FeatureMOVDIR64B | FeatureMOVDIRI | FeaturePTWRITE |
-    FeatureSERIALIZE | FeatureSHSTK | FeatureTSXLDTRK | FeatureUINTR |
-    FeatureWAITPKG | FeatureAVXVNNI;
+    FeatureAVX512FP16 | FeaturesICLServer | FeatureAMX_TILE | FeatureAMX_INT8 |
+    FeatureAMX_BF16 | FeatureAVX512BF16 | FeatureAVX512VP2INTERSECT |
+    FeatureCLDEMOTE | FeatureENQCMD | FeatureMOVDIR64B | FeatureMOVDIRI |
+    FeaturePTWRITE | FeatureSERIALIZE | FeatureSHSTK | FeatureTSXLDTRK |
+    FeatureUINTR | FeatureWAITPKG | FeatureAVXVNNI;
 
 // Intel Atom processors.
 // Bonnell has feature parity with Core2 and adds MOVBE.
 constexpr FeatureBitset FeaturesBonnell = FeaturesCore2 | FeatureMOVBE;
 // Silvermont has parity with Westmere and Bonnell plus PRFCHW and RDRND.
 constexpr FeatureBitset FeaturesSilvermont =
     FeaturesBonnell | FeaturesWestmere | FeaturePRFCHW | FeatureRDRND;
 constexpr FeatureBitset FeaturesGoldmont =

 constexpr FeatureBitset ImpliedFeaturesXOP = FeatureFMA4;
 
 // AMX Features
 constexpr FeatureBitset ImpliedFeaturesAMX_TILE = {};
 constexpr FeatureBitset ImpliedFeaturesAMX_BF16 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesAMX_INT8 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesHRESET = {};
 
+static constexpr FeatureBitset ImpliedFeaturesAVX512FP16 =
+    FeatureAVX512F | FeatureAVX512BW | FeatureAVX512DQ | FeatureAVX512VL;
 // Key Locker Features
 constexpr FeatureBitset ImpliedFeaturesKL = FeatureSSE2;
 constexpr FeatureBitset ImpliedFeaturesWIDEKL = FeatureKL;
 
 // AVXVNNI Features
 constexpr FeatureBitset ImpliedFeaturesAVXVNNI = FeatureAVX2;
 
 constexpr FeatureInfo FeatureInfos[X86::CPU_FEATURE_MAX] = {

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp

 //===-- X86AsmParser.cpp - Parse X86 assembly to MCInst instructions ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
if(getLexer().is(AsmToken::Integer)) {
       if (!BroadcastString.startswith("1to"))
         return TokError("Expected 1to<NUM> at this point");
       const char *BroadcastPrimitive =
           StringSwitch<const char *>(BroadcastString)
               .Case("1to2", "{1to2}")
               .Case("1to4", "{1to4}")
               .Case("1to8", "{1to8}")
               .Case("1to16", "{1to16}")
+              .Case("1to32", "{1to32}")
               .Default(nullptr);
       if (!BroadcastPrimitive)
         return TokError("Invalid memory broadcast primitive.");
       Parser.Lex(); // Eat trailing token of 1toN
       if (!getLexer().is(AsmToken::RCurly))
         return TokError("Expected } at this point");
       Parser.Lex();  // Eat "}"
       Operands.push_back(X86Operand::CreateToken(BroadcastPrimitive,

llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp

 //===-- X86Disassembler.cpp - Disassembler for x86 and x86_64 -------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
case XOP9_MAP:
     break;
   case XOPA_MAP:
     dec = &XOPA_MAP_SYM.opcodeDecisions[insnContext].modRMDecisions[opcode];
     break;
   case THREEDNOW_MAP:
     dec =
         &THREEDNOW_MAP_SYM.opcodeDecisions[insnContext].modRMDecisions[opcode];
     break;
+  case MAP5:
+    dec = &MAP5_SYM.opcodeDecisions[insnContext].modRMDecisions[opcode];
+    break;
+  case MAP6:
+    dec = &MAP6_SYM.opcodeDecisions[insnContext].modRMDecisions[opcode];
+    break;
   }
 
   switch (dec->modrm_type) {
   default:
     llvm_unreachable("Corrupt table!  Unknown modrm_type");
     return 0;
   case MODRM_ONEENTRY:
     return modRMTable[dec->instructionIDs];
if (byte == 0x62) {
     }
 
     if (peek(insn, byte2)) {
       LLVM_DEBUG(dbgs() << "Couldn't read third byte of EVEX prefix");
       return -1;
     }
 
     if ((insn->mode == MODE_64BIT || (byte1 & 0xc0) == 0xc0) &&
-        ((~byte1 & 0xc) == 0xc) && ((byte2 & 0x4) == 0x4)) {
+        ((~byte1 & 0x8) == 0x8) && ((byte2 & 0x4) == 0x4)) {
       insn->vectorExtensionType = TYPE_EVEX;
     } else {
       --insn->readerCursor; // unconsume byte1
       --insn->readerCursor; // unconsume byte
     }
 
     if (insn->vectorExtensionType == TYPE_EVEX) {
       insn->vectorExtensionPrefix[0] = byte;

 // Read the opcode (except the ModR/M byte in the case of extended or escape
 // opcodes).
 static bool readOpcode(struct InternalInstruction *insn) {
   uint8_t current;
   LLVM_DEBUG(dbgs() << "readOpcode()");
 
   insn->opcodeType = ONEBYTE;
   if (insn->vectorExtensionType == TYPE_EVEX) {
-    switch (mmFromEVEX2of4(insn->vectorExtensionPrefix[1])) {
+    switch (mmmFromEVEX2of4(insn->vectorExtensionPrefix[1])) {
     default:
       LLVM_DEBUG(
-          dbgs() << format("Unhandled mm field for instruction (0x%hhx)",
-                           mmFromEVEX2of4(insn->vectorExtensionPrefix[1])));
+          dbgs() << format("Unhandled mmm field for instruction (0x%hhx)",
+                           mmmFromEVEX2of4(insn->vectorExtensionPrefix[1])));
       return true;
     case VEX_LOB_0F:
       insn->opcodeType = TWOBYTE;
       return consume(insn, insn->opcode);
     case VEX_LOB_0F38:
       insn->opcodeType = THREEBYTE_38;
       return consume(insn, insn->opcode);
     case VEX_LOB_0F3A:
       insn->opcodeType = THREEBYTE_3A;
       return consume(insn, insn->opcode);
+    case VEX_LOB_MAP5:
+      insn->opcodeType = MAP5;
+      return consume(insn, insn->opcode);
+    case VEX_LOB_MAP6:
+      insn->opcodeType = MAP6;
+      return consume(insn, insn->opcode);
     }
   } else if (insn->vectorExtensionType == TYPE_VEX_3B) {
     switch (mmmmmFromVEX2of3(insn->vectorExtensionPrefix[1])) {
     default:
       LLVM_DEBUG(
           dbgs() << format("Unhandled m-mmmm field for instruction (0x%hhx)",
                            mmmmmFromVEX2of3(insn->vectorExtensionPrefix[1])));
       return true;
     case VEX_LOB_0F:
       insn->opcodeType = TWOBYTE;
       return consume(insn, insn->opcode);
     case VEX_LOB_0F38:
       insn->opcodeType = THREEBYTE_38;
       return consume(insn, insn->opcode);
     case VEX_LOB_0F3A:
       insn->opcodeType = THREEBYTE_3A;
       return consume(insn, insn->opcode);
+    case VEX_LOB_MAP5:
+      insn->opcodeType = MAP5;
+      return consume(insn, insn->opcode);
+    case VEX_LOB_MAP6:
+      insn->opcodeType = MAP6;
+      return consume(insn, insn->opcode);
     }
   } else if (insn->vectorExtensionType == TYPE_VEX_2B) {
     insn->opcodeType = TWOBYTE;
     return consume(insn, insn->opcode);
   } else if (insn->vectorExtensionType == TYPE_XOP) {
     switch (mmmmmFromXOP2of3(insn->vectorExtensionPrefix[1])) {
     default:
       LLVM_DEBUG(
case XOP9_MAP:
     decision = &XOP9_MAP_SYM;
     break;
   case XOPA_MAP:
     decision = &XOPA_MAP_SYM;
     break;
   case THREEDNOW_MAP:
     decision = &THREEDNOW_MAP_SYM;
     break;
+  case MAP5:
+    decision = &MAP5_SYM;
+    break;
+  case MAP6:
+    decision = &MAP6_SYM;
+    break;
   }
 
   if (decision->opcodeDecisions[insnCtx]
           .modRMDecisions[insn->opcode]
           .modrm_type != MODRM_ONEENTRY) {
     if (readModRM(insn))
       return -1;
     *instructionID =

llvm/lib/Target/X86/Disassembler/X86DisassemblerDecoder.h

 //===-- X86DisassemblerDecoderInternal.h - Disassembler decoder -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 #define rFromREX(rex)        (((rex) & 0x4) >> 2)
 #define xFromREX(rex)        (((rex) & 0x2) >> 1)
 #define bFromREX(rex)        ((rex) & 0x1)
 
 #define rFromEVEX2of4(evex)     (((~(evex)) & 0x80) >> 7)
 #define xFromEVEX2of4(evex)     (((~(evex)) & 0x40) >> 6)
 #define bFromEVEX2of4(evex)     (((~(evex)) & 0x20) >> 5)
 #define r2FromEVEX2of4(evex)    (((~(evex)) & 0x10) >> 4)
-#define mmFromEVEX2of4(evex)    ((evex) & 0x3)
+#define mmmFromEVEX2of4(evex)   ((evex) & 0x7)
 #define wFromEVEX3of4(evex)     (((evex) & 0x80) >> 7)
 #define vvvvFromEVEX3of4(evex)  (((~(evex)) & 0x78) >> 3)
 #define ppFromEVEX3of4(evex)    ((evex) & 0x3)
 #define zFromEVEX4of4(evex)     (((evex) & 0x80) >> 7)
 #define l2FromEVEX4of4(evex)    (((evex) & 0x40) >> 6)
 #define lFromEVEX4of4(evex)     (((evex) & 0x20) >> 5)
 #define bFromEVEX4of4(evex)     (((evex) & 0x10) >> 4)
 #define v2FromEVEX4of4(evex)    (((~evex) & 0x8) >> 3)
enum SegmentOverride {
   SEG_OVERRIDE_GS,
   SEG_OVERRIDE_max
 };
 
 /// Possible values for the VEX.m-mmmm field
 enum VEXLeadingOpcodeByte {
   VEX_LOB_0F = 0x1,
   VEX_LOB_0F38 = 0x2,
-  VEX_LOB_0F3A = 0x3
+  VEX_LOB_0F3A = 0x3,
+  VEX_LOB_MAP5 = 0x5,
+  VEX_LOB_MAP6 = 0x6
 };
 
 enum XOPMapSelect {
   XOP_MAP_SELECT_8 = 0x8,
   XOP_MAP_SELECT_9 = 0x9,
   XOP_MAP_SELECT_A = 0xA
 };
 

llvm/lib/Target/X86/MCTargetDesc/X86BaseInfo.h

 //===-- X86BaseInfo.h - Top level definitions for X86 -------- --*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum : uint64_t {
     //===------------------------------------------------------------------===//
     // Instruction encodings.  These are the standard/most common forms for X86
     // instructions.
     //
 
     // PseudoFrm - This represents an instruction that is a pseudo instruction
     // or one that has not been implemented yet.  It is illegal to code generate
     // it, but tolerated for intermediate implementation stages.
     Pseudo         = 0,
 
     /// Raw - This form is for instructions that don't have any operands, so
     /// they are just a fixed opcode value, like 'leave'.
     RawFrm         = 1,
 
     /// AddRegFrm - This form is used for instructions like 'push r32' that have
     /// their one register operand added to their opcode.
     AddRegFrm      = 2,
 
     /// RawFrmMemOffs - This form is for instructions that store an absolute
     /// memory offset as an immediate with a possible segment override.
     RawFrmMemOffs  = 3,
 
     /// RawFrmSrc - This form is for instructions that use the source index
     /// register SI/ESI/RSI with a possible segment override.
     RawFrmSrc      = 4,
 
     /// RawFrmDst - This form is for instructions that use the destination index
     /// register DI/EDI/RDI.
     RawFrmDst      = 5,
 
     /// RawFrmDstSrc - This form is for instructions that use the source index
     /// register SI/ESI/RSI with a possible segment override, and also the
     /// destination index register DI/EDI/RDI.
     RawFrmDstSrc   = 6,
 
     /// RawFrmImm8 - This is used for the ENTER instruction, which has two
     /// immediates, the first of which is a 16-bit immediate (specified by
     /// the imm encoding) and the second is a 8-bit fixed value.
     RawFrmImm8 = 7,
 
     /// RawFrmImm16 - This is used for CALL FAR instructions, which have two
     /// immediates, the first of which is a 16 or 32-bit immediate (specified by
enum : uint64_t {
     /// a Mod/RM byte, and use the middle field to hold extended opcode
     /// information.  In the intel manual these are represented as /0, /1, ...
     ///
 
     // Instructions operate on a register Reg/Opcode operand not the r/m field.
     MRMr0 = 21,
 
     /// MRMSrcMem - But force to use the SIB field.
     MRMSrcMemFSIB  = 22,
 
     /// MRMDestMem - But force to use the SIB field.
     MRMDestMemFSIB = 23,
 
     /// MRMDestMem - This form is used for instructions that use the Mod/RM byte
     /// to specify a destination, which in this case is memory.
     ///
     MRMDestMem     = 24,
 
     /// MRMSrcMem - This form is used for instructions that use the Mod/RM byte
     /// to specify a source, which in this case is memory.
     ///
     MRMSrcMem      = 25,
 
     /// MRMSrcMem4VOp3 - This form is used for instructions that encode
     /// operand 3 with VEX.VVVV and load from memory.
     ///
     MRMSrcMem4VOp3 = 26,
 
     /// MRMSrcMemOp4 - This form is used for instructions that use the Mod/RM
     /// byte to specify the fourth source, which in this case is memory.
     ///
     MRMSrcMemOp4   = 27,
 
     /// MRMSrcMemCC - This form is used for instructions that use the Mod/RM
     /// byte to specify the operands and also encodes a condition code.
     ///
     MRMSrcMemCC    = 28,
 
     /// MRMXm - This form is used for instructions that use the Mod/RM byte
     /// to specify a memory source, but doesn't use the middle field. And has
     /// a condition code.
     ///
     MRMXmCC = 30,
 
     /// MRMXm - This form is used for instructions that use the Mod/RM byte
     /// to specify a memory source, but doesn't use the middle field.
     ///
     MRMXm = 31,
 
     // Next, instructions that operate on a memory r/m operand...
     MRM0m = 32,  MRM1m = 33,  MRM2m = 34,  MRM3m = 35, // Format /0 /1 /2 /3
     MRM4m = 36,  MRM5m = 37,  MRM6m = 38,  MRM7m = 39, // Format /4 /5 /6 /7
 
     /// MRMDestReg - This form is used for instructions that use the Mod/RM byte
     /// to specify a destination, which in this case is a register.
     ///
     MRMDestReg     = 40,
 
     /// MRMSrcReg - This form is used for instructions that use the Mod/RM byte
     /// to specify a source, which in this case is a register.
     ///
     MRMSrcReg      = 41,
 
     /// MRMSrcReg4VOp3 - This form is used for instructions that encode
     /// operand 3 with VEX.VVVV and do not load from memory.
     ///
     MRMSrcReg4VOp3 = 42,
 
     /// MRMSrcRegOp4 - This form is used for instructions that use the Mod/RM
     /// byte to specify the fourth source, which in this case is a register.
     ///
     MRMSrcRegOp4   = 43,
 
     /// MRMSrcRegCC - This form is used for instructions that use the Mod/RM
     /// byte to specify the operands and also encodes a condition code
     ///
     MRMSrcRegCC    = 44,
 
     /// MRMXCCr - This form is used for instructions that use the Mod/RM byte
     /// to specify a register source, but doesn't use the middle field. And has
     /// a condition code.
     ///
     MRMXrCC = 46,
 
     /// MRMXr - This form is used for instructions that use the Mod/RM byte
     /// to specify a register source, but doesn't use the middle field.
     ///
     MRMXr = 47,
 
     // Instructions that operate on a register r/m operand...
     MRM0r = 48,  MRM1r = 49,  MRM2r = 50,  MRM3r = 51, // Format /0 /1 /2 /3
     MRM4r = 52,  MRM5r = 53,  MRM6r = 54,  MRM7r = 55, // Format /4 /5 /6 /7
 
     // Instructions that operate that have mod=11 and an opcode but ignore r/m.
     MRM0X = 56,  MRM1X = 57,  MRM2X = 58,  MRM3X = 59, // Format /0 /1 /2 /3
     MRM4X = 60,  MRM5X = 61,  MRM6X = 62,  MRM7X = 63, // Format /4 /5 /6 /7
 
     /// MRM_XX - A mod/rm byte of exactly 0xXX.
     MRM_C0 = 64,  MRM_C1 = 65,  MRM_C2 = 66,  MRM_C3 = 67,
     MRM_C4 = 68,  MRM_C5 = 69,  MRM_C6 = 70,  MRM_C7 = 71,
     MRM_C8 = 72,  MRM_C9 = 73,  MRM_CA = 74,  MRM_CB = 75,
     MRM_CC = 76,  MRM_CD = 77,  MRM_CE = 78,  MRM_CF = 79,
     MRM_D0 = 80,  MRM_D1 = 81,  MRM_D2 = 82,  MRM_D3 = 83,
     MRM_D4 = 84,  MRM_D5 = 85,  MRM_D6 = 86,  MRM_D7 = 87,
     MRM_D8 = 88,  MRM_D9 = 89,  MRM_DA = 90,  MRM_DB = 91,
     MRM_DC = 92,  MRM_DD = 93,  MRM_DE = 94,  MRM_DF = 95,
     MRM_E0 = 96,  MRM_E1 = 97,  MRM_E2 = 98,  MRM_E3 = 99,
enum : uint64_t {
 
     // OpSize - OpSizeFixed implies instruction never needs a 0x66 prefix.
     // OpSize16 means this is a 16-bit instruction and needs 0x66 prefix in
     // 32-bit mode. OpSize32 means this is a 32-bit instruction needs a 0x66
     // prefix in 16-bit mode.
     OpSizeShift = 7,
     OpSizeMask = 0x3 << OpSizeShift,
 
     OpSizeFixed  = 0 << OpSizeShift,
     OpSize16     = 1 << OpSizeShift,
     OpSize32     = 2 << OpSizeShift,
 
     // AsSize - AdSizeX implies this instruction determines its need of 0x67
     // prefix from a normal ModRM memory operand. The other types indicate that
     // an operand is encoded with a specific width and a prefix is needed if
     // it differs from the current mode.
     AdSizeShift = OpSizeShift + 2,
     AdSizeMask  = 0x3 << AdSizeShift,
 
     AdSizeX  = 0 << AdSizeShift,
     AdSize16 = 1 << AdSizeShift,
     AdSize32 = 2 << AdSizeShift,
     AdSize64 = 3 << AdSizeShift,
 
     //===------------------------------------------------------------------===//
     // OpPrefix - There are several prefix bytes that are used as opcode
     // extensions. These are 0x66, 0xF3, and 0xF2. If this field is 0 there is
     // no prefix.
     //
     OpPrefixShift = AdSizeShift + 2,
     OpPrefixMask  = 0x3 << OpPrefixShift,
 
     // PD - Prefix code for packed double precision vector floating point
     // operations performed in the SSE registers.
     PD = 1 << OpPrefixShift,
 
     // XS, XD - These prefix codes are for single and double precision scalar
     // floating point operations performed in the SSE registers.
     XS = 2 << OpPrefixShift,  XD = 3 << OpPrefixShift,
 
     //===------------------------------------------------------------------===//
     // OpMap - This field determines which opcode map this instruction
     // belongs to. i.e. one-byte, two-byte, 0x0f 0x38, 0x0f 0x3a, etc.
     //
     OpMapShift = OpPrefixShift + 2,
-    OpMapMask  = 0x7 << OpMapShift,
+    OpMapMask  = 0xF << OpMapShift,
 
     // OB - OneByte - Set if this instruction has a one byte opcode.
     OB = 0 << OpMapShift,
 
     // TB - TwoByte - Set if this instruction has a two byte opcode, which
     // starts with a 0x0F byte before the real opcode.
     TB = 1 << OpMapShift,
 
     // T8, TA - Prefix after the 0x0F prefix.
     T8 = 2 << OpMapShift,  TA = 3 << OpMapShift,
 
     // XOP8 - Prefix to include use of imm byte.
     XOP8 = 4 << OpMapShift,
 
     // XOP9 - Prefix to exclude use of imm byte.
     XOP9 = 5 << OpMapShift,
 
     // XOPA - Prefix to encode 0xA in VEX.MMMM of XOP instructions.
     XOPA = 6 << OpMapShift,
 
     /// ThreeDNow - This indicates that the instruction uses the
     /// wacky 0x0F 0x0F prefix for 3DNow! instructions.  The manual documents
     /// this as having a 0x0F prefix with a 0x0F opcode, and each instruction
     /// storing a classifier in the imm8 field.  To simplify our implementation,
     /// we handle this by storeing the classifier in the opcode field and using
     /// this flag to indicate that the encoder should do the wacky 3DNow! thing.
     ThreeDNow = 7 << OpMapShift,
 
+    // MAP5, MAP6 - Prefix after the 0x0F prefix.
+    T_MAP5 = 8 << OpMapShift,
+    T_MAP6 = 9 << OpMapShift,
+
     //===------------------------------------------------------------------===//
     // REX_W - REX prefixes are instruction prefixes used in 64-bit mode.
     // They are used to specify GPRs and SSE registers, 64-bit operand size,
     // etc. We only cares about REX.W and REX.R bits and only the former is
     // statically determined.
     //
-    REXShift    = OpMapShift + 3,
+    REXShift    = OpMapShift + 4,
     REX_W       = 1 << REXShift,
 
     //===------------------------------------------------------------------===//
     // This three-bit field describes the size of an immediate operand.  Zero is
     // unused so that we can tell if we forgot to set a value.
     ImmShift = REXShift + 1,
     ImmMask    = 15 << ImmShift,
     Imm8       = 1 << ImmShift,
     Imm8PCRel  = 2 << ImmShift,
     Imm8Reg    = 3 << ImmShift,
     Imm16      = 4 << ImmShift,
     Imm16PCRel = 5 << ImmShift,
     Imm32      = 6 << ImmShift,
     Imm32PCRel = 7 << ImmShift,
     Imm32S     = 8 << ImmShift,
     Imm64      = 9 << ImmShift,
 
     //===------------------------------------------------------------------===//
     // FP Instruction Classification...  Zero is non-fp instruction.
 
     // FPTypeMask - Mask for all of the FP types...
     FPTypeShift = ImmShift + 4,
     FPTypeMask  = 7 << FPTypeShift,
 
     // NotFP - The default, set for instructions that do not use FP registers.
     NotFP      = 0 << FPTypeShift,
 
     // ZeroArgFP - 0 arg FP instruction which implicitly pushes ST(0), f.e. fld0
     ZeroArgFP  = 1 << FPTypeShift,
 
     // OneArgFP - 1 arg FP instructions which implicitly read ST(0), such as fst
     OneArgFP   = 2 << FPTypeShift,
 
     // OneArgFPRW - 1 arg FP instruction which implicitly read ST(0) and write a
     // result back to ST(0).  For example, fcos, fsqrt, etc.
     //
     OneArgFPRW = 3 << FPTypeShift,
 
     // TwoArgFP - 2 arg FP instructions which implicitly read ST(0), and an
     // explicit argument, storing the result to either ST(0) or the implicit
     // argument.  For example: fadd, fsub, fmul, etc...
     TwoArgFP   = 4 << FPTypeShift,
 
     // CompareFP - 2 arg FP instructions which implicitly read ST(0) and an
     // explicit argument, but have no destination.  Example: fucom, fucomi, ...
     CompareFP  = 5 << FPTypeShift,
 
     // CondMovFP - "2 operand" floating point conditional move instructions.
     CondMovFP  = 6 << FPTypeShift,
 
     // SpecialFP - Special instruction forms.  Dispatch by opcode explicitly.
     SpecialFP  = 7 << FPTypeShift,
 
     // Lock prefix
     LOCKShift = FPTypeShift + 3,
     LOCK = 1 << LOCKShift,
 
     // REP prefix
     REPShift = LOCKShift + 1,
     REP = 1 << REPShift,
enum : uint64_t {
 
     // VEX_EVEX - Specifies that this instruction use EVEX form which provides
     // syntax support up to 32 512-bit register operands and up to 7 16-bit
     // mask operands as well as source operand data swizzling/memory operand
     // conversion, eviction hint, and rounding mode.
     EVEX = 3 << EncodingShift,
 
     // Opcode
     OpcodeShift   = EncodingShift + 2,
 
     /// VEX_W - Has a opcode specific functionality, but is used in the same
     /// way as REX_W is for regular SSE instructions.
     VEX_WShift  = OpcodeShift + 8,
     VEX_W       = 1ULL << VEX_WShift,
 
     /// VEX_4V - Used to specify an additional AVX/SSE register. Several 2
     /// address instructions in SSE are represented as 3 address ones in AVX
     /// and the additional register is encoded in VEX_VVVV prefix.
     VEX_4VShift = VEX_WShift + 1,
     VEX_4V      = 1ULL << VEX_4VShift,
 
     /// VEX_L - Stands for a bit in the VEX opcode prefix meaning the current
     /// instruction uses 256-bit wide registers. This is usually auto detected
     /// if a VR256 register is used, but some AVX instructions also have this
     /// field marked when using a f256 memory references.
     VEX_LShift = VEX_4VShift + 1,
     VEX_L       = 1ULL << VEX_LShift,
 
     // EVEX_K - Set if this instruction requires masking
     EVEX_KShift = VEX_LShift + 1,
     EVEX_K      = 1ULL << EVEX_KShift,
 
     // EVEX_Z - Set if this instruction has EVEX.Z field set.
     EVEX_ZShift = EVEX_KShift + 1,
     EVEX_Z      = 1ULL << EVEX_ZShift,
 
     // EVEX_L2 - Set if this instruction has EVEX.L' field set.
     EVEX_L2Shift = EVEX_ZShift + 1,
     EVEX_L2     = 1ULL << EVEX_L2Shift,
 
     // EVEX_B - Set if this instruction has EVEX.B field set.
     EVEX_BShift = EVEX_L2Shift + 1,
     EVEX_B      = 1ULL << EVEX_BShift,
 
     // The scaling factor for the AVX512's 8-bit compressed displacement.
     CD8_Scale_Shift = EVEX_BShift + 1,
     CD8_Scale_Mask = 127ULL << CD8_Scale_Shift,
 
     /// Explicitly specified rounding control
     EVEX_RCShift = CD8_Scale_Shift + 7,
     EVEX_RC = 1ULL << EVEX_RCShift,

llvm/lib/Target/X86/MCTargetDesc/X86MCCodeEmitter.cpp

 //===-- X86MCCodeEmitter.cpp - Convert X86 code to machine code -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
void X86MCCodeEmitter::emitVEXOpcodePrefix(int MemOperand, const MCInst &MI,
   // opcode extension, or ignored, depending on the opcode byte)
   uint8_t VEX_W = (TSFlags & X86II::VEX_W) ? 1 : 0;
 
   // VEX_5M (VEX m-mmmmm field):
   //
   //  0b00000: Reserved for future use
   //  0b00001: implied 0F leading opcode
   //  0b00010: implied 0F 38 leading opcode bytes
   //  0b00011: implied 0F 3A leading opcode bytes
   //  0b00100-0b11111: Reserved for future use
   //  0b01000: XOP map select - 08h instructions with imm byte
   //  0b01001: XOP map select - 09h instructions with no imm byte
   //  0b01010: XOP map select - 0Ah instructions with imm dword
   uint8_t VEX_5M;
   switch (TSFlags & X86II::OpMapMask) {
   default:
     llvm_unreachable("Invalid prefix!");
case X86II::XOP8:
     VEX_5M = 0x8;
     break;
   case X86II::XOP9:
     VEX_5M = 0x9;
     break;
   case X86II::XOPA:
     VEX_5M = 0xA;
     break;
+  case X86II::T_MAP5:
+    VEX_5M = 0x5;
+    break;
+  case X86II::T_MAP6:
+    VEX_5M = 0x6;
+    break;
   }
 
   // VEX_4V (VEX vvvv field): a register specifier
   // (in 1's complement form) or 1111 if unused.
   uint8_t VEX_4V = 0xf;
   uint8_t EVEX_V2 = 0x1;
 
   // EVEX_L2/VEX_L (Vector Length):
if (Encoding == X86II::VEX || Encoding == X86II::XOP) {
     emitByte(Encoding == X86II::XOP ? 0x8F : 0xC4, OS);
     emitByte(VEX_R << 7 | VEX_X << 6 | VEX_B << 5 | VEX_5M, OS);
     emitByte(LastByte | (VEX_W << 7), OS);
   } else {
     assert(Encoding == X86II::EVEX && "unknown encoding!");
     // EVEX opcode prefix can have 4 bytes
     //
     // +-----+ +--------------+ +-------------------+ +------------------------+
-    // | 62h | | RXBR' | 00mm | | W | vvvv | U | pp | | z | L'L | b | v' | aaa |
+    // | 62h | | RXBR' | 0mmm | | W | vvvv | U | pp | | z | L'L | b | v' | aaa |
     // +-----+ +--------------+ +-------------------+ +------------------------+
-    assert((VEX_5M & 0x3) == VEX_5M &&
-           "More than 2 significant bits in VEX.m-mmmm fields for EVEX!");
+    assert((VEX_5M & 0x7) == VEX_5M &&
+           "More than 3 significant bits in VEX.m-mmmm fields for EVEX!");
 
     emitByte(0x62, OS);
     emitByte((VEX_R << 7) | (VEX_X << 6) | (VEX_B << 5) | (EVEX_R2 << 4) |
                  VEX_5M,
              OS);
     emitByte((VEX_W << 7) | (VEX_4V << 3) | (EVEX_U << 2) | VEX_PP, OS);
     if (EncodeRC)
       emitByte((EVEX_z << 7) | (EVEX_rc << 5) | (EVEX_b << 4) | (EVEX_V2 << 3) |

llvm/lib/Target/X86/X86.td

def FeatureBF16    : SubtargetFeature<"avx512bf16", "HasBF16", "true",
                                       [FeatureBWI]>;
 def FeatureBITALG  : SubtargetFeature<"avx512bitalg", "HasBITALG", "true",
                        "Enable AVX-512 Bit Algorithms",
                         [FeatureBWI]>;
 def FeatureVP2INTERSECT  : SubtargetFeature<"avx512vp2intersect",
                                             "HasVP2INTERSECT", "true",
                                             "Enable AVX-512 vp2intersect",
                                             [FeatureAVX512]>;
+// FIXME: FP16 scalar intrinsics use the type v8f16, which is supposed to be
+// guarded under condition hasVLX. So we imply it in FeatureFP16 currently.
+// FIXME: FP16 conversion between f16 and i64 customise type v8i64, which is
+// supposed to be guarded under condition hasDQI. So we imply it in FeatureFP16
+// currently.
+def FeatureFP16    : SubtargetFeature<"avx512fp16", "HasFP16", "true",
+                           "Support 16-bit floating point",
+                           [FeatureBWI, FeatureVLX, FeatureDQI]>;
 def FeaturePCLMUL  : SubtargetFeature<"pclmul", "HasPCLMUL", "true",
                          "Enable packed carry-less multiplication instructions",
                                [FeatureSSE2]>;
 def FeatureGFNI    : SubtargetFeature<"gfni", "HasGFNI", "true",
                          "Enable Galois Field Arithmetic Instructions",
                                [FeatureSSE2]>;
 def FeatureVPCLMULQDQ : SubtargetFeature<"vpclmulqdq", "HasVPCLMULQDQ", "true",
                                          "Enable vpclmulqdq instructions",
def ProcessorFeatures {
   list<SubtargetFeature> SPRAdditionalFeatures = [FeatureAMXTILE,
                                                   FeatureAMXINT8,
                                                   FeatureAMXBF16,
                                                   FeatureBF16,
                                                   FeatureSERIALIZE,
                                                   FeatureCLDEMOTE,
                                                   FeatureWAITPKG,
                                                   FeaturePTWRITE,
+                                                  FeatureFP16,
                                                   FeatureAVXVNNI,
                                                   FeatureTSXLDTRK,
                                                   FeatureENQCMD,
                                                   FeatureSHSTK,
                                                   FeatureVP2INTERSECT,
                                                   FeatureMOVDIRI,
                                                   FeatureMOVDIR64B,
                                                   FeatureUINTR];

llvm/lib/Target/X86/X86CallingConv.td

def RetCC_X86Common : CallingConv<[
   CCIfType<[v8i1],  CCPromoteToType<v8i16>>,
   CCIfType<[v16i1], CCPromoteToType<v16i8>>,
   CCIfType<[v32i1], CCPromoteToType<v32i8>>,
   CCIfType<[v64i1], CCPromoteToType<v64i8>>,
 
   // Vector types are returned in XMM0 and XMM1, when they fit.  XMM2 and XMM3
   // can only be used by ABI non-compliant code. If the target doesn't have XMM
   // registers, it won't have vector types.
-  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
             CCAssignToReg<[XMM0,XMM1,XMM2,XMM3]>>,
 
   // 256-bit vectors are returned in YMM0 and XMM1, when they fit. YMM2 and YMM3
   // can only be used by ABI non-compliant code. This vector type is only
   // supported while using the AVX target feature.
-  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
             CCAssignToReg<[YMM0,YMM1,YMM2,YMM3]>>,
 
   // 512-bit vectors are returned in ZMM0 and ZMM1, when they fit. ZMM2 and ZMM3
   // can only be used by ABI non-compliant code. This vector type is only
   // supported while using the AVX-512 target feature.
-  CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
             CCAssignToReg<[ZMM0,ZMM1,ZMM2,ZMM3]>>,
 
   // MMX vector types are always returned in MM0. If the target doesn't have
   // MM0, it doesn't support these vector types.
   CCIfType<[x86mmx], CCAssignToReg<[MM0]>>,
 
   // Long double types are always returned in FP0 (even with SSE),
   // except on Win64.
   CCIfNotSubtarget<"isTargetWin64()", CCIfType<[f80], CCAssignToReg<[FP0, FP1]>>>
 ]>;
 
 // X86-32 C return-value convention.
 def RetCC_X86_32_C : CallingConv<[
   // The X86-32 calling convention returns FP values in FP0, unless marked
   // with "inreg" (used here to distinguish one kind of reg from another,
   // weirdly; this is really the sse-regparm calling convention) in which
   // case they use XMM0, otherwise it is the same as the common X86 calling
   // conv.
   CCIfInReg<CCIfSubtarget<"hasSSE2()",
     CCIfType<[f32, f64], CCAssignToReg<[XMM0,XMM1,XMM2]>>>>,
   CCIfType<[f32,f64], CCAssignToReg<[FP0, FP1]>>,
+  CCIfType<[f16], CCAssignToReg<[XMM0,XMM1,XMM2]>>,
   CCDelegateTo<RetCC_X86Common>
 ]>;
 
 // X86-32 FastCC return-value convention.
 def RetCC_X86_32_Fast : CallingConv<[
   // The X86-32 fastcc returns 1, 2, or 3 FP values in XMM0-2 if the target has
   // SSE2.
   // This can happen when a float, 2 x float, or 3 x float vector is split by
def RetCC_X86_32_VectorCall : CallingConv<[
 
   // Return integers in the standard way.
   CCDelegateTo<RetCC_X86Common>
 ]>;
 
 // X86-64 C return-value convention.
 def RetCC_X86_64_C : CallingConv<[
   // The X86-64 calling convention always returns FP values in XMM0.
+  CCIfType<[f16], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f32], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f64], CCAssignToReg<[XMM0, XMM1]>>,
   CCIfType<[f128], CCAssignToReg<[XMM0, XMM1]>>,
 
   // MMX vector types are always returned in XMM0.
   CCIfType<[x86mmx], CCAssignToReg<[XMM0, XMM1]>>,
 
   // Pointers are always returned in full 64-bit registers.
def CC_X86_64_C : CallingConv<[
   CCIfType<[v2i1],  CCPromoteToType<v2i64>>,
   CCIfType<[v4i1],  CCPromoteToType<v4i32>>,
   CCIfType<[v8i1],  CCPromoteToType<v8i16>>,
   CCIfType<[v16i1], CCPromoteToType<v16i8>>,
   CCIfType<[v32i1], CCPromoteToType<v32i8>>,
   CCIfType<[v64i1], CCPromoteToType<v64i8>>,
 
   // The first 8 FP/Vector arguments are passed in XMM registers.
-  CCIfType<[f32, f64, f128, v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+  CCIfType<[f16, f32, f64, f128, v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
             CCIfSubtarget<"hasSSE1()",
             CCAssignToReg<[XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7]>>>,
 
   // The first 8 256-bit vector arguments are passed in YMM registers, unless
   // this is a vararg function.
   // FIXME: This isn't precisely correct; the x86-64 ABI document says that
   // fixed arguments to vararg functions are supposed to be passed in
   // registers.  Actually modeling that would be a lot of work, though.
-  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
                           CCIfSubtarget<"hasAVX()",
                           CCAssignToReg<[YMM0, YMM1, YMM2, YMM3,
                                          YMM4, YMM5, YMM6, YMM7]>>>>,
 
   // The first 8 512-bit vector arguments are passed in ZMM registers.
-  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
             CCIfSubtarget<"hasAVX512()",
             CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3, ZMM4, ZMM5, ZMM6, ZMM7]>>>>,
 
   // Integer/FP values get stored in stack slots that are 8 bytes in size and
   // 8-byte aligned if there are no more registers to hold them.
-  CCIfType<[i32, i64, f32, f64], CCAssignToStack<8, 8>>,
+  CCIfType<[i32, i64, f16, f32, f64], CCAssignToStack<8, 8>>,
 
   // Long doubles get stack slots whose size and alignment depends on the
   // subtarget.
   CCIfType<[f80, f128], CCAssignToStack<0, 0>>,
 
   // Vectors get 16-byte stack slots that are 16-byte aligned.
-  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCAssignToStack<16, 16>>,
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64], CCAssignToStack<16, 16>>,
 
   // 256-bit vectors get 32-byte stack slots that are 32-byte aligned.
-  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
            CCAssignToStack<32, 32>>,
 
   // 512-bit vectors get 64-byte stack slots that are 64-byte aligned.
-  CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
            CCAssignToStack<64, 64>>
 ]>;
 
 // Calling convention for X86-64 HHVM.
 def CC_X86_64_HHVM : CallingConv<[
   // Use all/any GP registers for args, except RSP.
   CCIfType<[i64], CCAssignToReg<[RBX, R12, RBP, R15,
                                  RDI, RSI, RDX, RCX, R8, R9,
def CC_X86_Win64_C : CallingConv<[
   // Pass SwiftAsync in an otherwise callee saved register so that calls to
   // normal functions don't need to save it somewhere.
   CCIfSwiftAsync<CCIfType<[i64], CCAssignToReg<[R14]>>>,
 
   // The 'CFGuardTarget' parameter, if any, is passed in RAX.
   CCIfCFGuardTarget<CCAssignToReg<[RAX]>>,
 
   // 128 bit vectors are passed by pointer
-  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCPassIndirect<i64>>,
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64], CCPassIndirect<i64>>,
 
   // 256 bit vectors are passed by pointer
-  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64], CCPassIndirect<i64>>,
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64], CCPassIndirect<i64>>,
 
   // 512 bit vectors are passed by pointer
-  CCIfType<[v64i8, v32i16, v16i32, v16f32, v8f64, v8i64], CCPassIndirect<i64>>,
+  CCIfType<[v64i8, v32i16, v16i32, v32f16, v16f32, v8f64, v8i64], CCPassIndirect<i64>>,
 
   // Long doubles are passed by pointer
   CCIfType<[f80], CCPassIndirect<i64>>,
 
   // The first 4 MMX vector arguments are passed in GPRs.
   CCIfType<[x86mmx], CCBitConvertToType<i64>>,
 
   // If SSE was disabled, pass FP values smaller than 64-bits as integers in
   // GPRs or on the stack.
   CCIfType<[f32], CCIfNotSubtarget<"hasSSE1()", CCBitConvertToType<i32>>>,
   CCIfType<[f64], CCIfNotSubtarget<"hasSSE1()", CCBitConvertToType<i64>>>,
 
   // The first 4 FP/Vector arguments are passed in XMM registers.
-  CCIfType<[f32, f64],
+  CCIfType<[f16, f32, f64],
            CCAssignToRegWithShadow<[XMM0, XMM1, XMM2, XMM3],
                                    [RCX , RDX , R8  , R9  ]>>,
 
   // The first 4 integer arguments are passed in integer registers.
   CCIfType<[i8 ], CCAssignToRegWithShadow<[CL  , DL  , R8B , R9B ],
                                           [XMM0, XMM1, XMM2, XMM3]>>,
   CCIfType<[i16], CCAssignToRegWithShadow<[CX  , DX  , R8W , R9W ],
                                           [XMM0, XMM1, XMM2, XMM3]>>,
   CCIfType<[i32], CCAssignToRegWithShadow<[ECX , EDX , R8D , R9D ],
                                           [XMM0, XMM1, XMM2, XMM3]>>,
 
   // Do not pass the sret argument in RCX, the Win64 thiscall calling
   // convention requires "this" to be passed in RCX.
   CCIfCC<"CallingConv::X86_ThisCall",
     CCIfSRet<CCIfType<[i64], CCAssignToRegWithShadow<[RDX , R8  , R9  ],
                                                      [XMM1, XMM2, XMM3]>>>>,
 
   CCIfType<[i64], CCAssignToRegWithShadow<[RCX , RDX , R8  , R9  ],
                                           [XMM0, XMM1, XMM2, XMM3]>>,
 
   // Integer/FP values get stored in stack slots that are 8 bytes in size and
   // 8-byte aligned if there are no more registers to hold them.
-  CCIfType<[i8, i16, i32, i64, f32, f64], CCAssignToStack<8, 8>>
+  CCIfType<[i8, i16, i32, i64, f16, f32, f64], CCAssignToStack<8, 8>>
 ]>;
 
 def CC_X86_Win64_VectorCall : CallingConv<[
   CCCustom<"CC_X86_64_VectorCall">,
 
   // Delegate to fastcall to handle integer types.
   CCDelegateTo<CC_X86_Win64_C>
 ]>;

 //===----------------------------------------------------------------------===//
 // X86 C Calling Convention
 //===----------------------------------------------------------------------===//
 
 /// CC_X86_32_Vector_Common - In all X86-32 calling conventions, extra vector
 /// values are spilled on the stack.
 def CC_X86_32_Vector_Common : CallingConv<[
   // Other SSE vectors get 16-byte stack slots that are 16-byte aligned.
-  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCAssignToStack<16, 16>>,
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
+           CCAssignToStack<16, 16>>,
 
   // 256-bit AVX vectors get 32-byte stack slots that are 32-byte aligned.
-  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
            CCAssignToStack<32, 32>>,
 
   // 512-bit AVX 512-bit vectors get 64-byte stack slots that are 64-byte aligned.
-  CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
            CCAssignToStack<64, 64>>
 ]>;
 
 // CC_X86_32_Vector_Standard - The first 3 vector arguments are passed in
 // vector registers
 def CC_X86_32_Vector_Standard : CallingConv<[
   // SSE vector arguments are passed in XMM registers.
-  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
                 CCAssignToReg<[XMM0, XMM1, XMM2]>>>,
 
   // AVX 256-bit vector arguments are passed in YMM registers.
-  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
                 CCIfSubtarget<"hasAVX()",
                 CCAssignToReg<[YMM0, YMM1, YMM2]>>>>,
 
   // AVX 512-bit vector arguments are passed in ZMM registers.
-  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
                 CCAssignToReg<[ZMM0, ZMM1, ZMM2]>>>,
 
   CCDelegateTo<CC_X86_32_Vector_Common>
 ]>;
 
 // CC_X86_32_Vector_Darwin - The first 4 vector arguments are passed in
 // vector registers.
 def CC_X86_32_Vector_Darwin : CallingConv<[
   // SSE vector arguments are passed in XMM registers.
-  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
                 CCAssignToReg<[XMM0, XMM1, XMM2, XMM3]>>>,
 
   // AVX 256-bit vector arguments are passed in YMM registers.
-  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64],
                 CCIfSubtarget<"hasAVX()",
                 CCAssignToReg<[YMM0, YMM1, YMM2, YMM3]>>>>,
 
   // AVX 512-bit vector arguments are passed in ZMM registers.
-  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v32f16, v16f32, v8f64],
                 CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3]>>>,
 
   CCDelegateTo<CC_X86_32_Vector_Common>
 ]>;
 
 /// CC_X86_32_Common - In all X86-32 calling conventions, extra integers and FP
 /// values are spilled on the stack.
 def CC_X86_32_Common : CallingConv<[
   // Handles byval/preallocated parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
   CCIfPreallocated<CCPassByVal<4, 4>>,
 
   // The first 3 float or double arguments, if marked 'inreg' and if the call
   // is not a vararg call and if SSE2 is available, are passed in SSE registers.
   CCIfNotVarArg<CCIfInReg<CCIfType<[f32,f64],
                 CCIfSubtarget<"hasSSE2()",
                 CCAssignToReg<[XMM0,XMM1,XMM2]>>>>>,
 
+  CCIfNotVarArg<CCIfInReg<CCIfType<[f16], CCAssignToReg<[XMM0,XMM1,XMM2]>>>>,
+
   // The first 3 __m64 vector arguments are passed in mmx registers if the
   // call is not a vararg call.
   CCIfNotVarArg<CCIfType<[x86mmx],
                 CCAssignToReg<[MM0, MM1, MM2]>>>,
 
+  CCIfType<[f16], CCAssignToStack<4, 4>>,
+
   // Integer/Float values get stored in stack slots that are 4 bytes in
   // size and 4-byte aligned.
   CCIfType<[i32, f32], CCAssignToStack<4, 4>>,
 
   // Doubles get 8-byte slots that are 4-byte aligned.
   CCIfType<[f64], CCAssignToStack<8, 4>>,
 
   // Long doubles get slots whose size depends on the subtarget.

llvm/lib/Target/X86/X86FastISel.cpp

 //===-- X86FastISel.cpp - X86 FastISel implementation ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
class X86FastISel final : public FastISel {
   const X86Subtarget *Subtarget;
 
   /// X86ScalarSSEf32, X86ScalarSSEf64 - Select between SSE or x87
   /// floating point ops.
   /// When SSE is available, use it for f32 operations.
   /// When SSE2 is available, use it for f64 operations.
   bool X86ScalarSSEf64;
   bool X86ScalarSSEf32;
+  bool X86ScalarAVXf16;
 
 public:
   explicit X86FastISel(FunctionLoweringInfo &funcInfo,
                        const TargetLibraryInfo *libInfo)
       : FastISel(funcInfo, libInfo) {
     Subtarget = &funcInfo.MF->getSubtarget<X86Subtarget>();
     X86ScalarSSEf64 = Subtarget->hasSSE2();
     X86ScalarSSEf32 = Subtarget->hasSSE1();
+    X86ScalarAVXf16 = Subtarget->hasFP16();
   }
 
   bool fastSelectInstruction(const Instruction *I) override;
 
   /// The specified machine instr operand is a vreg, and that
   /// vreg is being provided by the specified load instruction.  If possible,
   /// try to fold the load as an operand to the instruction, returning true if
   /// possible.
private:
   unsigned fastMaterializeAlloca(const AllocaInst *C) override;
 
   unsigned fastMaterializeFloatZero(const ConstantFP *CF) override;
 
   /// isScalarFPTypeInSSEReg - Return true if the specified scalar FP type is
   /// computed in an SSE register, not on the X87 floating point stack.
   bool isScalarFPTypeInSSEReg(EVT VT) const {
     return (VT == MVT::f64 && X86ScalarSSEf64) || // f64 is when SSE2
-      (VT == MVT::f32 && X86ScalarSSEf32);   // f32 is when SSE1
+           (VT == MVT::f32 && X86ScalarSSEf32) || // f32 is when SSE1
+           (VT == MVT::f16 && X86ScalarAVXf16);   // f16 is when AVX512FP16
   }
 
   bool isTypeLegal(Type *Ty, MVT &VT, bool AllowI1 = false);
 
   bool IsMemcpySmall(uint64_t Len);
 
   bool TryEmitSmallMemcpy(X86AddressMode DestAM,
                           X86AddressMode SrcAM, uint64_t Len);

 }
 
 bool X86FastISel::X86FastEmitPseudoSelect(MVT RetVT, const Instruction *I) {
   // These are pseudo CMOV instructions and will be later expanded into control-
   // flow.
   unsigned Opc;
   switch (RetVT.SimpleTy) {
   default: return false;
-  case MVT::i8:  Opc = X86::CMOV_GR8;  break;
-  case MVT::i16: Opc = X86::CMOV_GR16; break;
-  case MVT::i32: Opc = X86::CMOV_GR32; break;
+  case MVT::i8:  Opc = X86::CMOV_GR8;   break;
+  case MVT::i16: Opc = X86::CMOV_GR16;  break;
+  case MVT::f16: Opc = X86::CMOV_FR16X; break;
+  case MVT::i32: Opc = X86::CMOV_GR32;  break;
   case MVT::f32: Opc = Subtarget->hasAVX512() ? X86::CMOV_FR32X
                                               : X86::CMOV_FR32; break;
   case MVT::f64: Opc = Subtarget->hasAVX512() ? X86::CMOV_FR64X
                                               : X86::CMOV_FR64; break;
   }
 
   const Value *Cond = I->getOperand(0);
   X86::CondCode CC = X86::COND_NE;

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

 //===- X86ISelDAGToDAG.cpp - A DAG pattern matching inst selector for X86 -===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
for (SelectionDAG::allnodes_iterator I = CurDAG->allnodes_begin(),
     case X86ISD::FOR:
     case X86ISD::FXOR: {
       // Widen scalar fp logic ops to vector to reduce isel patterns.
       // FIXME: Can we do this during lowering/combine.
       MVT VT = N->getSimpleValueType(0);
       if (VT.isVector() || VT == MVT::f128)
         break;
 
-      MVT VecVT = VT == MVT::f64 ? MVT::v2f64 : MVT::v4f32;
+      MVT VecVT = VT == MVT::f64   ? MVT::v2f64
+                  : VT == MVT::f32 ? MVT::v4f32
+                                   : MVT::v8f16;
+
       SDLoc dl(N);
       SDValue Op0 = CurDAG->getNode(ISD::SCALAR_TO_VECTOR, dl, VecVT,
                                     N->getOperand(0));
       SDValue Op1 = CurDAG->getNode(ISD::SCALAR_TO_VECTOR, dl, VecVT,
                                     N->getOperand(1));
 
       SDValue Res;
       if (Subtarget->hasSSE2()) {

llvm/lib/Target/X86/X86ISelLowering.h

 //===-- X86ISelLowering.h - X86 DAG Lowering Interface ----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum NodeType : unsigned {
     SHUF128,
     MOVDDUP,
     MOVSHDUP,
     MOVSLDUP,
     MOVLHPS,
     MOVHLPS,
     MOVSD,
     MOVSS,
+    MOVSH,
     UNPCKL,
     UNPCKH,
     VPERMILPV,
     VPERMILPI,
     VPERMI,
     VPERM2X128,
 
     // Variable Permute (VPERM).
public:
 
     bool isCheapToSpeculateCttz() const override;
 
     bool isCheapToSpeculateCtlz() const override;
 
     bool isCtlzFast() const override;
 
     bool hasBitPreservingFPLogic(EVT VT) const override {
-      return VT == MVT::f32 || VT == MVT::f64 || VT.isVector();
+      return VT == MVT::f32 || VT == MVT::f64 || VT.isVector() ||
+             (VT == MVT::f16 && X86ScalarAVXf16);
     }
 
     bool isMultiStoresCheaperThanBitsMerge(EVT LTy, EVT HTy) const override {
       // If the pair to store is a mixture of float and int values, we will
       // save two bitwise instructions and one float-to-int instruction and
       // increase one store instruction. There is potentially a more
       // significant benefit because it avoids the float->int domain switch
       // for input value. So It is more likely a win.
public:
     /// load node to a smaller type.
     bool shouldReduceLoadWidth(SDNode *Load, ISD::LoadExtType ExtTy,
                                EVT NewVT) const override;
 
     /// Return true if the specified scalar FP type is computed in an SSE
     /// register, not on the X87 floating point stack.
     bool isScalarFPTypeInSSEReg(EVT VT) const {
       return (VT == MVT::f64 && X86ScalarSSEf64) || // f64 is when SSE2
-             (VT == MVT::f32 && X86ScalarSSEf32);   // f32 is when SSE1
+             (VT == MVT::f32 && X86ScalarSSEf32) || // f32 is when SSE1
+             (VT == MVT::f16 && X86ScalarAVXf16);   // f16 is when AVX512FP16
     }
 
     /// Returns true if it is beneficial to convert a load of a constant
     /// to just the constant itself.
     bool shouldConvertConstantLoadToIntImm(const APInt &Imm,
                                            Type *Ty) const override;
 
     bool reduceSelectOfFPConstantLoads(EVT CmpOpVT) const override;
private:
     /// make the right decision when generating code for different targets.
     const X86Subtarget &Subtarget;
 
     /// Select between SSE or x87 floating point ops.
     /// When SSE is available, use it for f32 operations.
     /// When SSE2 is available, use it for f64 operations.
     bool X86ScalarSSEf32;
     bool X86ScalarSSEf64;
+    bool X86ScalarAVXf16;
 
     /// A list of legal FP immediates.
     std::vector<APFloat> LegalFPImmediates;
 
     /// Indicate that this x86 target can instruction
     /// select the specified FP immediate natively.
     void addLegalFPImmediate(const APFloat& Imm) {
       LegalFPImmediates.push_back(Imm);

llvm/lib/Target/X86/X86ISelLowering.cpp

 //===-- X86ISelLowering.cpp - X86 DAG Lowering Implementation -------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 }
 
 X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
                                      const X86Subtarget &STI)
     : TargetLowering(TM), Subtarget(STI) {
   bool UseX87 = !Subtarget.useSoftFloat() && Subtarget.hasX87();
   X86ScalarSSEf64 = Subtarget.hasSSE2();
   X86ScalarSSEf32 = Subtarget.hasSSE1();
+  X86ScalarAVXf16 = Subtarget.hasFP16();
   MVT PtrVT = MVT::getIntegerVT(TM.getPointerSizeInBits(0));
 
   // Set up the TargetLowering object.
 
   // X86 is weird. It always uses i8 for shift amounts and setcc results.
   setBooleanContents(ZeroOrOneBooleanContent);
   // X86-SSE is even stranger. It uses -1 or 0 for vector masks.
   setBooleanVectorContents(ZeroOrNegativeOneBooleanContent);
if (!Subtarget.useSoftFloat() && Subtarget.hasBWI()) {
     // TODO: Custom widen in lowering on non-VLX and drop the isel patterns?
 
     if (Subtarget.hasBITALG()) {
       for (auto VT : { MVT::v16i8, MVT::v32i8, MVT::v8i16, MVT::v16i16 })
         setOperationAction(ISD::CTPOP, VT, Legal);
     }
   }
 
+  if (!Subtarget.useSoftFloat() && Subtarget.hasFP16()) {
+    auto setGroup = [&] (MVT VT) {
+      setOperationAction(ISD::LOAD,               VT, Legal);
+      setOperationAction(ISD::STORE,              VT, Legal);
+
+      setOperationAction(ISD::VSELECT,            VT, Legal);
+      setOperationAction(ISD::BUILD_VECTOR,       VT, Custom);
+      setOperationAction(ISD::SELECT,             VT, Custom);
+
+      setOperationAction(ISD::FNEG,               VT, Custom);
+      setOperationAction(ISD::FABS,               VT, Custom);
+      setOperationAction(ISD::FCOPYSIGN,          VT, Custom);
+      setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
+      setOperationAction(ISD::VECTOR_SHUFFLE,     VT, Custom);
+    };
+
+    // AVX512_FP16 scalar operations
+    setGroup(MVT::f16);
+    addRegisterClass(MVT::f16,    &X86::FR16XRegClass);
+
+    if (Subtarget.useAVX512Regs()) {
+      setGroup(MVT::v32f16);
+      addRegisterClass(MVT::v32f16, &X86::VR512RegClass);
+      setOperationAction(ISD::SCALAR_TO_VECTOR,       MVT::v32f16, Custom);
+      setOperationAction(ISD::INSERT_VECTOR_ELT,      MVT::v32f16, Custom);
+
+      setOperationAction(ISD::EXTRACT_SUBVECTOR,      MVT::v16f16, Legal);
+      setOperationAction(ISD::INSERT_SUBVECTOR,       MVT::v32f16, Legal);
+      setOperationAction(ISD::CONCAT_VECTORS,         MVT::v32f16, Custom);
+
+      setLoadExtAction(ISD::EXTLOAD, MVT::v8f64,  MVT::v8f16,  Legal);
+      setLoadExtAction(ISD::EXTLOAD, MVT::v16f32, MVT::v16f16, Legal);
+    }
+
+    if (Subtarget.hasVLX()) {
+      addRegisterClass(MVT::v8f16,  &X86::VR128XRegClass);
+      addRegisterClass(MVT::v16f16, &X86::VR256XRegClass);
+      for (auto VT : { MVT::v8f16, MVT::v16f16 }) {
+        setGroup(VT);
+      }
+      setOperationAction(ISD::SCALAR_TO_VECTOR,   MVT::v8f16,  Legal);
+      setOperationAction(ISD::SCALAR_TO_VECTOR,   MVT::v16f16, Custom);
+
+      // INSERT_VECTOR_ELT v8f16 extended to VECTOR_SHUFFLE
+      setOperationAction(ISD::INSERT_VECTOR_ELT,    MVT::v8f16,  Custom);
+      setOperationAction(ISD::INSERT_VECTOR_ELT,    MVT::v16f16, Custom);
+
+      setOperationAction(ISD::EXTRACT_SUBVECTOR,    MVT::v8f16, Legal);
+      setOperationAction(ISD::INSERT_SUBVECTOR,     MVT::v16f16, Legal);
+      setOperationAction(ISD::CONCAT_VECTORS,       MVT::v16f16, Custom);
+
+      setLoadExtAction(ISD::EXTLOAD, MVT::v4f64, MVT::v4f16, Legal);
+      setLoadExtAction(ISD::EXTLOAD, MVT::v2f64, MVT::v2f16, Legal);
+      setLoadExtAction(ISD::EXTLOAD, MVT::v8f32, MVT::v8f16, Legal);
+      setLoadExtAction(ISD::EXTLOAD, MVT::v4f32, MVT::v4f16, Legal);
+
+      // Need to custom widen these to prevent scalarization.
+      setOperationAction(ISD::LOAD,  MVT::v4f16, Custom);
+      setOperationAction(ISD::STORE, MVT::v4f16, Custom);
+    }
+
+    // Support fp16 0 immediate
+    addLegalFPImmediate(APFloat::getZero(APFloat::IEEEhalf()));
+  }
+
   if (!Subtarget.useSoftFloat() && Subtarget.hasVLX()) {
     setTruncStoreAction(MVT::v4i64, MVT::v4i8,  Legal);
     setTruncStoreAction(MVT::v4i64, MVT::v4i16, Legal);
     setTruncStoreAction(MVT::v4i64, MVT::v4i32, Legal);
     setTruncStoreAction(MVT::v8i32, MVT::v8i8,  Legal);
     setTruncStoreAction(MVT::v8i32, MVT::v8i16, Legal);
 
     setTruncStoreAction(MVT::v2i64, MVT::v2i8,  Legal);
if (VA.isRegLoc()) {
         if (RegVT == MVT::i8)
           RC = &X86::GR8RegClass;
         else if (RegVT == MVT::i16)
           RC = &X86::GR16RegClass;
         else if (RegVT == MVT::i32)
           RC = &X86::GR32RegClass;
         else if (Is64Bit && RegVT == MVT::i64)
           RC = &X86::GR64RegClass;
+        else if (RegVT == MVT::f16)
+          RC = &X86::FR16XRegClass;
         else if (RegVT == MVT::f32)
           RC = Subtarget.hasAVX512() ? &X86::FR32XRegClass : &X86::FR32RegClass;
         else if (RegVT == MVT::f64)
           RC = Subtarget.hasAVX512() ? &X86::FR64XRegClass : &X86::FR64RegClass;
         else if (RegVT == MVT::f80)
           RC = &X86::RFP80RegClass;
         else if (RegVT == MVT::f128)
           RC = &X86::VR128RegClass;
static bool isTargetShuffle(unsigned Opcode) {
   case X86ISD::VSRLDQ:
   case X86ISD::MOVLHPS:
   case X86ISD::MOVHLPS:
   case X86ISD::MOVSHDUP:
   case X86ISD::MOVSLDUP:
   case X86ISD::MOVDDUP:
   case X86ISD::MOVSS:
   case X86ISD::MOVSD:
+  case X86ISD::MOVSH:
   case X86ISD::UNPCKL:
   case X86ISD::UNPCKH:
   case X86ISD::VBROADCAST:
   case X86ISD::VPERMILPI:
   case X86ISD::VPERMILPV:
   case X86ISD::VPERM2X128:
   case X86ISD::SHUF128:
   case X86ISD::VPERMIL2:
static bool getTargetShuffleMask(SDNode *N, MVT VT, bool AllowSentinelZero,
   case X86ISD::VPERMI:
     assert(N->getOperand(0).getValueType() == VT && "Unexpected value type");
     ImmN = N->getConstantOperandVal(N->getNumOperands() - 1);
     DecodeVPERMMask(NumElems, ImmN, Mask);
     IsUnary = true;
     break;
   case X86ISD::MOVSS:
   case X86ISD::MOVSD:
+  case X86ISD::MOVSH:
     assert(N->getOperand(0).getValueType() == VT && "Unexpected value type");
     assert(N->getOperand(1).getValueType() == VT && "Unexpected value type");
     DecodeScalarMoveMask(NumElems, /* IsLoad */ false, Mask);
     break;
   case X86ISD::VPERM2X128:
     assert(N->getOperand(0).getValueType() == VT && "Unexpected value type");
     assert(N->getOperand(1).getValueType() == VT && "Unexpected value type");
     ImmN = N->getConstantOperandVal(N->getNumOperands() - 1);
if (UndefMask.extractBits(HalfNumElems, HalfNumElems).isAllOnesValue()) {
       if (HalfLD)
         return DAG.getNode(ISD::INSERT_SUBVECTOR, DL, VT, DAG.getUNDEF(VT),
                            HalfLD, DAG.getIntPtrConstant(0, DL));
     }
   }
 
   // VZEXT_LOAD - consecutive 32/64-bit load/undefs followed by zeros/undefs.
   if (IsConsecutiveLoad && FirstLoadedElt == 0 &&
-      (LoadSizeInBits == 32 || LoadSizeInBits == 64) &&
+      ((LoadSizeInBits == 16 && Subtarget.hasFP16()) || LoadSizeInBits == 32 ||
+       LoadSizeInBits == 64) &&
       ((VT.is128BitVector() || VT.is256BitVector() || VT.is512BitVector()))) {
     MVT VecSVT = VT.isFloatingPoint() ? MVT::getFloatingPointVT(LoadSizeInBits)
                                       : MVT::getIntegerVT(LoadSizeInBits);
     MVT VecVT = MVT::getVectorVT(VecSVT, VT.getSizeInBits() / LoadSizeInBits);
     // Allow v4f32 on SSE1 only targets.
     // FIXME: Add more isel patterns so we can just use VT directly.
     if (!Subtarget.hasSSE2() && VT == MVT::v4f32)
       VecVT = MVT::v4f32;
if (ConstSplatVal && (Subtarget.hasAVX2() || OptForSize)) {
     EVT CVT = Ld.getValueType();
     assert(!CVT.isVector() && "Must not broadcast a vector type");
 
     // Splat f32, i32, v4f64, v4i64 in all cases with AVX2.
     // For size optimization, also splat v2f64 and v2i64, and for size opt
     // with AVX2, also splat i8 and i16.
     // With pattern matching, the VBROADCAST node may become a VMOVDDUP.
     if (ScalarSize == 32 || (IsGE256 && ScalarSize == 64) ||
+        (ScalarSize == 16 && Subtarget.hasFP16() && CVT.isFloatingPoint()) ||
         (OptForSize && (ScalarSize == 64 || Subtarget.hasAVX2()))) {
       const Constant *C = nullptr;
       if (ConstantSDNode *CI = dyn_cast<ConstantSDNode>(Ld))
         C = CI->getConstantIntValue();
       else if (ConstantFPSDNode *CF = dyn_cast<ConstantFPSDNode>(Ld))
         C = CF->getConstantFPValue();
 
       assert(C && "Invalid constant type");
if (Subtarget.hasInt256() && Ld.getValueType().isInteger() &&
     SDValue Ops[] = {LN->getChain(), LN->getBasePtr()};
     SDValue BCast =
         DAG.getMemIntrinsicNode(X86ISD::VBROADCAST_LOAD, dl, Tys, Ops,
                                 LN->getMemoryVT(), LN->getMemOperand());
     DAG.ReplaceAllUsesOfValueWith(SDValue(LN, 1), BCast.getValue(1));
     return BCast;
   }
 
+  if (ScalarSize == 16 && Subtarget.hasFP16() && IsGE256)
+    return DAG.getNode(X86ISD::VBROADCAST, dl, VT, Ld);
+
   // Unsupported broadcast.
   return SDValue();
 }
 
 /// For an EXTRACT_VECTOR_ELT with a constant index return the real
 /// underlying vector and index.
 ///
 /// Modifies \p ExtractedFromVec to the real vector and returns the real
if (NumNonZero == 1) {
     // If we have a constant or non-constant insertion into the low element of
     // a vector, we can do this with SCALAR_TO_VECTOR + shuffle of zero into
     // the rest of the elements.  This will be matched as movd/movq/movss/movsd
     // depending on what the source datatype is.
     if (Idx == 0) {
       if (NumZero == 0)
         return DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VT, Item);
 
-      if (EltVT == MVT::i32 || EltVT == MVT::f32 || EltVT == MVT::f64 ||
-          (EltVT == MVT::i64 && Subtarget.is64Bit())) {
+      if (EltVT == MVT::i32 || EltVT == MVT::f16 || EltVT == MVT::f32 ||
+          EltVT == MVT::f64 || (EltVT == MVT::i64 && Subtarget.is64Bit())) {
         assert((VT.is128BitVector() || VT.is256BitVector() ||
                 VT.is512BitVector()) &&
                "Expected an SSE value type!");
         Item = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VT, Item);
         // Turn it into a MOVL (i.e. movss, movsd, or movd) to a zero vector.
         return getShuffleVectorZeroOrUndef(Item, 0, true, Subtarget, DAG);
       }
 
       // We can't directly insert an i8 or i16 into a vector, so zero extend
X86TargetLowering::LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const {
   }
 
   // If element VT is < 32 bits, convert it to inserts into a zero vector.
   if (EVTBits == 8 && NumElems == 16)
     if (SDValue V = LowerBuildVectorv16i8(Op, NonZeroMask, NumNonZero, NumZero,
                                           DAG, Subtarget))
       return V;
 
-  if (EVTBits == 16 && NumElems == 8)
+  if (EltVT == MVT::i16 && NumElems == 8)
     if (SDValue V = LowerBuildVectorv8i16(Op, NonZeroMask, NumNonZero, NumZero,
                                           DAG, Subtarget))
       return V;
 
   // If element VT is == 32 bits and has 4 elems, try to generate an INSERTPS
   if (EVTBits == 32 && NumElems == 4)
     if (SDValue V = LowerBuildVectorv4x32(Op, DAG, Subtarget))
       return V;
X86TargetLowering::LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const {
 
   assert(Values.size() > 1 && "Expected non-undef and non-splat vector");
 
   // Check for a build vector from mostly shuffle plus few inserting.
   if (SDValue Sh = buildFromShuffleMostly(Op, DAG))
     return Sh;
 
   // For SSE 4.1, use insertps to put the high elements into the low element.
-  if (Subtarget.hasSSE41()) {
+  if (Subtarget.hasSSE41() && EltVT != MVT::f16) {
     SDValue Result;
     if (!Op.getOperand(0).isUndef())
       Result = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VT, Op.getOperand(0));
     else
       Result = DAG.getUNDEF(VT);
 
     for (unsigned i = 1; i < NumElems; ++i) {
       if (Op.getOperand(i).isUndef()) continue;
if (!VT.isFloatingPoint() || V2Index != 0)
       return SDValue();
     SmallVector<int, 8> V1Mask(Mask.begin(), Mask.end());
     V1Mask[V2Index] = -1;
     if (!isNoopShuffleMask(V1Mask))
       return SDValue();
     if (!VT.is128BitVector())
       return SDValue();
 
-    // Otherwise, use MOVSD or MOVSS.
-    assert((EltVT == MVT::f32 || EltVT == MVT::f64) &&
-           "Only two types of floating point element types to handle!");
-    return DAG.getNode(EltVT == MVT::f32 ? X86ISD::MOVSS : X86ISD::MOVSD, DL,
-                       ExtVT, V1, V2);
+    // Otherwise, use MOVSD, MOVSS or MOVSH.
+    unsigned MovOpc = 0;
+    if (EltVT == MVT::f16) {
+      MovOpc = X86ISD::MOVSH;
+    } else if (EltVT == MVT::f32) {
+      MovOpc = X86ISD::MOVSS;
+    } else if (EltVT == MVT::f64) {
+      MovOpc = X86ISD::MOVSD;
+    }
+    assert((MovOpc != 0) &&
+           "Unsupported floating point element type to handle!");
+    return DAG.getNode(MovOpc, DL, ExtVT, V1, V2);
   }
 
   // This lowering only works for the low element with floating point vectors.
   if (VT.isFloatingPoint() && V2Index != 0)
     return SDValue();
 
   V2 = DAG.getNode(X86ISD::VZEXT_MOVL, DL, ExtVT, V2);
   if (ExtVT != VT)
static SDValue lowerV8I16Shuffle(const SDLoc &DL, ArrayRef<int> Mask,
   }
 
   // We can always bit-blend if we have to so the fallback strategy is to
   // decompose into single-input permutes and blends/unpacks.
   return lowerShuffleAsDecomposedShuffleMerge(DL, MVT::v8i16, V1, V2,
                                               Mask, Subtarget, DAG);
 }
 
+/// Lower 8-lane 16-bit floating point shuffles.
+static SDValue lowerV8F16Shuffle(const SDLoc &DL, ArrayRef<int> Mask,
+                                 const APInt &Zeroable, SDValue V1, SDValue V2,
+                                 const X86Subtarget &Subtarget,
+                                 SelectionDAG &DAG) {
+  assert(V1.getSimpleValueType() == MVT::v8f16 && "Bad operand type!");
+  assert(V2.getSimpleValueType() == MVT::v8f16 && "Bad operand type!");
+  assert(Mask.size() == 8 && "Unexpected mask size for v8 shuffle!");
+  int NumV2Elements = count_if(Mask, [](int M) { return M >= 8; });
+
+  if (NumV2Elements == 0) {
+    // Check for being able to broadcast a single element.
+    if (SDValue Broadcast = lowerShuffleAsBroadcast(DL, MVT::v8f16, V1, V2,
+                                                    Mask, Subtarget, DAG))
+      return Broadcast;
+  }
+  if (NumV2Elements == 1 && Mask[0] >= 8)
+    if (SDValue V = lowerShuffleAsElementInsertion(DL, MVT::v8f16, V1, V2, Mask,
+                                                   Zeroable, Subtarget, DAG))
+      return V;
+
+  V1 = DAG.getBitcast(MVT::v8i16, V1);
+  V2 = DAG.getBitcast(MVT::v8i16, V2);
+  return DAG.getBitcast(MVT::v8f16,
+                        DAG.getVectorShuffle(MVT::v8i16, DL, V1, V2, Mask));
+}
+
 // Lowers unary/binary shuffle as VPERMV/VPERMV3, for non-VLX targets,
 // sub-512-bit shuffles are padded to 512-bits for the shuffle and then
 // the active subvector is extracted.
 static SDValue lowerShuffleWithPERMV(const SDLoc &DL, MVT VT,
                                      ArrayRef<int> Mask, SDValue V1, SDValue V2,
                                      const X86Subtarget &Subtarget,
                                      SelectionDAG &DAG) {
   MVT MaskVT = VT.changeTypeToInteger();
static SDValue lower128BitShuffle(const SDLoc &DL, ArrayRef<int> Mask,
   case MVT::v2f64:
     return lowerV2F64Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v4i32:
     return lowerV4I32Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v4f32:
     return lowerV4F32Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v8i16:
     return lowerV8I16Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
+  case MVT::v8f16:
+    return lowerV8F16Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v16i8:
     return lowerV16I8Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
 
   default:
     llvm_unreachable("Unimplemented!");
   }
 }
 
if (VT.isInteger() && !Subtarget.hasAVX2()) {
 
     MVT FpVT = MVT::getVectorVT(MVT::getFloatingPointVT(ElementBits),
                                 VT.getVectorNumElements());
     V1 = DAG.getBitcast(FpVT, V1);
     V2 = DAG.getBitcast(FpVT, V2);
     return DAG.getBitcast(VT, DAG.getVectorShuffle(FpVT, DL, V1, V2, Mask));
   }
 
+  if (VT == MVT::v16f16) {
+    V1 = DAG.getBitcast(MVT::v16i16, V1);
+    V2 = DAG.getBitcast(MVT::v16i16, V2);
+    return DAG.getBitcast(MVT::v16f16,
+                          DAG.getVectorShuffle(MVT::v16i16, DL, V1, V2, Mask));
+  }
+
   switch (VT.SimpleTy) {
   case MVT::v4f64:
     return lowerV4F64Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v4i64:
     return lowerV4I64Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v8f32:
     return lowerV8F32Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v8i32:
if (SDValue V = lowerShuffleAsBitMask(DL, VT, V1, V2, Mask, Zeroable,
                                           Subtarget, DAG))
       return V;
     if (SDValue V = lowerShuffleAsBitBlend(DL, VT, V1, V2, Mask, DAG))
       return V;
 
     return splitAndLowerShuffle(DL, VT, V1, V2, Mask, DAG);
   }
 
+  if (VT == MVT::v32f16) {
+    V1 = DAG.getBitcast(MVT::v32i16, V1);
+    V2 = DAG.getBitcast(MVT::v32i16, V2);
+    return DAG.getBitcast(MVT::v32f16,
+                          DAG.getVectorShuffle(MVT::v32i16, DL, V1, V2, Mask));
+  }
+
   // Dispatch to each element type for lowering. If we don't have support for
   // specific element type shuffles at 512 bits, immediately split them and
   // lower them. Each lowering routine of a given type is allowed to assume that
   // the requisite ISA extensions for that element type are available.
   switch (VT.SimpleTy) {
   case MVT::v8f64:
     return lowerV8F64Shuffle(DL, Mask, Zeroable, V1, V2, Subtarget, DAG);
   case MVT::v16f32:
if (VecVT.is256BitVector() || VecVT.is512BitVector()) {
     return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, Op.getValueType(), Vec,
                        DAG.getIntPtrConstant(IdxVal, dl));
   }
 
   assert(VecVT.is128BitVector() && "Unexpected vector length");
 
   MVT VT = Op.getSimpleValueType();
 
-  if (VT.getSizeInBits() == 16) {
+  if (VT == MVT::i16) {
     // If IdxVal is 0, it's cheaper to do a move instead of a pextrw, unless
     // we're going to zero extend the register or fold the store (SSE41 only).
     if (IdxVal == 0 && !MayFoldIntoZeroExtend(Op) &&
-        !(Subtarget.hasSSE41() && MayFoldIntoStore(Op)))
+        !(Subtarget.hasSSE41() && MayFoldIntoStore(Op))) {
+      if (Subtarget.hasFP16())
+        return Op;
+
       return DAG.getNode(ISD::TRUNCATE, dl, MVT::i16,
                          DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::i32,
                                      DAG.getBitcast(MVT::v4i32, Vec), Idx));
+    }
 
     SDValue Extract = DAG.getNode(X86ISD::PEXTRW, dl, MVT::i32, Vec,
                                   DAG.getTargetConstant(IdxVal, dl, MVT::i8));
     return DAG.getNode(ISD::TRUNCATE, dl, VT, Extract);
   }
 
   if (Subtarget.hasSSE41())
     if (SDValue Res = LowerEXTRACT_VECTOR_ELT_SSE4(Op, DAG))
SDValue Res = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, MVT::i16,
                               DAG.getIntPtrConstant(WordIdx, dl));
     int ShiftVal = (IdxVal % 2) * 8;
     if (ShiftVal != 0)
       Res = DAG.getNode(ISD::SRL, dl, MVT::i16, Res,
                         DAG.getConstant(ShiftVal, dl, MVT::i8));
     return DAG.getNode(ISD::TRUNCATE, dl, VT, Res);
   }
 
-  if (VT.getSizeInBits() == 32) {
+  if (VT == MVT::f16 || VT.getSizeInBits() == 32) {
     if (IdxVal == 0)
       return Op;
 
     // SHUFPS the element to the lowest double word, then movss.
-    int Mask[4] = { static_cast<int>(IdxVal), -1, -1, -1 };
+    SmallVector<int, 8> Mask(VecVT.getVectorNumElements(), -1);
+    Mask[0] = static_cast<int>(IdxVal);
     Vec = DAG.getVectorShuffle(VecVT, dl, Vec, DAG.getUNDEF(VecVT), Mask);
     return DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, VT, Vec,
                        DAG.getIntPtrConstant(0, dl));
   }
 
   if (VT.getSizeInBits() == 64) {
     // FIXME: .td only matches this for <2 x f64>, not <2 x i64> on 32b
     // FIXME: seems like this should be unnecessary if mov{h,l}pd were taught
if (VT.is256BitVector() || VT.is512BitVector()) {
     V = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, V.getValueType(), V, N1,
                     DAG.getIntPtrConstant(IdxIn128, dl));
 
     // Insert the changed part back into the bigger vector
     return insert128BitVector(N0, V, IdxVal, DAG, dl);
   }
   assert(VT.is128BitVector() && "Only 128-bit vector types should be left!");
 
   // This will be just movd/movq/movss/movsd.
   if (IdxVal == 0 && ISD::isBuildVectorAllZeros(N0.getNode())) {
     if (EltVT == MVT::i32 || EltVT == MVT::f32 || EltVT == MVT::f64 ||
-        EltVT == MVT::i64) {
+        EltVT == MVT::f16 || EltVT == MVT::i64) {
       N1 = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VT, N1);
       return getShuffleVectorZeroOrUndef(N1, 0, true, Subtarget, DAG);
     }
 
     // We can't directly insert an i8 or i16 into a vector, so zero extend
     // it to i32 first.
     if (EltVT == MVT::i16 || EltVT == MVT::i8) {
       N1 = DAG.getNode(ISD::ZERO_EXTEND, dl, MVT::i32, N1);
if (IsFABS)
     for (SDNode *User : Op->uses())
       if (User->getOpcode() == ISD::FNEG)
         return Op;
 
   SDLoc dl(Op);
   MVT VT = Op.getSimpleValueType();
 
   bool IsF128 = (VT == MVT::f128);
-  assert((VT == MVT::f64 || VT == MVT::f32 || VT == MVT::f128 ||
-          VT == MVT::v2f64 || VT == MVT::v4f64 || VT == MVT::v4f32 ||
-          VT == MVT::v8f32 || VT == MVT::v8f64 || VT == MVT::v16f32) &&
+  assert(VT.isFloatingPoint() && VT != MVT::f80 &&
+         DAG.getTargetLoweringInfo().isTypeLegal(VT) &&
          "Unexpected type in LowerFABSorFNEG");
 
   // FIXME: Use function attribute "OptimizeForSize" and/or CodeGenOpt::Level to
   // decide if we should generate a 16-byte constant mask when we only need 4 or
   // 8 bytes for the scalar case.
 
   // There are no scalar bitwise logical SSE/AVX instructions, so we
   // generate a 16-byte vector constant and logic op even for the scalar case.
   // Using a 16-byte mask allows folding the load of the mask with
   // the logic op, so it can save (~4 bytes) on code size.
   bool IsFakeVector = !VT.isVector() && !IsF128;
   MVT LogicVT = VT;
   if (IsFakeVector)
-    LogicVT = (VT == MVT::f64) ? MVT::v2f64 : MVT::v4f32;
+    LogicVT = (VT == MVT::f64)   ? MVT::v2f64
+              : (VT == MVT::f32) ? MVT::v4f32
+                                 : MVT::v8f16;
 
   unsigned EltBits = VT.getScalarSizeInBits();
   // For FABS, mask is 0x7f...; for FNEG, mask is 0x80...
   APInt MaskElt = IsFABS ? APInt::getSignedMaxValue(EltBits) :
                            APInt::getSignMask(EltBits);
   const fltSemantics &Sem = SelectionDAG::EVTToAPFloatSemantics(VT);
   SDValue Mask = DAG.getConstantFP(APFloat(Sem, MaskElt), dl, LogicVT);
 
static SDValue LowerFCOPYSIGN(SDValue Op, SelectionDAG &DAG) {
   // And if it is bigger, shrink it first.
   if (Sign.getSimpleValueType().bitsGT(VT))
     Sign =
         DAG.getNode(ISD::FP_ROUND, dl, VT, Sign, DAG.getIntPtrConstant(0, dl));
 
   // At this point the operands and the result should have the same
   // type, and that won't be f80 since that is not custom lowered.
   bool IsF128 = (VT == MVT::f128);
-  assert((VT == MVT::f64 || VT == MVT::f32 || VT == MVT::f128 ||
-          VT == MVT::v2f64 || VT == MVT::v4f64 || VT == MVT::v4f32 ||
-          VT == MVT::v8f32 || VT == MVT::v8f64 || VT == MVT::v16f32) &&
+  assert(VT.isFloatingPoint() && VT != MVT::f80 &&
+         DAG.getTargetLoweringInfo().isTypeLegal(VT) &&
          "Unexpected type in LowerFCOPYSIGN");
 
   const fltSemantics &Sem = SelectionDAG::EVTToAPFloatSemantics(VT);
 
   // Perform all scalar logic operations as 16-byte vectors because there are no
   // scalar FP logic instructions in SSE.
   // TODO: This isn't necessary. If we used scalar types, we might avoid some
   // unnecessary splats, but we might miss load folding opportunities. Should
   // this decision be based on OptimizeForSize?
   bool IsFakeVector = !VT.isVector() && !IsF128;
   MVT LogicVT = VT;
   if (IsFakeVector)
-    LogicVT = (VT == MVT::f64) ? MVT::v2f64 : MVT::v4f32;
+    LogicVT = (VT == MVT::f64)   ? MVT::v2f64
+              : (VT == MVT::f32) ? MVT::v4f32
+                                 : MVT::v8f16;
 
   // The mask constants are automatically splatted for vector types.
   unsigned EltSizeInBits = VT.getScalarSizeInBits();
   SDValue SignMask = DAG.getConstantFP(
       APFloat(Sem, APInt::getSignMask(EltSizeInBits)), dl, LogicVT);
   SDValue MagMask = DAG.getConstantFP(
       APFloat(Sem, APInt::getSignedMaxValue(EltSizeInBits)), dl, LogicVT);
 
static SDValue LowerVSETCC(SDValue Op, const X86Subtarget &Subtarget,
   MVT VT = Op->getSimpleValueType(0);
   ISD::CondCode Cond = cast<CondCodeSDNode>(CC)->get();
   bool isFP = Op1.getSimpleValueType().isFloatingPoint();
   SDLoc dl(Op);
 
   if (isFP) {
 #ifndef NDEBUG
     MVT EltVT = Op0.getSimpleValueType().getVectorElementType();
-    assert(EltVT == MVT::f32 || EltVT == MVT::f64);
+    assert(EltVT == MVT::f16 || EltVT == MVT::f32 || EltVT == MVT::f64);
 #endif
 
     bool IsSignaling = Op.getOpcode() == ISD::STRICT_FSETCCS;
     SDValue Chain = IsStrict ? Op.getOperand(0) : SDValue();
 
     // If we have a strict compare with a vXi1 result and the input is 128/256
     // bits we can't use a masked compare unless we have VLX. If we use a wider
     // compare like we do for non-strict, we might trigger spurious exceptions
     // from the upper elements. Instead emit a AVX compare and convert to mask.
     unsigned Opc;
     if (Subtarget.hasAVX512() && VT.getVectorElementType() == MVT::i1 &&
         (!IsStrict || Subtarget.hasVLX() ||
          Op0.getSimpleValueType().is512BitVector())) {
-      assert(VT.getVectorNumElements() <= 16);
+      assert(VT.getVectorNumElements() <= 16 || Subtarget.hasFP16());
       Opc = IsStrict ? X86ISD::STRICT_CMPM : X86ISD::CMPM;
     } else {
       Opc = IsStrict ? X86ISD::STRICT_CMPP : X86ISD::CMPP;
       // The SSE/AVX packed FP comparison nodes are defined with a
       // floating-point vector result that matches the operand type. This allows
       // them to work with an SSE1 target (integer vector types are not legal).
       VT = Op0.getSimpleValueType();
     }
if (VT == MVT::v2i32) {
 
       // The FP_TO_INTHelper below only handles f32/f64/f80 scalar inputs,
       // so early out here.
       return;
     }
 
     assert(!VT.isVector() && "Vectors should have been handled above!");
 
-    if (Subtarget.hasDQI() && VT == MVT::i64 &&
-        (SrcVT == MVT::f32 || SrcVT == MVT::f64)) {
+    if ((Subtarget.hasDQI() && VT == MVT::i64 &&
+         (SrcVT == MVT::f32 || SrcVT == MVT::f64)) ||
+        (Subtarget.hasFP16() && SrcVT == MVT::f16)) {
       assert(!Subtarget.is64Bit() && "i64 should be legal");
       unsigned NumElts = Subtarget.hasVLX() ? 2 : 8;
       // If we use a 128-bit result we might need to use a target specific node.
       unsigned SrcElts =
           std::max(NumElts, 128U / (unsigned)SrcVT.getSizeInBits());
       MVT VecVT = MVT::getVectorVT(MVT::i64, NumElts);
       MVT VecInVT = MVT::getVectorVT(SrcVT.getSimpleVT(), SrcElts);
       unsigned Opc = N->getOpcode();
#define NODE_NAME_CASE(NODE) case X86ISD::NODE: return "X86ISD::" #NODE;
   NODE_NAME_CASE(SHUF128)
   NODE_NAME_CASE(MOVLHPS)
   NODE_NAME_CASE(MOVHLPS)
   NODE_NAME_CASE(MOVDDUP)
   NODE_NAME_CASE(MOVSHDUP)
   NODE_NAME_CASE(MOVSLDUP)
   NODE_NAME_CASE(MOVSD)
   NODE_NAME_CASE(MOVSS)
+  NODE_NAME_CASE(MOVSH)
   NODE_NAME_CASE(UNPCKL)
   NODE_NAME_CASE(UNPCKH)
   NODE_NAME_CASE(VBROADCAST)
   NODE_NAME_CASE(VBROADCAST_LOAD)
   NODE_NAME_CASE(VBROADCASTM)
   NODE_NAME_CASE(SUBV_BROADCAST_LOAD)
   NODE_NAME_CASE(VPERMILPV)
   NODE_NAME_CASE(VPERMILPI)
if (isTargetShuffleEquivalent(MaskVT, Mask, {0, 3}) &&
       return true;
     }
     if (isTargetShuffleEquivalent(MaskVT, Mask, {4, 1, 2, 3}) &&
         (AllowFloatDomain || !Subtarget.hasSSE41())) {
       Shuffle = X86ISD::MOVSS;
       SrcVT = DstVT = MVT::v4f32;
       return true;
     }
+    if (isTargetShuffleEquivalent(MaskVT, Mask, {8, 1, 2, 3, 4, 5, 6, 7}) &&
+        Subtarget.hasFP16()) {
+      Shuffle = X86ISD::MOVSH;
+      SrcVT = DstVT = MVT::v8f16;
+      return true;
+    }
   }
 
   // Attempt to match against either an unary or binary PACKSS/PACKUS shuffle.
   if (((MaskVT == MVT::v8i16 || MaskVT == MVT::v16i8) && Subtarget.hasSSE2()) ||
       ((MaskVT == MVT::v16i16 || MaskVT == MVT::v32i8) && Subtarget.hasInt256()) ||
       ((MaskVT == MVT::v32i16 || MaskVT == MVT::v64i8) && Subtarget.hasBWI())) {
     if (matchShuffleWithPACK(MaskVT, SrcVT, V1, V2, Shuffle, Mask, DAG,
                              Subtarget)) {
static SDValue combineX86ShufflesRecursively(
   // Directly rip through bitcasts to find the underlying operand.
   SDValue Op = SrcOps[SrcOpIndex];
   Op = peekThroughOneUseBitcasts(Op);
 
   EVT VT = Op.getValueType();
   if (!VT.isVector() || !VT.isSimple())
     return SDValue(); // Bail if we hit a non-simple non-vector.
 
+  // FIXME: Just bail on f16 for now.
+  if (VT.getVectorElementType() == MVT::f16)
+    return SDValue();
+
   assert((RootSizeInBits % VT.getSizeInBits()) == 0 &&
          "Can only combine shuffles upto size of the root op.");
 
   // Extract target shuffle mask and resolve sentinels and inputs.
   // TODO - determine Op's demanded elts from RootMask.
   SmallVector<int, 64> OpMask;
   SmallVector<SDValue, 2> OpInputs;
   APInt OpUndef, OpZero;
static SDValue combineTargetShuffle(SDValue N, SelectionDAG &DAG,
   }
   case X86ISD::PSHUFD:
   case X86ISD::PSHUFLW:
   case X86ISD::PSHUFHW:
     Mask = getPSHUFShuffleMask(N);
     assert(Mask.size() == 4);
     break;
   case X86ISD::MOVSD:
+  case X86ISD::MOVSH:
   case X86ISD::MOVSS: {
     SDValue N0 = N.getOperand(0);
     SDValue N1 = N.getOperand(1);
 
     // Canonicalize scalar FPOps:
     // MOVS*(N0, OP(N0, N1)) --> MOVS*(N0, SCALAR_TO_VECTOR(OP(N0[0], N1[0])))
     // If commutable, allow OP(N1[0], N0[0]).
     unsigned Opcode1 = N1.getOpcode();
if (IsSubAdd)
     return SDValue();
 
   // Do not generate X86ISD::ADDSUB node for 512-bit types even though
   // the ADDSUB idiom has been successfully recognized. There are no known
   // X86 targets with 512-bit ADDSUB instructions!
   if (VT.is512BitVector())
     return SDValue();
 
+  // Do not generate X86ISD::ADDSUB node for FP16's vector types even though
+  // the ADDSUB idiom has been successfully recognized. There are no known
+  // X86 targets with FP16 ADDSUB instructions!
+  if (VT.getVectorElementType() == MVT::f16)
+    return SDValue();
+
   return DAG.getNode(X86ISD::ADDSUB, DL, VT, Opnd0, Opnd1);
 }
 
 // We are looking for a shuffle where both sources are concatenated with undef
 // and have a width that is half of the output's width. AVX2 has VPERMD/Q, so
 // if we can express this as a single-source shuffle, that's preferable.
 static SDValue combineShuffleOfConcatUndef(SDNode *N, SelectionDAG &DAG,
                                            const X86Subtarget &Subtarget) {
switch (N0.getOpcode()) {
     case ISD::OR:  FPOpcode = X86ISD::FOR;  break;
     case ISD::XOR: FPOpcode = X86ISD::FXOR; break;
     default: return SDValue();
   }
 
   // Check if we have a bitcast from another integer type as well.
   if (!((Subtarget.hasSSE1() && VT == MVT::f32) ||
         (Subtarget.hasSSE2() && VT == MVT::f64) ||
+        (Subtarget.hasFP16() && VT == MVT::f16) ||
         (Subtarget.hasSSE2() && VT.isInteger() && VT.isVector() &&
          TLI.isTypeLegal(VT))))
     return SDValue();
 
   SDValue LogicOp0 = N0.getOperand(0);
   SDValue LogicOp1 = N0.getOperand(1);
   SDLoc DL0(N0);
 
if (Vec.getOpcode() == ISD::SETCC && VT == MVT::i1) {
     SDLoc DL(ExtElt);
     SDValue Ext0 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, OpVT,
                                Vec.getOperand(0), Index);
     SDValue Ext1 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, OpVT,
                                Vec.getOperand(1), Index);
     return DAG.getNode(Vec.getOpcode(), DL, VT, Ext0, Ext1, Vec.getOperand(2));
   }
 
-  if (VT != MVT::f32 && VT != MVT::f64)
+  if (VT != MVT::f16 && VT != MVT::f32 && VT != MVT::f64)
     return SDValue();
 
   // Vector FP selects don't fit the pattern of FP math ops (because the
   // condition has a different type and we have to change the opcode), so deal
   // with those here.
   // FIXME: This is restricted to pre type legalization by ensuring the setcc
   // has i1 elements. If we loosen this we need to convert vector bool to a
   // scalar bool.
if (Subtarget.hasSSE2() && isAndOrOfSetCCs(SDValue(N, 0U), opcode)) {
     // The SETCCs should both refer to the same CMP.
     if (CMP0.getOpcode() != X86ISD::FCMP || CMP0 != CMP1)
       return SDValue();
 
     SDValue CMP00 = CMP0->getOperand(0);
     SDValue CMP01 = CMP0->getOperand(1);
     EVT     VT    = CMP00.getValueType();
 
-    if (VT == MVT::f32 || VT == MVT::f64) {
+    if (VT == MVT::f32 || VT == MVT::f64 ||
+        (VT == MVT::f16 && Subtarget.hasFP16())) {
       bool ExpectingFlags = false;
       // Check for any users that want flags:
       for (SDNode::use_iterator UI = N->use_begin(), UE = N->use_end();
            !ExpectingFlags && UI != UE; ++UI)
         switch (UI->getOpcode()) {
         default:
         case ISD::BR_CC:
         case ISD::BRCOND:
if (N0.getOpcode() != ISD::BITCAST || N1.getOpcode() != ISD::BITCAST)
     return SDValue();
 
   SDValue N00 = N0.getOperand(0);
   SDValue N10 = N1.getOperand(0);
   EVT N00Type = N00.getValueType();
   EVT N10Type = N10.getValueType();
 
   // Ensure that both types are the same and are legal scalar fp types.
-  if (N00Type != N10Type ||
-      !((Subtarget.hasSSE1() && N00Type == MVT::f32) ||
-        (Subtarget.hasSSE2() && N00Type == MVT::f64)))
+  if (N00Type != N10Type || !((Subtarget.hasSSE1() && N00Type == MVT::f32) ||
+                              (Subtarget.hasSSE2() && N00Type == MVT::f64) ||
+                              (Subtarget.hasFP16() && N00Type == MVT::f16)))
     return SDValue();
 
   unsigned FPOpcode = convertIntLogicToFPLogicOpcode(N->getOpcode());
   SDValue FPLogic = DAG.getNode(FPOpcode, DL, N00Type, N00, N10);
   return DAG.getBitcast(VT, FPLogic);
 }
 
 // Attempt to fold BITOP(MOVMSK(X),MOVMSK(Y)) -> MOVMSK(BITOP(X,Y))
SDValue X86TargetLowering::PerformDAGCombine(SDNode *N,
   case X86ISD::PSHUFD:
   case X86ISD::PSHUFHW:
   case X86ISD::PSHUFLW:
   case X86ISD::MOVSHDUP:
   case X86ISD::MOVSLDUP:
   case X86ISD::MOVDDUP:
   case X86ISD::MOVSS:
   case X86ISD::MOVSD:
+  case X86ISD::MOVSH:
   case X86ISD::VBROADCAST:
   case X86ISD::VPPERM:
   case X86ISD::VPERMI:
   case X86ISD::VPERMV:
   case X86ISD::VPERMV3:
   case X86ISD::VPERMIL2:
   case X86ISD::VPERMILPI:
   case X86ISD::VPERMILPV:

llvm/lib/Target/X86/X86InstrAVX512.td

class X86VectorVTInfo<int numelts, ValueType eltvt, RegisterClass rc,
   // Suffix used in the instruction mnemonic.
   string Suffix = suffix;
 
   // VTName is a string name for vector VT. For vector types it will be
   // v # NumElts # EltVT, so for vector of 8 elements of i32 it will be v8i32
   // It is a little bit complex for scalar types, where NumElts = 1.
   // In this case we build v4f32 or v2f64
   string VTName = "v" # !if (!eq (NumElts, 1),
+                        !if (!eq (EltVT.Size, 16), 8,
                         !if (!eq (EltVT.Size, 32), 4,
-                        !if (!eq (EltVT.Size, 64), 2, NumElts)), NumElts) # EltVT;
+                        !if (!eq (EltVT.Size, 64), 2, NumElts))), NumElts) # EltVT;
 
   // The vector VT.
   ValueType VT = !cast<ValueType>(VTName);
 
   string EltTypeName = !cast<string>(EltVT);
   // Size of the element type in bits, e.g. 32 for v16i32.
   string EltSizeName = !subst("i", "", !subst("f", "", EltTypeName));
   int EltSize = EltVT.Size;
 
   // "i" for integer types and "f" for floating-point types
   string TypeVariantName = !subst(EltSizeName, "", EltTypeName);
 
   // Size of RC in bits, e.g. 512 for VR512.
   int Size = VT.Size;
 
   // The corresponding memory operand, e.g. i512mem for VR512.
   X86MemOperand MemOp = !cast<X86MemOperand>(TypeVariantName # Size # "mem");
   X86MemOperand ScalarMemOp = !cast<X86MemOperand>(EltVT # "mem");
   // FP scalar memory operand for intrinsics - ssmem/sdmem.
-  Operand IntScalarMemOp = !if (!eq (EltTypeName, "f32"), !cast<Operand>("ssmem"),
-                           !if (!eq (EltTypeName, "f64"), !cast<Operand>("sdmem"), ?));
+  Operand IntScalarMemOp = !if (!eq (EltTypeName, "f16"), !cast<Operand>("shmem"),
+                           !if (!eq (EltTypeName, "f32"), !cast<Operand>("ssmem"),
+                           !if (!eq (EltTypeName, "f64"), !cast<Operand>("sdmem"), ?)));
 
   // Load patterns
   PatFrag LdFrag = !cast<PatFrag>("load" # VTName);
 
   PatFrag AlignedLdFrag = !cast<PatFrag>("alignedload" # VTName);
 
   PatFrag ScalarLdFrag = !cast<PatFrag>("load" # EltVT);
   PatFrag BroadcastLdFrag = !cast<PatFrag>("X86VBroadcastld" # EltSizeName);
 
-  PatFrags ScalarIntMemFrags = !if (!eq (EltTypeName, "f32"),
-                                           !cast<PatFrags>("sse_load_f32"),
+  PatFrags ScalarIntMemFrags = !if (!eq (EltTypeName, "f16"),
+                                           !cast<PatFrags>("sse_load_f16"),
+                               !if (!eq (EltTypeName, "f32"),
                                !if (!eq (EltTypeName, "f64"),
                                      !cast<PatFrags>("sse_load_f64"),
-                               ?));
+                               ?)));
 
   // The string to specify embedded broadcast in assembly.
   string BroadcastStr = "{1to" # NumElts # "}";
 
   // 8-bit compressed displacement tuple/subvector format.  This is only
   // defined for NumElts <= 8.
   CD8VForm CD8TupleForm = !if (!eq (!srl(NumElts, 4), 0),
                                !cast<CD8VForm>("CD8VT" # NumElts), ?);
 
   SubRegIndex SubRegIdx = !if (!eq (Size, 128), sub_xmm,
                           !if (!eq (Size, 256), sub_ymm, ?));
 
   Domain ExeDomain = !if (!eq (EltTypeName, "f32"), SSEPackedSingle,
                      !if (!eq (EltTypeName, "f64"), SSEPackedDouble,
-                     SSEPackedInt));
+                     !if (!eq (EltTypeName, "f16"), SSEPackedSingle, // FIXME?
+                     SSEPackedInt)));
 
-  RegisterClass FRC = !if (!eq (EltTypeName, "f32"), FR32X, FR64X);
+  RegisterClass FRC = !if (!eq (EltTypeName, "f32"), FR32X,
+                      !if (!eq (EltTypeName, "f16"), FR16X,
+                      FR64X));
 
   dag ImmAllZerosV = (VT immAllZerosV);
 
   string ZSuffix = !if (!eq (Size, 128), "Z128",
                    !if (!eq (Size, 256), "Z256", "Z"));
 }
 
 def v64i8_info  : X86VectorVTInfo<64,  i8, VR512, "b">;
 def v32i16_info : X86VectorVTInfo<32, i16, VR512, "w">;
 def v16i32_info : X86VectorVTInfo<16, i32, VR512, "d">;
 def v8i64_info  : X86VectorVTInfo<8,  i64, VR512, "q">;
+def v32f16_info : X86VectorVTInfo<32, f16, VR512, "ph">;
 def v16f32_info : X86VectorVTInfo<16, f32, VR512, "ps">;
 def v8f64_info  : X86VectorVTInfo<8,  f64, VR512, "pd">;
 
 // "x" in v32i8x_info means RC = VR256X
 def v32i8x_info  : X86VectorVTInfo<32,  i8, VR256X, "b">;
 def v16i16x_info : X86VectorVTInfo<16, i16, VR256X, "w">;
 def v8i32x_info  : X86VectorVTInfo<8,  i32, VR256X, "d">;
 def v4i64x_info  : X86VectorVTInfo<4,  i64, VR256X, "q">;
+def v16f16x_info : X86VectorVTInfo<16, f16, VR256X, "ph">;
 def v8f32x_info  : X86VectorVTInfo<8,  f32, VR256X, "ps">;
 def v4f64x_info  : X86VectorVTInfo<4,  f64, VR256X, "pd">;
 
 def v16i8x_info  : X86VectorVTInfo<16,  i8, VR128X, "b">;
 def v8i16x_info  : X86VectorVTInfo<8,  i16, VR128X, "w">;
 def v4i32x_info  : X86VectorVTInfo<4,  i32, VR128X, "d">;
 def v2i64x_info  : X86VectorVTInfo<2,  i64, VR128X, "q">;
+def v8f16x_info  : X86VectorVTInfo<8,  f16, VR128X, "ph">;
 def v4f32x_info  : X86VectorVTInfo<4,  f32, VR128X, "ps">;
 def v2f64x_info  : X86VectorVTInfo<2,  f64, VR128X, "pd">;
 
 // We map scalar types to the smallest (128-bit) vector type
 // with the appropriate element type. This allows to use the same masking logic.
 def i32x_info    : X86VectorVTInfo<1,  i32, GR32, "si">;
 def i64x_info    : X86VectorVTInfo<1,  i64, GR64, "sq">;
+def f16x_info    : X86VectorVTInfo<1,  f16, VR128X, "sh">;
 def f32x_info    : X86VectorVTInfo<1,  f32, VR128X, "ss">;
 def f64x_info    : X86VectorVTInfo<1,  f64, VR128X, "sd">;
 
 class AVX512VLVectorVTInfo<X86VectorVTInfo i512, X86VectorVTInfo i256,
                            X86VectorVTInfo i128> {
   X86VectorVTInfo info512 = i512;
   X86VectorVTInfo info256 = i256;
   X86VectorVTInfo info128 = i128;
 }
 
 def avx512vl_i8_info  : AVX512VLVectorVTInfo<v64i8_info, v32i8x_info,
                                              v16i8x_info>;
 def avx512vl_i16_info : AVX512VLVectorVTInfo<v32i16_info, v16i16x_info,
                                              v8i16x_info>;
 def avx512vl_i32_info : AVX512VLVectorVTInfo<v16i32_info, v8i32x_info,
                                              v4i32x_info>;
 def avx512vl_i64_info : AVX512VLVectorVTInfo<v8i64_info, v4i64x_info,
                                              v2i64x_info>;
+def avx512vl_f16_info : AVX512VLVectorVTInfo<v32f16_info, v16f16x_info,
+                                             v8f16x_info>;
 def avx512vl_f32_info : AVX512VLVectorVTInfo<v16f32_info, v8f32x_info,
                                              v4f32x_info>;
 def avx512vl_f64_info : AVX512VLVectorVTInfo<v8f64_info, v4f64x_info,
                                              v2f64x_info>;
 
 class X86KVectorVTInfo<RegisterClass _krc, RegisterClass _krcwm,
                        ValueType _vt> {
   RegisterClass KRC = _krc;

 def : Pat<(v2f64 immAllZerosV), (AVX512_128_SET0)>;
 def : Pat<(v32i8 immAllZerosV), (AVX512_256_SET0)>;
 def : Pat<(v16i16 immAllZerosV), (AVX512_256_SET0)>;
 def : Pat<(v4i64 immAllZerosV), (AVX512_256_SET0)>;
 def : Pat<(v8f32 immAllZerosV), (AVX512_256_SET0)>;
 def : Pat<(v4f64 immAllZerosV), (AVX512_256_SET0)>;
 }
 
+let Predicates = [HasFP16] in {
+def : Pat<(v8f16 immAllZerosV), (AVX512_128_SET0)>;
+def : Pat<(v16f16 immAllZerosV), (AVX512_256_SET0)>;
+def : Pat<(v32f16 immAllZerosV), (AVX512_512_SET0)>;
+}
+
 // Alias instructions that map fld0 to xorps for sse or vxorps for avx.
 // This is expanded by ExpandPostRAPseudos.
 let isReMaterializable = 1, isAsCheapAsAMove = 1, canFoldAsLoad = 1,
     isPseudo = 1, SchedRW = [WriteZero], Predicates = [HasAVX512] in {
   def AVX512_FsFLD0SS : I<0, Pseudo, (outs FR32X:$dst), (ins), "",
                           [(set FR32X:$dst, fp32imm0)]>;
   def AVX512_FsFLD0SD : I<0, Pseudo, (outs FR64X:$dst), (ins), "",
                           [(set FR64X:$dst, fp64imm0)]>;
   def AVX512_FsFLD0F128 : I<0, Pseudo, (outs VR128X:$dst), (ins), "",
                             [(set VR128X:$dst, fp128imm0)]>;
 }
 
+let isReMaterializable = 1, isAsCheapAsAMove = 1, canFoldAsLoad = 1,
+    isPseudo = 1, SchedRW = [WriteZero], Predicates = [HasFP16] in {
+  def AVX512_FsFLD0SH : I<0, Pseudo, (outs FR16X:$dst), (ins), "",
+                          [(set FR16X:$dst, fp16imm0)]>;
+}
+
 //===----------------------------------------------------------------------===//
 // AVX-512 - VECTOR INSERT
 //
 
 // Supports two different pattern operators for mask and unmasked ops. Allows
 // null_frag to be passed for one.
 multiclass vinsert_for_size_split<int Opcode, X86VectorVTInfo From,
                                   X86VectorVTInfo To,

 defm : vinsert_for_size_lowering<"VINSERTI64x4Z", v8i32x_info, v16i32_info,
               vinsert256_insert, INSERT_get_vinsert256_imm, [HasAVX512]>;
 
 // Codegen pattern with the alternative types insert VEC128 into VEC256
 defm : vinsert_for_size_lowering<"VINSERTI32x4Z256", v8i16x_info, v16i16x_info,
               vinsert128_insert, INSERT_get_vinsert128_imm, [HasVLX]>;
 defm : vinsert_for_size_lowering<"VINSERTI32x4Z256", v16i8x_info, v32i8x_info,
               vinsert128_insert, INSERT_get_vinsert128_imm, [HasVLX]>;
+defm : vinsert_for_size_lowering<"VINSERTF32x4Z256", v8f16x_info, v16f16x_info,
+              vinsert128_insert, INSERT_get_vinsert128_imm, [HasFP16, HasVLX]>;
 // Codegen pattern with the alternative types insert VEC128 into VEC512
 defm : vinsert_for_size_lowering<"VINSERTI32x4Z", v8i16x_info, v32i16_info,
               vinsert128_insert, INSERT_get_vinsert128_imm, [HasAVX512]>;
 defm : vinsert_for_size_lowering<"VINSERTI32x4Z", v16i8x_info, v64i8_info,
                vinsert128_insert, INSERT_get_vinsert128_imm, [HasAVX512]>;
+defm : vinsert_for_size_lowering<"VINSERTF32x4Z", v8f16x_info, v32f16_info,
+              vinsert128_insert, INSERT_get_vinsert128_imm, [HasFP16]>;
 // Codegen pattern with the alternative types insert VEC256 into VEC512
 defm : vinsert_for_size_lowering<"VINSERTI64x4Z", v16i16x_info, v32i16_info,
               vinsert256_insert, INSERT_get_vinsert256_imm, [HasAVX512]>;
 defm : vinsert_for_size_lowering<"VINSERTI64x4Z", v32i8x_info, v64i8_info,
               vinsert256_insert, INSERT_get_vinsert256_imm, [HasAVX512]>;
+defm : vinsert_for_size_lowering<"VINSERTF64x4Z", v16f16x_info, v32f16_info,
+              vinsert256_insert, INSERT_get_vinsert256_imm, [HasFP16]>;
 
 
 multiclass vinsert_for_mask_cast<string InstrStr, X86VectorVTInfo From,
                                  X86VectorVTInfo To, X86VectorVTInfo Cast,
                                  PatFrag vinsert_insert,
                                  SDNodeXForm INSERT_get_vinsert_imm,
                                  list<Predicate> p> {
 let Predicates = p in {

 defm : vextract_for_size_lowering<"VEXTRACTI32x4Z256", v4i64x_info, v2i64x_info,
           vextract128_extract, EXTRACT_get_vextract128_imm, [HasVLX]>;
 
 // Codegen pattern with the alternative types extract VEC128 from VEC256
 defm : vextract_for_size_lowering<"VEXTRACTI32x4Z256", v16i16x_info, v8i16x_info,
           vextract128_extract, EXTRACT_get_vextract128_imm, [HasVLX]>;
 defm : vextract_for_size_lowering<"VEXTRACTI32x4Z256", v32i8x_info, v16i8x_info,
           vextract128_extract, EXTRACT_get_vextract128_imm, [HasVLX]>;
+defm : vextract_for_size_lowering<"VEXTRACTF32x4Z256", v16f16x_info, v8f16x_info,
+          vextract128_extract, EXTRACT_get_vextract128_imm, [HasFP16, HasVLX]>;
 
 // Codegen pattern with the alternative types extract VEC128 from VEC512
 defm : vextract_for_size_lowering<"VEXTRACTI32x4Z", v32i16_info, v8i16x_info,
                  vextract128_extract, EXTRACT_get_vextract128_imm, [HasAVX512]>;
 defm : vextract_for_size_lowering<"VEXTRACTI32x4Z", v64i8_info, v16i8x_info,
                  vextract128_extract, EXTRACT_get_vextract128_imm, [HasAVX512]>;
+defm : vextract_for_size_lowering<"VEXTRACTF32x4Z", v32f16_info, v8f16x_info,
+                 vextract128_extract, EXTRACT_get_vextract128_imm, [HasFP16]>;
 // Codegen pattern with the alternative types extract VEC256 from VEC512
 defm : vextract_for_size_lowering<"VEXTRACTI64x4Z", v32i16_info, v16i16x_info,
                  vextract256_extract, EXTRACT_get_vextract256_imm, [HasAVX512]>;
 defm : vextract_for_size_lowering<"VEXTRACTI64x4Z", v64i8_info, v32i8x_info,
                  vextract256_extract, EXTRACT_get_vextract256_imm, [HasAVX512]>;
+defm : vextract_for_size_lowering<"VEXTRACTF64x4Z", v32f16_info, v16f16x_info,
+                 vextract256_extract, EXTRACT_get_vextract256_imm, [HasFP16]>;
 
 
 // A 128-bit extract from bits [255:128] of a 512-bit vector should use a
 // smaller extract to enable EVEX->VEX.
 let Predicates = [NoVLX] in {
 def : Pat<(v2i64 (extract_subvector (v8i64 VR512:$src), (iPTR 2))),
           (v2i64 (VEXTRACTI128rr
                   (v4i64 (EXTRACT_SUBREG (v8i64 VR512:$src), sub_ymm)),
def : Pat<(v8i16 (extract_subvector (v32i16 VR512:$src), (iPTR 8))),
                   (v16i16 (EXTRACT_SUBREG (v32i16 VR512:$src), sub_ymm)),
                   (iPTR 1)))>;
 def : Pat<(v16i8 (extract_subvector (v64i8 VR512:$src), (iPTR 16))),
           (v16i8 (VEXTRACTI32x4Z256rr
                   (v32i8 (EXTRACT_SUBREG (v64i8 VR512:$src), sub_ymm)),
                   (iPTR 1)))>;
 }
 
+let Predicates = [HasFP16, HasVLX] in
+def : Pat<(v8f16 (extract_subvector (v32f16 VR512:$src), (iPTR 8))),
+          (v8f16 (VEXTRACTF32x4Z256rr
+                  (v16f16 (EXTRACT_SUBREG (v32f16 VR512:$src), sub_ymm)),
+                  (iPTR 1)))>;
+
 
 // Additional patterns for handling a bitcast between the vselect and the
 // extract_subvector.
 multiclass vextract_for_mask_cast<string InstrStr, X86VectorVTInfo From,
                                   X86VectorVTInfo To, X86VectorVTInfo Cast,
                                   PatFrag vextract_extract,
                                   SDNodeXForm EXTRACT_get_vextract_imm,
                                   list<Predicate> p> {
multiclass avx512_subvec_broadcast_rm_dq<bits<8> opc, string OpcodeStr,
   let hasSideEffects = 0, mayLoad = 1 in
   defm rm : AVX512_maskable_split<opc, MRMSrcMem, _Dst, (outs _Dst.RC:$dst),
                            (ins _Src.MemOp:$src), OpcodeStr, "$src", "$src",
                            (null_frag),
                            (_Dst.VT (OpNode addr:$src))>,
                            Sched<[SchedWriteShuffle.YMM.Folded]>,
                            AVX5128IBase, EVEX;
 }
+let Predicates = [HasFP16] in {
+  def : Pat<(v32f16 (X86VBroadcastld16 addr:$src)),
+            (VPBROADCASTWZrm addr:$src)>;
+
+  def : Pat<(v32f16 (X86VBroadcast (v8f16 VR128X:$src))),
+            (VPBROADCASTWZrr VR128X:$src)>;
+  def : Pat<(v32f16 (X86VBroadcast (f16 FR16X:$src))),
+            (VPBROADCASTWZrr (COPY_TO_REGCLASS FR16X:$src, VR128X))>;
+}
+let Predicates = [HasVLX, HasFP16] in {
+  def : Pat<(v8f16 (X86VBroadcastld16 addr:$src)),
+            (VPBROADCASTWZ128rm addr:$src)>;
+  def : Pat<(v16f16 (X86VBroadcastld16 addr:$src)),
+            (VPBROADCASTWZ256rm addr:$src)>;
+
+  def : Pat<(v8f16 (X86VBroadcast (v8f16 VR128X:$src))),
+            (VPBROADCASTWZ128rr VR128X:$src)>;
+  def : Pat<(v16f16 (X86VBroadcast (v8f16 VR128X:$src))),
+            (VPBROADCASTWZ256rr VR128X:$src)>;
+
+  def : Pat<(v8f16 (X86VBroadcast (f16 FR16X:$src))),
+            (VPBROADCASTWZ128rr (COPY_TO_REGCLASS FR16X:$src, VR128X))>;
+  def : Pat<(v16f16 (X86VBroadcast (f16 FR16X:$src))),
+            (VPBROADCASTWZ256rr (COPY_TO_REGCLASS FR16X:$src, VR128X))>;
+}
 
 //===----------------------------------------------------------------------===//
 // AVX-512 BROADCAST SUBVECTORS
 //
 
 defm VBROADCASTI32X4 : avx512_subvec_broadcast_rm<0x5a, "vbroadcasti32x4",
                        X86SubVBroadcastld128, v16i32_info, v4i32x_info>,
                        EVEX_V512, EVEX_CD8<32, CD8VT4>;
 defm VBROADCASTF32X4 : avx512_subvec_broadcast_rm<0x1a, "vbroadcastf32x4",
                        X86SubVBroadcastld128, v16f32_info, v4f32x_info>,
                        EVEX_V512, EVEX_CD8<32, CD8VT4>;
 defm VBROADCASTI64X4 : avx512_subvec_broadcast_rm<0x5b, "vbroadcasti64x4",
                        X86SubVBroadcastld256, v8i64_info, v4i64x_info>, VEX_W,
                        EVEX_V512, EVEX_CD8<64, CD8VT4>;
 defm VBROADCASTF64X4 : avx512_subvec_broadcast_rm<0x1b, "vbroadcastf64x4",
                        X86SubVBroadcastld256, v8f64_info, v4f64x_info>, VEX_W,
                        EVEX_V512, EVEX_CD8<64, CD8VT4>;
 
 let Predicates = [HasAVX512] in {
 def : Pat<(v8f64 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTF64X4rm addr:$src)>;
 def : Pat<(v16f32 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTF64X4rm addr:$src)>;
+def : Pat<(v32f16 (X86SubVBroadcastld256 addr:$src)),
+          (VBROADCASTF64X4rm addr:$src)>;
 def : Pat<(v8i64 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTI64X4rm addr:$src)>;
 def : Pat<(v16i32 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTI64X4rm addr:$src)>;
 def : Pat<(v32i16 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTI64X4rm addr:$src)>;
 def : Pat<(v64i8 (X86SubVBroadcastld256 addr:$src)),
           (VBROADCASTI64X4rm addr:$src)>;
 
 def : Pat<(v8f64 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTF32X4rm addr:$src)>;
 def : Pat<(v16f32 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTF32X4rm addr:$src)>;
+def : Pat<(v32f16 (X86SubVBroadcastld128 addr:$src)),
+          (VBROADCASTF32X4rm addr:$src)>;
 def : Pat<(v8i64 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4rm addr:$src)>;
 def : Pat<(v16i32 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4rm addr:$src)>;
 def : Pat<(v32i16 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4rm addr:$src)>;
 def : Pat<(v64i8 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4rm addr:$src)>;

 defm VBROADCASTF32X4Z256 : avx512_subvec_broadcast_rm<0x1a, "vbroadcastf32x4",
                            X86SubVBroadcastld128, v8f32x_info, v4f32x_info>,
                            EVEX_V256, EVEX_CD8<32, CD8VT4>;
 
 def : Pat<(v4f64 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTF32X4Z256rm addr:$src)>;
 def : Pat<(v8f32 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTF32X4Z256rm addr:$src)>;
+def : Pat<(v16f16 (X86SubVBroadcastld128 addr:$src)),
+          (VBROADCASTF32X4Z256rm addr:$src)>;
 def : Pat<(v4i64 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4Z256rm addr:$src)>;
 def : Pat<(v8i32 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4Z256rm addr:$src)>;
 def : Pat<(v16i16 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4Z256rm addr:$src)>;
 def : Pat<(v32i8 (X86SubVBroadcastld128 addr:$src)),
           (VBROADCASTI32X4Z256rm addr:$src)>;
def : Pat<(alignedstore (v32i8 VR256X:$src), addr:$dst),
             (VMOVDQA64Z256mr addr:$dst, VR256X:$src)>;
   def : Pat<(store (v8i32 VR256X:$src), addr:$dst),
             (VMOVDQU64Z256mr addr:$dst, VR256X:$src)>;
   def : Pat<(store (v16i16 VR256X:$src), addr:$dst),
             (VMOVDQU64Z256mr addr:$dst, VR256X:$src)>;
   def : Pat<(store (v32i8 VR256X:$src), addr:$dst),
             (VMOVDQU64Z256mr addr:$dst, VR256X:$src)>;
 }
+let Predicates = [HasFP16] in {
+  def : Pat<(v32f16 (vselect VK32WM:$mask, (v32f16 VR512:$src1), (v32f16 VR512:$src0))),
+            (VMOVDQU16Zrrk VR512:$src0, VK32WM:$mask, VR512:$src1)>;
+  def : Pat<(v32f16 (vselect VK32WM:$mask, (v32f16 VR512:$src1), v32f16_info.ImmAllZerosV)),
+            (VMOVDQU16Zrrkz VK32WM:$mask, VR512:$src1)>;
+  def : Pat<(v32f16 (alignedloadv32f16 addr:$src)),
+            (VMOVAPSZrm addr:$src)>;
+  def : Pat<(v32f16 (vselect VK32WM:$mask,
+                     (v32f16 (alignedloadv32f16 addr:$src)), (v32f16 VR512:$src0))),
+            (VMOVDQU16Zrmk VR512:$src0, VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (vselect VK32WM:$mask,
+                     (v32f16 (alignedloadv32f16 addr:$src)), v32f16_info.ImmAllZerosV)),
+            (VMOVDQU16Zrmkz VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (loadv32f16 addr:$src)),
+            (VMOVUPSZrm addr:$src)>;
+  def : Pat<(v32f16 (vselect VK32WM:$mask,
+                     (v32f16 (loadv32f16 addr:$src)), (v32f16 VR512:$src0))),
+            (VMOVDQU16Zrmk VR512:$src0, VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (vselect VK32WM:$mask,
+                     (v32f16 (loadv32f16 addr:$src)), v32f16_info.ImmAllZerosV)),
+            (VMOVDQU16Zrmkz VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (masked_load addr:$src, VK32WM:$mask, (v32f16 VR512:$src0))),
+            (VMOVDQU16Zrmk VR512:$src0, VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (masked_load addr:$src, VK32WM:$mask, undef)),
+            (VMOVDQU16Zrmkz VK32WM:$mask, addr:$src)>;
+  def : Pat<(v32f16 (masked_load addr:$src, VK32WM:$mask, v32f16_info.ImmAllZerosV)),
+            (VMOVDQU16Zrmkz VK32WM:$mask, addr:$src)>;
+
+  def : Pat<(alignedstore (v32f16 VR512:$src), addr:$dst),
+            (VMOVAPSZmr addr:$dst, VR512:$src)>;
+  def : Pat<(store (v32f16 VR512:$src), addr:$dst),
+            (VMOVUPSZmr addr:$dst, VR512:$src)>;
+  def : Pat<(masked_store (v32f16 VR512:$src), addr:$dst, VK32WM:$mask),
+            (VMOVDQU16Zmrk addr:$dst, VK32WM:$mask, VR512:$src)>;
+}
+let Predicates = [HasFP16, HasVLX] in {
+  def : Pat<(v16f16 (vselect VK16WM:$mask, (v16f16 VR256X:$src1), (v16f16 VR256X:$src0))),
+            (VMOVDQU16Z256rrk VR256X:$src0, VK16WM:$mask, VR256X:$src1)>;
+  def : Pat<(v16f16 (vselect VK16WM:$mask, (v16f16 VR256X:$src1), v16f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z256rrkz VK16WM:$mask, VR256X:$src1)>;
+  def : Pat<(v16f16 (alignedloadv16f16 addr:$src)),
+            (VMOVAPSZ256rm addr:$src)>;
+  def : Pat<(v16f16 (vselect VK16WM:$mask,
+                     (v16f16 (alignedloadv16f16 addr:$src)), (v16f16 VR256X:$src0))),
+            (VMOVDQU16Z256rmk VR256X:$src0, VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (vselect VK16WM:$mask,
+                     (v16f16 (alignedloadv16f16 addr:$src)), v16f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z256rmkz VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (loadv16f16 addr:$src)),
+            (VMOVUPSZ256rm addr:$src)>;
+  def : Pat<(v16f16 (vselect VK16WM:$mask,
+                     (v16f16 (loadv16f16 addr:$src)), (v16f16 VR256X:$src0))),
+            (VMOVDQU16Z256rmk VR256X:$src0, VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (vselect VK16WM:$mask,
+                     (v16f16 (loadv16f16 addr:$src)), v16f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z256rmkz VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (masked_load addr:$src, VK16WM:$mask, (v16f16 VR256X:$src0))),
+            (VMOVDQU16Z256rmk VR256X:$src0, VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (masked_load addr:$src, VK16WM:$mask, undef)),
+            (VMOVDQU16Z256rmkz VK16WM:$mask, addr:$src)>;
+  def : Pat<(v16f16 (masked_load addr:$src, VK16WM:$mask, v16f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z256rmkz VK16WM:$mask, addr:$src)>;
+
+  def : Pat<(alignedstore (v16f16 VR256X:$src), addr:$dst),
+            (VMOVAPSZ256mr addr:$dst, VR256X:$src)>;
+  def : Pat<(store (v16f16 VR256X:$src), addr:$dst),
+            (VMOVUPSZ256mr addr:$dst, VR256X:$src)>;
+  def : Pat<(masked_store (v16f16 VR256X:$src), addr:$dst, VK16WM:$mask),
+            (VMOVDQU16Z256mrk addr:$dst, VK16WM:$mask, VR256X:$src)>;
+
+  def : Pat<(v8f16 (vselect VK8WM:$mask, (v8f16 VR128X:$src1), (v8f16 VR128X:$src0))),
+            (VMOVDQU16Z128rrk VR128X:$src0, VK8WM:$mask, VR128X:$src1)>;
+  def : Pat<(v8f16 (vselect VK8WM:$mask, (v8f16 VR128X:$src1), v8f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z128rrkz VK8WM:$mask, VR128X:$src1)>;
+  def : Pat<(v8f16 (alignedloadv8f16 addr:$src)),
+            (VMOVAPSZ128rm addr:$src)>;
+  def : Pat<(v8f16 (vselect VK8WM:$mask,
+                     (v8f16 (alignedloadv8f16 addr:$src)), (v8f16 VR128X:$src0))),
+            (VMOVDQU16Z128rmk VR128X:$src0, VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (vselect VK8WM:$mask,
+                     (v8f16 (alignedloadv8f16 addr:$src)), v8f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z128rmkz VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (loadv8f16 addr:$src)),
+            (VMOVUPSZ128rm addr:$src)>;
+  def : Pat<(v8f16 (vselect VK8WM:$mask,
+                     (v8f16 (loadv8f16 addr:$src)), (v8f16 VR128X:$src0))),
+            (VMOVDQU16Z128rmk VR128X:$src0, VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (vselect VK8WM:$mask,
+                     (v8f16 (loadv8f16 addr:$src)), v8f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z128rmkz VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (masked_load addr:$src, VK8WM:$mask, (v8f16 VR128X:$src0))),
+            (VMOVDQU16Z128rmk VR128X:$src0, VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (masked_load addr:$src, VK8WM:$mask, undef)),
+            (VMOVDQU16Z128rmkz VK8WM:$mask, addr:$src)>;
+  def : Pat<(v8f16 (masked_load addr:$src, VK8WM:$mask, v8f16x_info.ImmAllZerosV)),
+            (VMOVDQU16Z128rmkz VK8WM:$mask, addr:$src)>;
+
+  def : Pat<(alignedstore (v8f16 VR128X:$src), addr:$dst),
+            (VMOVAPSZ128mr addr:$dst, VR128X:$src)>;
+  def : Pat<(store (v8f16 VR128X:$src), addr:$dst),
+            (VMOVUPSZ128mr addr:$dst, VR128X:$src)>;
+  def : Pat<(masked_store (v8f16 VR128X:$src), addr:$dst, VK8WM:$mask),
+            (VMOVDQU16Z128mrk addr:$dst, VK8WM:$mask, VR128X:$src)>;
+}
 
 // Move Int Doubleword to Packed Double Int
 //
 let ExeDomain = SSEPackedInt in {
 def VMOVDI2PDIZrr : AVX512BI<0x6E, MRMSrcReg, (outs VR128X:$dst), (ins GR32:$src),
                       "vmovd\t{$src, $dst|$dst, $src}",
                       [(set VR128X:$dst,
                         (v4i32 (scalar_to_vector GR32:$src)))]>,
def : Pat<(f32 (bitconvert VK32:$src)),
           (VMOVDI2SSZrr (KMOVDrk VK32:$src))>;
 
 def : Pat<(v64i1 (bitconvert FR64X:$src)),
           (KMOVQkr (VMOVSDto64Zrr FR64X:$src))>;
 def : Pat<(f64 (bitconvert VK64:$src)),
           (VMOV64toSDZrr (KMOVQrk VK64:$src))>;
 
 //===----------------------------------------------------------------------===//
-// AVX-512  MOVSS, MOVSD
+// AVX-512  MOVSH, MOVSS, MOVSD
 //===----------------------------------------------------------------------===//
 
 multiclass avx512_move_scalar<string asm, SDNode OpNode, PatFrag vzload_frag,
-                              X86VectorVTInfo _> {
-  let Predicates = [HasAVX512, OptForSize] in
+                              X86VectorVTInfo _,
+                              list<Predicate> prd = [HasAVX512, OptForSize]> {
+  let Predicates = prd in
   def rr : AVX512PI<0x10, MRMSrcReg, (outs _.RC:$dst),
              (ins _.RC:$src1, _.RC:$src2),
              !strconcat(asm, "\t{$src2, $src1, $dst|$dst, $src1, $src2}"),
              [(set _.RC:$dst, (_.VT (OpNode _.RC:$src1, _.RC:$src2)))],
              _.ExeDomain>, EVEX_4V, Sched<[SchedWriteFShuffle.XMM]>;
   def rrkz : AVX512PI<0x10, MRMSrcReg, (outs _.RC:$dst),
               (ins _.KRCWM:$mask, _.RC:$src1, _.RC:$src2),
               !strconcat(asm, "\t{$src2, $src1, $dst {${mask}} {z}|",

 }
 
 defm VMOVSSZ : avx512_move_scalar<"vmovss", X86Movss, X86vzload32, f32x_info>,
                                   VEX_LIG, XS, EVEX_CD8<32, CD8VT1>;
 
 defm VMOVSDZ : avx512_move_scalar<"vmovsd", X86Movsd, X86vzload64, f64x_info>,
                                   VEX_LIG, XD, VEX_W, EVEX_CD8<64, CD8VT1>;
 
+defm VMOVSHZ : avx512_move_scalar<"vmovsh", X86Movsh, X86vzload16, f16x_info,
+                                  [HasFP16]>,
+                                  VEX_LIG, T_MAP5XS, EVEX_CD8<16, CD8VT1>;
 
 multiclass avx512_move_scalar_lowering<string InstrStr, SDNode OpNode,
                                        PatLeaf ZeroFP, X86VectorVTInfo _> {
 
 def : Pat<(_.VT (OpNode _.RC:$src0,
                         (_.VT (scalar_to_vector
                                   (_.EltVT (X86selects VK1WM:$mask,
                                                        (_.EltVT _.FRC:$src1),

 
 def : Pat<(_.info128.VT (masked_load addr:$srcAddr, Mask128,
                          (_.info128.VT (X86vzmovl _.info128.RC:$src)))),
           (!cast<Instruction>(InstrStr#rmk) _.info128.RC:$src,
                       (COPY_TO_REGCLASS (i32 (INSERT_SUBREG (IMPLICIT_DEF), MaskRC:$mask, subreg)), VK1WM),
                       addr:$srcAddr)>;
 }
 
+defm : avx512_move_scalar_lowering<"VMOVSHZ", X86Movsh, fp16imm0, v8f16x_info>;
 defm : avx512_move_scalar_lowering<"VMOVSSZ", X86Movss, fp32imm0, v4f32x_info>;
 defm : avx512_move_scalar_lowering<"VMOVSDZ", X86Movsd, fp64imm0, v2f64x_info>;
 
+defm : avx512_store_scalar_lowering<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (bitconvert (and GR32:$mask, (i32 1)))), GR32>;
+defm : avx512_store_scalar_lowering_subreg<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (bitconvert (and GR32:$mask, (i32 1)))), GR32, sub_32bit>;
 defm : avx512_store_scalar_lowering<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (bitconvert (i16 (trunc (and GR32:$mask, (i32 1)))))), GR32>;
 defm : avx512_store_scalar_lowering_subreg<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (bitconvert (i16 (and GR16:$mask, (i16 1))))), GR16, sub_16bit>;
 defm : avx512_store_scalar_lowering_subreg<"VMOVSDZ", avx512vl_f64_info,
                    (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))), GR8, sub_8bit>;
 
+defm : avx512_store_scalar_lowering_subreg2<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (insert_subvector
+                           (v32i1 immAllZerosV),
+                           (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
+                           (iPTR 0))),
+                   (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
+                   GR8, sub_8bit>;
 defm : avx512_store_scalar_lowering_subreg2<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (insert_subvector
                            (v16i1 immAllZerosV),
                            (v4i1 (extract_subvector
                                   (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
                                   (iPTR 0))),
                            (iPTR 0))),
                    (v4i1 (extract_subvector
defm : avx512_store_scalar_lowering_subreg2<"VMOVSDZ", avx512vl_f64_info,
                               (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))),
                               (iPTR 0))),
                        (iPTR 0))),
                      (iPTR 0))),
                    (v2i1 (extract_subvector
                           (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))),
                           (iPTR 0))), GR8, sub_8bit>;
 
+defm : avx512_load_scalar_lowering<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (bitconvert (and GR32:$mask, (i32 1)))), GR32>;
+defm : avx512_load_scalar_lowering_subreg<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (bitconvert (and GR32:$mask, (i32 1)))), GR32, sub_32bit>;
 defm : avx512_load_scalar_lowering<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (bitconvert (i16 (trunc (and GR32:$mask, (i32 1)))))), GR32>;
 defm : avx512_load_scalar_lowering_subreg<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (bitconvert (i16 (and GR16:$mask, (i16 1))))), GR16, sub_16bit>;
 defm : avx512_load_scalar_lowering_subreg<"VMOVSDZ", avx512vl_f64_info,
                    (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))), GR8, sub_8bit>;
 
+defm : avx512_load_scalar_lowering_subreg2<"VMOVSHZ", avx512vl_f16_info,
+                   (v32i1 (insert_subvector
+                           (v32i1 immAllZerosV),
+                           (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
+                           (iPTR 0))),
+                   (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
+                   GR8, sub_8bit>;
 defm : avx512_load_scalar_lowering_subreg2<"VMOVSSZ", avx512vl_f32_info,
                    (v16i1 (insert_subvector
                            (v16i1 immAllZerosV),
                            (v4i1 (extract_subvector
                                   (v8i1 (bitconvert (and GR8:$mask, (i8 1)))),
                                   (iPTR 0))),
                            (iPTR 0))),
                    (v4i1 (extract_subvector
defm : avx512_load_scalar_lowering_subreg2<"VMOVSDZ", avx512vl_f64_info,
                               (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))),
                               (iPTR 0))),
                        (iPTR 0))),
                      (iPTR 0))),
                    (v2i1 (extract_subvector
                           (v8i1 (bitconvert (i8 (and GR8:$mask, (i8 1))))),
                           (iPTR 0))), GR8, sub_8bit>;
 
+def : Pat<(f16 (X86selects VK1WM:$mask, (f16 FR16X:$src1), (f16 FR16X:$src2))),
+          (COPY_TO_REGCLASS (v8f16 (VMOVSHZrrk
+           (v8f16 (COPY_TO_REGCLASS FR16X:$src2, VR128X)),
+           VK1WM:$mask, (v8f16 (IMPLICIT_DEF)),
+           (v8f16 (COPY_TO_REGCLASS FR16X:$src1, VR128X)))), FR16X)>;
+
+def : Pat<(f16 (X86selects VK1WM:$mask, (f16 FR16X:$src1), fp16imm0)),
+          (COPY_TO_REGCLASS (v8f16 (VMOVSHZrrkz VK1WM:$mask, (v8f16 (IMPLICIT_DEF)),
+           (v8f16 (COPY_TO_REGCLASS FR16X:$src1, VR128X)))), FR16X)>;
+
 def : Pat<(f32 (X86selects VK1WM:$mask, (f32 FR32X:$src1), (f32 FR32X:$src2))),
           (COPY_TO_REGCLASS (v4f32 (VMOVSSZrrk
            (v4f32 (COPY_TO_REGCLASS FR32X:$src2, VR128X)),
            VK1WM:$mask, (v4f32 (IMPLICIT_DEF)),
            (v4f32 (COPY_TO_REGCLASS FR32X:$src1, VR128X)))), FR32X)>;
 
 def : Pat<(f32 (X86selects VK1WM:$mask, (f32 FR32X:$src1), fp32imm0)),
           (COPY_TO_REGCLASS (v4f32 (VMOVSSZrrkz VK1WM:$mask, (v4f32 (IMPLICIT_DEF)),
def : Pat<(v2f64 (X86selects VK1WM:$mask, (v2f64 VR128X:$src1), (v2f64 VR128X:$src2))),
           (VMOVSDZrrk VR128X:$src2, VK1WM:$mask, VR128X:$src1, VR128X:$src1)>;
 
 def : Pat<(v4f32 (X86selects VK1WM:$mask, (v4f32 VR128X:$src1), (v4f32 immAllZerosV))),
           (VMOVSSZrrkz VK1WM:$mask, VR128X:$src1, VR128X:$src1)>;
 def : Pat<(v2f64 (X86selects VK1WM:$mask, (v2f64 VR128X:$src1), (v2f64 immAllZerosV))),
           (VMOVSDZrrkz VK1WM:$mask, VR128X:$src1, VR128X:$src1)>;
 
 let hasSideEffects = 0, isCodeGenOnly = 1, ForceDisassemble = 1 in {
+  let Predicates = [HasFP16] in {
+    def VMOVSHZrr_REV: AVX512<0x11, MRMDestReg, (outs VR128X:$dst),
+        (ins VR128X:$src1, VR128X:$src2),
+        "vmovsh\t{$src2, $src1, $dst|$dst, $src1, $src2}",
+        []>, T_MAP5XS, EVEX_4V, VEX_LIG,
+        FoldGenData<"VMOVSHZrr">,
+        Sched<[SchedWriteFShuffle.XMM]>;
+
+    let Constraints = "$src0 = $dst" in
+    def VMOVSHZrrk_REV: AVX512<0x11, MRMDestReg, (outs VR128X:$dst),
+        (ins f16x_info.RC:$src0, f16x_info.KRCWM:$mask,
+         VR128X:$src1, VR128X:$src2),
+        "vmovsh\t{$src2, $src1, $dst {${mask}}|"#
+          "$dst {${mask}}, $src1, $src2}",
+        []>, T_MAP5XS, EVEX_K, EVEX_4V, VEX_LIG,
+        FoldGenData<"VMOVSHZrrk">,
+        Sched<[SchedWriteFShuffle.XMM]>;
+
+    def VMOVSHZrrkz_REV: AVX512<0x11, MRMDestReg, (outs VR128X:$dst),
+        (ins f16x_info.KRCWM:$mask, VR128X:$src1, VR128X:$src2),
+        "vmovsh\t{$src2, $src1, $dst {${mask}} {z}|"#
+          "$dst {${mask}} {z}, $src1, $src2}",
+        []>, EVEX_KZ, T_MAP5XS, EVEX_4V, VEX_LIG,
+        FoldGenData<"VMOVSHZrrkz">,
+        Sched<[SchedWriteFShuffle.XMM]>;
+  }
   def VMOVSSZrr_REV: AVX512<0x11, MRMDestReg, (outs VR128X:$dst),
                            (ins VR128X:$src1, VR128X:$src2),
                            "vmovss\t{$src2, $src1, $dst|$dst, $src1, $src2}",
                            []>, XS, EVEX_4V, VEX_LIG,
                            FoldGenData<"VMOVSSZrr">,
                            Sched<[SchedWriteFShuffle.XMM]>;
 
   let Constraints = "$src0 = $dst" in
def VMOVSDZrrkz_REV: AVX512<0x11, MRMDestReg, (outs VR128X:$dst),
                                                           VR128X:$src2),
                               "vmovsd\t{$src2, $src1, $dst {${mask}} {z}|"#
                                          "$dst {${mask}} {z}, $src1, $src2}",
                               []>, EVEX_KZ, XD, EVEX_4V, VEX_LIG,
                               VEX_W, FoldGenData<"VMOVSDZrrkz">,
                               Sched<[SchedWriteFShuffle.XMM]>;
 }
 
+def : InstAlias<"vmovsh.s\t{$src2, $src1, $dst|$dst, $src1, $src2}",
+                (VMOVSHZrr_REV VR128X:$dst, VR128X:$src1, VR128X:$src2), 0>;
+def : InstAlias<"vmovsh.s\t{$src2, $src1, $dst {${mask}}|"#
+                             "$dst {${mask}}, $src1, $src2}",
+                (VMOVSHZrrk_REV VR128X:$dst, VK1WM:$mask,
+                                VR128X:$src1, VR128X:$src2), 0>;
+def : InstAlias<"vmovsh.s\t{$src2, $src1, $dst {${mask}} {z}|"#
+                             "$dst {${mask}} {z}, $src1, $src2}",
+                (VMOVSHZrrkz_REV VR128X:$dst, VK1WM:$mask,
+                                 VR128X:$src1, VR128X:$src2), 0>;
 def : InstAlias<"vmovss.s\t{$src2, $src1, $dst|$dst, $src1, $src2}",
                 (VMOVSSZrr_REV VR128X:$dst, VR128X:$src1, VR128X:$src2), 0>;
 def : InstAlias<"vmovss.s\t{$src2, $src1, $dst {${mask}}|"#
                              "$dst {${mask}}, $src1, $src2}",
                 (VMOVSSZrrk_REV VR128X:$dst, VK1WM:$mask,
                                 VR128X:$src1, VR128X:$src2), 0>;
 def : InstAlias<"vmovss.s\t{$src2, $src1, $dst {${mask}} {z}|"#
                              "$dst {${mask}} {z}, $src1, $src2}",
let Predicates = [HasAVX512] in {
 
   // Represent the same patterns above but in the form they appear for
   // 512-bit types
   def : Pat<(v16f32 (X86vzload32 addr:$src)),
             (SUBREG_TO_REG (i32 0), (VMOVSSZrm addr:$src), sub_xmm)>;
   def : Pat<(v8f64 (X86vzload64 addr:$src)),
             (SUBREG_TO_REG (i32 0), (VMOVSDZrm addr:$src), sub_xmm)>;
 }
+let Predicates = [HasFP16] in {
+  def : Pat<(v8f16 (X86vzmovl (v8f16 VR128X:$src))),
+            (VMOVSHZrr (v8f16 (AVX512_128_SET0)), VR128X:$src)>;
+
+  // FIXME we need better canonicalization in dag combine
+  def : Pat<(v16f16 (X86vzmovl (v16f16 VR256X:$src))),
+            (SUBREG_TO_REG (i32 0),
+             (v8f16 (VMOVSHZrr (v8f16 (AVX512_128_SET0)),
+              (v8f16 (EXTRACT_SUBREG (v16f16 VR256X:$src), sub_xmm)))), sub_xmm)>;
+  def : Pat<(v32f16 (X86vzmovl (v32f16 VR512:$src))),
+            (SUBREG_TO_REG (i32 0),
+             (v8f16 (VMOVSHZrr (v8f16 (AVX512_128_SET0)),
+              (v8f16 (EXTRACT_SUBREG (v32f16 VR512:$src), sub_xmm)))), sub_xmm)>;
+
+  def : Pat<(v8f16 (X86vzload16 addr:$src)),
+            (VMOVSHZrm addr:$src)>;
+
+  def : Pat<(v16f16 (X86vzload16 addr:$src)),
+            (SUBREG_TO_REG (i32 0), (VMOVSHZrm addr:$src), sub_xmm)>;
+
+  def : Pat<(v32f16 (X86vzload16 addr:$src)),
+            (SUBREG_TO_REG (i32 0), (VMOVSHZrm addr:$src), sub_xmm)>;
+}
 
 let ExeDomain = SSEPackedInt, SchedRW = [SchedWriteVecLogic.XMM] in {
 def VMOVZPQILo2PQIZrr : AVX512XSI<0x7E, MRMSrcReg, (outs VR128X:$dst),
                                 (ins VR128X:$src),
                                 "vmovq\t{$src, $dst|$dst, $src}",
                                 [(set VR128X:$dst, (v2i64 (X86vzmovl
                                                    (v2i64 VR128X:$src))))]>,
                                 EVEX, VEX_W;
defm Z128 : avx512_dpbf16ps_rm<opc, OpcodeStr, OpNode, sched.XMM, _.info128,
                                    src_v.info128>, EVEX_V128;
   }
 }
 
 let ExeDomain = SSEPackedSingle in
 defm VDPBF16PS : avx512_dpbf16ps_sizes<0x52, "vdpbf16ps", X86dpbf16ps, SchedWriteFMA,
                                        avx512vl_f32_info, avx512vl_i32_info,
                                        HasBF16>, T8XS, EVEX_CD8<32, CD8VF>;
+
+//===----------------------------------------------------------------------===//
+// AVX512FP16
+//===----------------------------------------------------------------------===//
+
+let Predicates = [HasFP16] in {
+// Move word ( r/m16) to Packed word
+def VMOVW2SHrr : AVX512<0x6E, MRMSrcReg, (outs VR128X:$dst), (ins GR32:$src),
+                      "vmovw\t{$src, $dst|$dst, $src}", []>, T_MAP5PD, EVEX, Sched<[WriteVecMoveFromGpr]>;
+def VMOVWrm : AVX512<0x6E, MRMSrcMem, (outs VR128X:$dst), (ins i16mem:$src),
+                      "vmovw\t{$src, $dst|$dst, $src}",
+                      [(set VR128X:$dst,
+                        (v8i16 (scalar_to_vector (loadi16 addr:$src))))]>,
+                      T_MAP5PD, EVEX, EVEX_CD8<16, CD8VT1>, Sched<[WriteFLoad]>;
+
+def : Pat<(f16 (bitconvert GR16:$src)),
+          (f16 (COPY_TO_REGCLASS
+                (VMOVW2SHrr
+                 (INSERT_SUBREG (IMPLICIT_DEF), GR16:$src, sub_16bit)),
+                FR16X))>;
+def : Pat<(v8i16 (scalar_to_vector (i16 GR16:$src))),
+          (VMOVW2SHrr (INSERT_SUBREG (IMPLICIT_DEF), GR16:$src, sub_16bit))>;
+def : Pat<(v4i32 (X86vzmovl (scalar_to_vector (and GR32:$src, 0xffff)))),
+          (VMOVW2SHrr GR32:$src)>;
+// FIXME: We should really find a way to improve these patterns.
+def : Pat<(v8i32 (X86vzmovl
+                  (insert_subvector undef,
+                                    (v4i32 (scalar_to_vector
+                                            (and GR32:$src, 0xffff))),
+                                    (iPTR 0)))),
+          (SUBREG_TO_REG (i32 0), (VMOVW2SHrr GR32:$src), sub_xmm)>;
+def : Pat<(v16i32 (X86vzmovl
+                   (insert_subvector undef,
+                                     (v4i32 (scalar_to_vector
+                                             (and GR32:$src, 0xffff))),
+                                     (iPTR 0)))),
+          (SUBREG_TO_REG (i32 0), (VMOVW2SHrr GR32:$src), sub_xmm)>;
+
+def : Pat<(v4i32 (scalar_to_vector (i32 (extloadi16 addr:$src)))),
+          (VMOVWrm addr:$src)>;
+def : Pat<(v4i32 (X86vzmovl (scalar_to_vector (i32 (zextloadi16 addr:$src))))),
+          (VMOVWrm addr:$src)>;
+def : Pat<(v8i32 (X86vzmovl
+                  (insert_subvector undef,
+                                    (v4i32 (scalar_to_vector
+                                            (i32 (zextloadi16 addr:$src)))),
+                                    (iPTR 0)))),
+          (SUBREG_TO_REG (i32 0), (VMOVWrm addr:$src), sub_xmm)>;
+def : Pat<(v16i32 (X86vzmovl
+                   (insert_subvector undef,
+                                     (v4i32 (scalar_to_vector
+                                             (i32 (zextloadi16 addr:$src)))),
+                                     (iPTR 0)))),
+          (SUBREG_TO_REG (i32 0), (VMOVWrm addr:$src), sub_xmm)>;
+
+// Move word from xmm register to r/m16
+def VMOVSH2Wrr  : AVX512<0x7E, MRMDestReg, (outs GR32:$dst), (ins VR128X:$src),
+                       "vmovw\t{$src, $dst|$dst, $src}", []>, T_MAP5PD, EVEX, Sched<[WriteVecMoveToGpr]>;
+def VMOVWmr  : AVX512<0x7E, MRMDestMem, (outs),
+                       (ins i16mem:$dst, VR128X:$src),
+                       "vmovw\t{$src, $dst|$dst, $src}",
+                       [(store (i16 (extractelt (v8i16 VR128X:$src),
+                                     (iPTR 0))), addr:$dst)]>,
+                       T_MAP5PD, EVEX, EVEX_CD8<16, CD8VT1>, Sched<[WriteFStore]>;
+
+def : Pat<(i16 (bitconvert FR16X:$src)),
+          (i16 (EXTRACT_SUBREG
+                (VMOVSH2Wrr (COPY_TO_REGCLASS FR16X:$src, VR128X)),
+                sub_16bit))>;
+def : Pat<(i16 (extractelt (v8i16 VR128X:$src), (iPTR 0))),
+          (i16 (EXTRACT_SUBREG (VMOVSH2Wrr VR128X:$src), sub_16bit))>;
+}
+
+// Allow "vmovw" to use GR64
+let hasSideEffects = 0 in {
+  def VMOVW64toSHrr : AVX512<0x6E, MRMSrcReg, (outs VR128X:$dst), (ins GR64:$src),
+                     "vmovw\t{$src, $dst|$dst, $src}", []>, T_MAP5PD, EVEX, VEX_W, Sched<[WriteVecMoveFromGpr]>;
+  def VMOVSHtoW64rr : AVX512<0x7E, MRMDestReg, (outs GR64:$dst), (ins VR128X:$src),
+                     "vmovw\t{$src, $dst|$dst, $src}", []>, T_MAP5PD, EVEX, VEX_W, Sched<[WriteVecMoveToGpr]>;
+}

llvm/lib/Target/X86/X86InstrCompiler.td

let usesCustomInserter = 1, hasNoSchedulingInfo = 1, Uses = [EFLAGS] in {
   let Predicates = [FPStackf64] in
     defm _RFP64 : CMOVrr_PSEUDO<RFP64, f64>;
 
   defm _RFP80 : CMOVrr_PSEUDO<RFP80, f80>;
 
   let Predicates = [HasMMX] in
     defm _VR64   : CMOVrr_PSEUDO<VR64, x86mmx>;
 
+  defm _FR16X    : CMOVrr_PSEUDO<FR16X, f16>;
   let Predicates = [HasSSE1,NoAVX512] in
     defm _FR32   : CMOVrr_PSEUDO<FR32, f32>;
   let Predicates = [HasSSE2,NoAVX512] in
     defm _FR64   : CMOVrr_PSEUDO<FR64, f64>;
   let Predicates = [HasAVX512] in {
     defm _FR32X  : CMOVrr_PSEUDO<FR32X, f32>;
     defm _FR64X  : CMOVrr_PSEUDO<FR64X, f64>;
   }
let Predicates = [NoVLX] in {
   def : Pat<(v4f64 (X86cmov VR256:$t, VR256:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256 VR256:$t, VR256:$f, timm:$cond)>;
 }
 let Predicates = [HasVLX] in {
   def : Pat<(v16i8 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
   def : Pat<(v8i16 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
+  def : Pat<(v8f16 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
+            (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
   def : Pat<(v4i32 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
   def : Pat<(v4f32 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
   def : Pat<(v2f64 (X86cmov VR128X:$t, VR128X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR128X VR128X:$t, VR128X:$f, timm:$cond)>;
 
   def : Pat<(v32i8 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
   def : Pat<(v16i16 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
+  def : Pat<(v16f16 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
+            (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
   def : Pat<(v8i32 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
   def : Pat<(v8f32 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
   def : Pat<(v4f64 (X86cmov VR256X:$t, VR256X:$f, timm:$cond, EFLAGS)),
             (CMOV_VR256X VR256X:$t, VR256X:$f, timm:$cond)>;
 }
 
 def : Pat<(v64i8 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
           (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
 def : Pat<(v32i16 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
           (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
+def : Pat<(v32f16 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
+          (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
 def : Pat<(v16i32 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
           (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
 def : Pat<(v16f32 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
           (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
 def : Pat<(v8f64 (X86cmov VR512:$t, VR512:$f, timm:$cond, EFLAGS)),
           (CMOV_VR512 VR512:$t, VR512:$f, timm:$cond)>;
 
 //===----------------------------------------------------------------------===//

llvm/lib/Target/X86/X86InstrFormats.td

 def PS     : Prefix<4>; // Similar to NoPrfx, but disassembler uses this to know
                         // that other instructions with this opcode use PD/XS/XD
                         // and if any of those is not supported they shouldn't
                         // decode to this instruction. e.g. ANDSS/ANDSD don't
                         // exist, but the 0xf2/0xf3 encoding shouldn't
                         // disable to ANDPS.
 
 // Class specifying the opcode map.
-class Map<bits<3> val> {
-  bits<3> Value = val;
+class Map<bits<4> val> {
+  bits<4> Value = val;
 }
 def OB        : Map<0>;
 def TB        : Map<1>;
 def T8        : Map<2>;
 def TA        : Map<3>;
 def XOP8      : Map<4>;
 def XOP9      : Map<5>;
 def XOPA      : Map<6>;
 def ThreeDNow : Map<7>;
+def T_MAP5    : Map<8>;
+def T_MAP6    : Map<9>;
 
 // Class specifying the encoding
 class Encoding<bits<2> val> {
   bits<2> Value = val;
 }
 def EncNormal : Encoding<0>;
 def EncVEX    : Encoding<1>;
 def EncXOP    : Encoding<2>;

 class REP    { bit hasREPPrefix = 1; }
 class TB     { Map OpMap = TB; }
 class T8     { Map OpMap = T8; }
 class TA     { Map OpMap = TA; }
 class XOP8   { Map OpMap = XOP8; Prefix OpPrefix = PS; }
 class XOP9   { Map OpMap = XOP9; Prefix OpPrefix = PS; }
 class XOPA   { Map OpMap = XOPA; Prefix OpPrefix = PS; }
 class ThreeDNow { Map OpMap = ThreeDNow; }
+class T_MAP5     { Map OpMap = T_MAP5; }
+class T_MAP5PS : T_MAP5 { Prefix OpPrefix = PS; } // none
+class T_MAP5PD : T_MAP5 { Prefix OpPrefix = PD; } // 0x66
+class T_MAP5XS : T_MAP5 { Prefix OpPrefix = XS; } // 0xF3
+class T_MAP5XD : T_MAP5 { Prefix OpPrefix = XD; } // 0xF2
+class T_MAP6     { Map OpMap = T_MAP6; }
+class T_MAP6PS : T_MAP6 { Prefix OpPrefix = PS; }
+class T_MAP6PD : T_MAP6 { Prefix OpPrefix = PD; }
+class T_MAP6XS : T_MAP6 { Prefix OpPrefix = XS; }
+class T_MAP6XD : T_MAP6 { Prefix OpPrefix = XD; }
 class OBXS   { Prefix OpPrefix = XS; }
 class PS   : TB { Prefix OpPrefix = PS; }
 class PD   : TB { Prefix OpPrefix = PD; }
 class XD   : TB { Prefix OpPrefix = XD; }
 class XS   : TB { Prefix OpPrefix = XS; }
 class T8PS : T8 { Prefix OpPrefix = PS; }
 class T8PD : T8 { Prefix OpPrefix = PD; }
 class T8XD : T8 { Prefix OpPrefix = XD; }
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   bits<2> OpSizeBits = OpSize.Value;
   AddressSize AdSize = AdSizeX; // Does this instruction's encoding change
                                 // based on address size of the mode?
   bits<2> AdSizeBits = AdSize.Value;
 
   Prefix OpPrefix = NoPrfx; // Which prefix byte does this inst have?
   bits<3> OpPrefixBits = OpPrefix.Value;
   Map OpMap = OB;           // Which opcode map does this inst have?
-  bits<3> OpMapBits = OpMap.Value;
+  bits<4> OpMapBits = OpMap.Value;
   bit hasREX_WPrefix  = 0;  // Does this inst require the REX.W prefix?
   FPFormat FPForm = NotFP;  // What flavor of FP instruction is this?
   bit hasLockPrefix = 0;    // Does this inst have a 0xF0 prefix?
   Domain ExeDomain = d;
   bit hasREPPrefix = 0;     // Does this inst have a REP prefix?
   Encoding OpEnc = EncNormal; // Encoding used by this instruction
   bits<2> OpEncBits = OpEnc.Value;
   bit HasVEX_W = 0;         // Does this inst set the VEX_W field?
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   // Force to check predicate before compress EVEX to VEX encoding.
   bit checkVEXPredicate = 0;
   // TSFlags layout should be kept in sync with X86BaseInfo.h.
   let TSFlags{6-0}   = FormBits;
   let TSFlags{8-7}   = OpSizeBits;
   let TSFlags{10-9}  = AdSizeBits;
   // No need for 3rd bit, we don't need to distinguish NoPrfx from PS.
   let TSFlags{12-11} = OpPrefixBits{1-0};
-  let TSFlags{15-13} = OpMapBits;
-  let TSFlags{16}    = hasREX_WPrefix;
-  let TSFlags{20-17} = ImmT.Value;
-  let TSFlags{23-21} = FPForm.Value;
-  let TSFlags{24}    = hasLockPrefix;
-  let TSFlags{25}    = hasREPPrefix;
-  let TSFlags{27-26} = ExeDomain.Value;
-  let TSFlags{29-28} = OpEncBits;
-  let TSFlags{37-30} = Opcode;
+  let TSFlags{16-13} = OpMapBits;
+  let TSFlags{17}    = hasREX_WPrefix;
+  let TSFlags{21-18} = ImmT.Value;
+  let TSFlags{24-22} = FPForm.Value;
+  let TSFlags{25}    = hasLockPrefix;
+  let TSFlags{26}    = hasREPPrefix;
+  let TSFlags{28-27} = ExeDomain.Value;
+  let TSFlags{30-29} = OpEncBits;
+  let TSFlags{38-31} = Opcode;
   // Currently no need for second bit in TSFlags - W Ignore is equivalent to 0.
-  let TSFlags{38}    = HasVEX_W;
-  let TSFlags{39}    = hasVEX_4V;
-  let TSFlags{40}    = hasVEX_L;
-  let TSFlags{41}    = hasEVEX_K;
-  let TSFlags{42}    = hasEVEX_Z;
-  let TSFlags{43}    = hasEVEX_L2;
-  let TSFlags{44}    = hasEVEX_B;
+  let TSFlags{39}    = HasVEX_W;
+  let TSFlags{40}    = hasVEX_4V;
+  let TSFlags{41}    = hasVEX_L;
+  let TSFlags{42}    = hasEVEX_K;
+  let TSFlags{43}    = hasEVEX_Z;
+  let TSFlags{44}    = hasEVEX_L2;
+  let TSFlags{45}    = hasEVEX_B;
   // If we run out of TSFlags bits, it's possible to encode this in 3 bits.
-  let TSFlags{51-45} = CD8_Scale;
-  let TSFlags{52}    = hasEVEX_RC;
-  let TSFlags{53}    = hasNoTrackPrefix;
-  let TSFlags{54}    = ExplicitVEXPrefix;
+  let TSFlags{52-46} = CD8_Scale;
+  let TSFlags{53}    = hasEVEX_RC;
+  let TSFlags{54}    = hasNoTrackPrefix;
+  let TSFlags{55}    = ExplicitVEXPrefix;
 }
 
 class PseudoI<dag oops, dag iops, list<dag> pattern>
   : X86Inst<0, Pseudo, NoImm, oops, iops, ""> {
   let Pattern = pattern;
 }
 
 class I<bits<8> o, Format f, dag outs, dag ins, string asm,

llvm/lib/Target/X86/X86InstrFragmentsSIMD.td

 def X86PShuflw : SDNode<"X86ISD::PSHUFLW", SDTShuff2OpI>;
 
 def X86Shufp   : SDNode<"X86ISD::SHUFP", SDTShuff3OpI>;
 def X86Shuf128 : SDNode<"X86ISD::SHUF128", SDTShuff3OpI>;
 
 def X86Movddup  : SDNode<"X86ISD::MOVDDUP", SDTShuff1Op>;
 def X86Movshdup : SDNode<"X86ISD::MOVSHDUP", SDTShuff1Op>;
 def X86Movsldup : SDNode<"X86ISD::MOVSLDUP", SDTShuff1Op>;
+def X86Movsh : SDNode<"X86ISD::MOVSH",
+                      SDTypeProfile<1, 2, [SDTCisVT<0, v8f16>,
+                                           SDTCisVT<1, v8f16>,
+                                           SDTCisVT<2, v8f16>]>>;
 
 def X86Movsd : SDNode<"X86ISD::MOVSD",
                       SDTypeProfile<1, 2, [SDTCisVT<0, v2f64>,
                                            SDTCisVT<1, v2f64>,
                                            SDTCisVT<2, v2f64>]>>;
 def X86Movss : SDNode<"X86ISD::MOVSS",
                       SDTypeProfile<1, 2, [SDTCisVT<0, v4f32>,
                                            SDTCisVT<1, v4f32>,
def SDTX86MaskedStore: SDTypeProfile<0, 3, [       // masked store
   SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>, SDTCisSameNumEltsAs<0, 2>
 ]>;
 
 //===----------------------------------------------------------------------===//
 // SSE pattern fragments
 //===----------------------------------------------------------------------===//
 
 // 128-bit load pattern fragments
+def loadv8f16    : PatFrag<(ops node:$ptr), (v8f16 (load node:$ptr))>;
 def loadv4f32    : PatFrag<(ops node:$ptr), (v4f32 (load node:$ptr))>;
 def loadv2f64    : PatFrag<(ops node:$ptr), (v2f64 (load node:$ptr))>;
 def loadv2i64    : PatFrag<(ops node:$ptr), (v2i64 (load node:$ptr))>;
 def loadv4i32    : PatFrag<(ops node:$ptr), (v4i32 (load node:$ptr))>;
 def loadv8i16    : PatFrag<(ops node:$ptr), (v8i16 (load node:$ptr))>;
 def loadv16i8    : PatFrag<(ops node:$ptr), (v16i8 (load node:$ptr))>;
 
 // 256-bit load pattern fragments
+def loadv16f16   : PatFrag<(ops node:$ptr), (v16f16 (load node:$ptr))>;
 def loadv8f32    : PatFrag<(ops node:$ptr), (v8f32  (load node:$ptr))>;
 def loadv4f64    : PatFrag<(ops node:$ptr), (v4f64  (load node:$ptr))>;
 def loadv4i64    : PatFrag<(ops node:$ptr), (v4i64  (load node:$ptr))>;
 def loadv8i32    : PatFrag<(ops node:$ptr), (v8i32  (load node:$ptr))>;
 def loadv16i16   : PatFrag<(ops node:$ptr), (v16i16 (load node:$ptr))>;
 def loadv32i8    : PatFrag<(ops node:$ptr), (v32i8  (load node:$ptr))>;
 
 // 512-bit load pattern fragments
+def loadv32f16   : PatFrag<(ops node:$ptr), (v32f16 (load node:$ptr))>;
 def loadv16f32   : PatFrag<(ops node:$ptr), (v16f32 (load node:$ptr))>;
 def loadv8f64    : PatFrag<(ops node:$ptr), (v8f64  (load node:$ptr))>;
 def loadv8i64    : PatFrag<(ops node:$ptr), (v8i64  (load node:$ptr))>;
 def loadv16i32   : PatFrag<(ops node:$ptr), (v16i32 (load node:$ptr))>;
 def loadv32i16   : PatFrag<(ops node:$ptr), (v32i16 (load node:$ptr))>;
 def loadv64i8    : PatFrag<(ops node:$ptr), (v64i8  (load node:$ptr))>;
 
 // 128-/256-/512-bit extload pattern fragments
 def extloadv2f32 : PatFrag<(ops node:$ptr), (extloadvf32 node:$ptr)>;
 def extloadv4f32 : PatFrag<(ops node:$ptr), (extloadvf32 node:$ptr)>;
 def extloadv8f32 : PatFrag<(ops node:$ptr), (extloadvf32 node:$ptr)>;
+def extloadv2f16 : PatFrag<(ops node:$ptr), (extloadvf16 node:$ptr)>;
+def extloadv4f16 : PatFrag<(ops node:$ptr), (extloadvf16 node:$ptr)>;
+def extloadv8f16 : PatFrag<(ops node:$ptr), (extloadvf16 node:$ptr)>;
+def extloadv16f16 : PatFrag<(ops node:$ptr), (extloadvf16 node:$ptr)>;
 
 // Like 'store', but always requires vector size alignment.
 def alignedstore : PatFrag<(ops node:$val, node:$ptr),
                            (store node:$val, node:$ptr), [{
   auto *St = cast<StoreSDNode>(N);
   return St->getAlignment() >= St->getMemoryVT().getStoreSize();
 }]>;
 
 // Like 'load', but always requires vector size alignment.
 def alignedload : PatFrag<(ops node:$ptr), (load node:$ptr), [{
   auto *Ld = cast<LoadSDNode>(N);
   return Ld->getAlignment() >= Ld->getMemoryVT().getStoreSize();
 }]>;
 
 // 128-bit aligned load pattern fragments
 // NOTE: all 128-bit integer vector loads are promoted to v2i64
+def alignedloadv8f16 : PatFrag<(ops node:$ptr),
+                               (v8f16 (alignedload node:$ptr))>;
 def alignedloadv4f32 : PatFrag<(ops node:$ptr),
                                (v4f32 (alignedload node:$ptr))>;
 def alignedloadv2f64 : PatFrag<(ops node:$ptr),
                                (v2f64 (alignedload node:$ptr))>;
 def alignedloadv2i64 : PatFrag<(ops node:$ptr),
                                (v2i64 (alignedload node:$ptr))>;
 def alignedloadv4i32 : PatFrag<(ops node:$ptr),
                                (v4i32 (alignedload node:$ptr))>;
 def alignedloadv8i16 : PatFrag<(ops node:$ptr),
                                (v8i16 (alignedload node:$ptr))>;
 def alignedloadv16i8 : PatFrag<(ops node:$ptr),
                                (v16i8 (alignedload node:$ptr))>;
 
 // 256-bit aligned load pattern fragments
 // NOTE: all 256-bit integer vector loads are promoted to v4i64
+def alignedloadv16f16 : PatFrag<(ops node:$ptr),
+                                (v16f16 (alignedload node:$ptr))>;
 def alignedloadv8f32  : PatFrag<(ops node:$ptr),
                                 (v8f32  (alignedload node:$ptr))>;
 def alignedloadv4f64  : PatFrag<(ops node:$ptr),
                                 (v4f64  (alignedload node:$ptr))>;
 def alignedloadv4i64  : PatFrag<(ops node:$ptr),
                                 (v4i64  (alignedload node:$ptr))>;
 def alignedloadv8i32  : PatFrag<(ops node:$ptr),
                                 (v8i32  (alignedload node:$ptr))>;
 def alignedloadv16i16 : PatFrag<(ops node:$ptr),
                                 (v16i16 (alignedload node:$ptr))>;
 def alignedloadv32i8  : PatFrag<(ops node:$ptr),
                                 (v32i8  (alignedload node:$ptr))>;
 
 // 512-bit aligned load pattern fragments
+def alignedloadv32f16 : PatFrag<(ops node:$ptr),
+                                (v32f16 (alignedload node:$ptr))>;
 def alignedloadv16f32 : PatFrag<(ops node:$ptr),
                                 (v16f32 (alignedload node:$ptr))>;
 def alignedloadv8f64  : PatFrag<(ops node:$ptr),
                                 (v8f64  (alignedload node:$ptr))>;
 def alignedloadv8i64  : PatFrag<(ops node:$ptr),
                                 (v8i64  (alignedload node:$ptr))>;
 def alignedloadv16i32 : PatFrag<(ops node:$ptr),
                                 (v16i32 (alignedload node:$ptr))>;

 // 512-bit bitconvert pattern fragments
 def bc_v64i8 : PatFrag<(ops node:$in), (v64i8 (bitconvert node:$in))>;
 def bc_v32i16 : PatFrag<(ops node:$in), (v32i16 (bitconvert node:$in))>;
 def bc_v16i32 : PatFrag<(ops node:$in), (v16i32 (bitconvert node:$in))>;
 def bc_v8i64 : PatFrag<(ops node:$in), (v8i64 (bitconvert node:$in))>;
 def bc_v8f64 : PatFrag<(ops node:$in), (v8f64 (bitconvert node:$in))>;
 def bc_v16f32 : PatFrag<(ops node:$in), (v16f32 (bitconvert node:$in))>;
 
+def X86vzload16 : PatFrag<(ops node:$src),
+                          (X86vzld node:$src), [{
+  return cast<MemIntrinsicSDNode>(N)->getMemoryVT().getStoreSize() == 2;
+}]>;
+
 def X86vzload32 : PatFrag<(ops node:$src),
                           (X86vzld node:$src), [{
   return cast<MemIntrinsicSDNode>(N)->getMemoryVT().getStoreSize() == 4;
 }]>;
 
 def X86vzload64 : PatFrag<(ops node:$src),
                           (X86vzld node:$src), [{
   return cast<MemIntrinsicSDNode>(N)->getMemoryVT().getStoreSize() == 8;
def X86SubVBroadcastld128 : PatFrag<(ops node:$src),
   return cast<MemIntrinsicSDNode>(N)->getMemoryVT().getStoreSize() == 16;
 }]>;
 
 def X86SubVBroadcastld256 : PatFrag<(ops node:$src),
                                     (X86SubVBroadcastld node:$src), [{
   return cast<MemIntrinsicSDNode>(N)->getMemoryVT().getStoreSize() == 32;
 }]>;
 
+def fp16imm0 : PatLeaf<(f16 fpimm), [{
+  return N->isExactlyValue(+0.0);
+}]>;
+
 // Scalar SSE intrinsic fragments to match several different types of loads.
 // Used by scalar SSE intrinsic instructions which have 128 bit types, but
 // only load a single element.
 // FIXME: We should add more canolicalizing in DAGCombine. Particulary removing
 // the simple_load case.
+def sse_load_f16 : PatFrags<(ops node:$ptr),
+                            [(v8f16 (simple_load node:$ptr)),
+                             (v8f16 (X86vzload16 node:$ptr)),
+                             (v8f16 (scalar_to_vector (loadf16 node:$ptr)))]>;
 def sse_load_f32 : PatFrags<(ops node:$ptr),
                             [(v4f32 (simple_load node:$ptr)),
                              (v4f32 (X86vzload32 node:$ptr)),
                              (v4f32 (scalar_to_vector (loadf32 node:$ptr)))]>;
 def sse_load_f64 : PatFrags<(ops node:$ptr),
                             [(v2f64 (simple_load node:$ptr)),
                              (v2f64 (X86vzload64 node:$ptr)),
                              (v2f64 (scalar_to_vector (loadf64 node:$ptr)))]>;
 
+def shmem : X86MemOperand<"printwordmem", X86Mem16AsmOperand>;
 def ssmem : X86MemOperand<"printdwordmem", X86Mem32AsmOperand>;
 def sdmem : X86MemOperand<"printqwordmem", X86Mem64AsmOperand>;
 
 
 def fp32imm0 : PatLeaf<(f32 fpimm), [{
   return N->isExactlyValue(+0.0);
 }]>;
 

llvm/lib/Target/X86/X86InstrInfo.cpp

 //===-- X86InstrInfo.cpp - X86 Instruction Information --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
static bool isFrameLoadOpcode(int Opcode, unsigned &MemBytes) {
   default:
     return false;
   case X86::MOV8rm:
   case X86::KMOVBkm:
     MemBytes = 1;
     return true;
   case X86::MOV16rm:
   case X86::KMOVWkm:
+  case X86::VMOVSHZrm:
+  case X86::VMOVSHZrm_alt:
     MemBytes = 2;
     return true;
   case X86::MOV32rm:
   case X86::MOVSSrm:
   case X86::MOVSSrm_alt:
   case X86::VMOVSSrm:
   case X86::VMOVSSrm_alt:
   case X86::VMOVSSZrm:
static bool isFrameStoreOpcode(int Opcode, unsigned &MemBytes) {
   default:
     return false;
   case X86::MOV8mr:
   case X86::KMOVBmk:
     MemBytes = 1;
     return true;
   case X86::MOV16mr:
   case X86::KMOVWmk:
+  case X86::VMOVSHZmr:
     MemBytes = 2;
     return true;
   case X86::MOV32mr:
   case X86::MOVSSmr:
   case X86::VMOVSSmr:
   case X86::VMOVSSZmr:
   case X86::KMOVDmk:
     MemBytes = 4;
bool X86InstrInfo::isReallyTriviallyReMaterializable(const MachineInstr &MI,
   case X86::LOAD_STACK_GUARD:
   case X86::AVX1_SETALLONES:
   case X86::AVX2_SETALLONES:
   case X86::AVX512_128_SET0:
   case X86::AVX512_256_SET0:
   case X86::AVX512_512_SET0:
   case X86::AVX512_512_SETALLONES:
   case X86::AVX512_FsFLD0SD:
+  case X86::AVX512_FsFLD0SH:
   case X86::AVX512_FsFLD0SS:
   case X86::AVX512_FsFLD0F128:
   case X86::AVX_SET0:
   case X86::FsFLD0SD:
   case X86::FsFLD0SS:
   case X86::FsFLD0F128:
   case X86::KSET0D:
   case X86::KSET0Q:
bool X86InstrInfo::isReallyTriviallyReMaterializable(const MachineInstr &MI,
   case X86::VMOVDQUYrm:
   case X86::MMX_MOVD64rm:
   case X86::MMX_MOVQ64rm:
   // AVX-512
   case X86::VMOVSSZrm:
   case X86::VMOVSSZrm_alt:
   case X86::VMOVSDZrm:
   case X86::VMOVSDZrm_alt:
+  case X86::VMOVSHZrm:
+  case X86::VMOVSHZrm_alt:
   case X86::VMOVAPDZ128rm:
   case X86::VMOVAPDZ256rm:
   case X86::VMOVAPDZrm:
   case X86::VMOVAPSZ128rm:
   case X86::VMOVAPSZ256rm:
   case X86::VMOVAPSZ128rm_NOVLX:
   case X86::VMOVAPSZ256rm_NOVLX:
   case X86::VMOVAPSZrm:
if (STI.is64Bit())
       // Copying to or from a physical H register on x86-64 requires a NOREX
       // move.  Otherwise use a normal move.
       if (isHReg(Reg) || X86::GR8_ABCD_HRegClass.hasSubClassEq(RC))
         return load ? X86::MOV8rm_NOREX : X86::MOV8mr_NOREX;
     return load ? X86::MOV8rm : X86::MOV8mr;
   case 2:
     if (X86::VK16RegClass.hasSubClassEq(RC))
       return load ? X86::KMOVWkm : X86::KMOVWmk;
+    if (X86::FR16XRegClass.hasSubClassEq(RC)) {
+      assert(STI.hasFP16());
+      return load ? X86::VMOVSHZrm_alt : X86::VMOVSHZmr;
+    }
     assert(X86::GR16RegClass.hasSubClassEq(RC) && "Unknown 2-byte regclass");
     return load ? X86::MOV16rm : X86::MOV16mr;
   case 4:
     if (X86::GR32RegClass.hasSubClassEq(RC))
       return load ? X86::MOV32rm : X86::MOV32mr;
     if (X86::FR32XRegClass.hasSubClassEq(RC))
       return load ?
         (HasAVX512 ? X86::VMOVSSZrm_alt :
case X86::AVX_SET0: {
     Register SrcReg = MIB.getReg(0);
     Register XReg = TRI->getSubReg(SrcReg, X86::sub_xmm);
     MIB->getOperand(0).setReg(XReg);
     Expand2AddrUndef(MIB, get(X86::VXORPSrr));
     MIB.addReg(SrcReg, RegState::ImplicitDefine);
     return true;
   }
   case X86::AVX512_128_SET0:
+  case X86::AVX512_FsFLD0SH:
   case X86::AVX512_FsFLD0SS:
   case X86::AVX512_FsFLD0SD:
   case X86::AVX512_FsFLD0F128: {
     bool HasVLX = Subtarget.hasVLX();
     Register SrcReg = MIB.getReg(0);
     const TargetRegisterInfo *TRI = &getRegisterInfo();
     if (HasVLX || TRI->getEncodingValue(SrcReg) < 16)
       return Expand2AddrUndef(MIB,
else
     case X86::FsFLD0SD:
     case X86::AVX512_FsFLD0SD:
       Alignment = Align(8);
       break;
     case X86::FsFLD0SS:
     case X86::AVX512_FsFLD0SS:
       Alignment = Align(4);
       break;
+    case X86::AVX512_FsFLD0SH:
+      Alignment = Align(2);
+      break;
     default:
       return nullptr;
     }
   if (Ops.size() == 2 && Ops[0] == 0 && Ops[1] == 1) {
     unsigned NewOpc = 0;
     switch (MI.getOpcode()) {
     default: return nullptr;
     case X86::TEST8rr:  NewOpc = X86::CMP8ri; break;
MachineInstr *X86InstrInfo::foldMemoryOperandImpl(
   case X86::V_SETALLONES:
   case X86::AVX2_SETALLONES:
   case X86::AVX1_SETALLONES:
   case X86::AVX_SET0:
   case X86::AVX512_128_SET0:
   case X86::AVX512_256_SET0:
   case X86::AVX512_512_SET0:
   case X86::AVX512_512_SETALLONES:
+  case X86::AVX512_FsFLD0SH:
   case X86::FsFLD0SD:
   case X86::AVX512_FsFLD0SD:
   case X86::FsFLD0SS:
   case X86::AVX512_FsFLD0SS:
   case X86::FsFLD0F128:
   case X86::AVX512_FsFLD0F128: {
     // Folding a V_SET0 or V_SETALLONES as a load, to ease register pressure.
     // Create a constant-pool entry and operands to load from it.
case X86::AVX512_FsFLD0F128: {
     Type *Ty;
     unsigned Opc = LoadMI.getOpcode();
     if (Opc == X86::FsFLD0SS || Opc == X86::AVX512_FsFLD0SS)
       Ty = Type::getFloatTy(MF.getFunction().getContext());
     else if (Opc == X86::FsFLD0SD || Opc == X86::AVX512_FsFLD0SD)
       Ty = Type::getDoubleTy(MF.getFunction().getContext());
     else if (Opc == X86::FsFLD0F128 || Opc == X86::AVX512_FsFLD0F128)
       Ty = Type::getFP128Ty(MF.getFunction().getContext());
+    else if (Opc == X86::AVX512_FsFLD0SH)
+      Ty = Type::getHalfTy(MF.getFunction().getContext());
     else if (Opc == X86::AVX512_512_SET0 || Opc == X86::AVX512_512_SETALLONES)
       Ty = FixedVectorType::get(Type::getInt32Ty(MF.getFunction().getContext()),
                                 16);
     else if (Opc == X86::AVX2_SETALLONES || Opc == X86::AVX_SET0 ||
              Opc == X86::AVX512_256_SET0 || Opc == X86::AVX1_SETALLONES)
       Ty = FixedVectorType::get(Type::getInt32Ty(MF.getFunction().getContext()),
                                 8);
     else if (Opc == X86::MMX_SET0)

llvm/lib/Target/X86/X86InstrInfo.td

 
 def i8mem   : X86MemOperand<"printbytemem",   X86Mem8AsmOperand>;
 def i16mem  : X86MemOperand<"printwordmem",  X86Mem16AsmOperand>;
 def i32mem  : X86MemOperand<"printdwordmem",  X86Mem32AsmOperand>;
 def i64mem  : X86MemOperand<"printqwordmem",  X86Mem64AsmOperand>;
 def i128mem : X86MemOperand<"printxmmwordmem", X86Mem128AsmOperand>;
 def i256mem : X86MemOperand<"printymmwordmem", X86Mem256AsmOperand>;
 def i512mem : X86MemOperand<"printzmmwordmem", X86Mem512AsmOperand>;
+def f16mem  : X86MemOperand<"printwordmem",   X86Mem16AsmOperand>;
 def f32mem  : X86MemOperand<"printdwordmem",  X86Mem32AsmOperand>;
 def f64mem  : X86MemOperand<"printqwordmem",  X86Mem64AsmOperand>;
 def f80mem  : X86MemOperand<"printtbytemem",  X86Mem80AsmOperand>;
 def f128mem : X86MemOperand<"printxmmwordmem", X86Mem128AsmOperand>;
 def f256mem : X86MemOperand<"printymmwordmem", X86Mem256AsmOperand>;
 def f512mem : X86MemOperand<"printzmmwordmem", X86Mem512AsmOperand>;
 
 // Gather mem operands

 def HasVLX       : Predicate<"Subtarget->hasVLX()">;
 def NoVLX        : Predicate<"!Subtarget->hasVLX()">;
 def NoVLX_Or_NoBWI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasBWI()">;
 def NoVLX_Or_NoDQI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasDQI()">;
 def PKU        : Predicate<"Subtarget->hasPKU()">;
 def HasVNNI    : Predicate<"Subtarget->hasVNNI()">;
 def HasVP2INTERSECT : Predicate<"Subtarget->hasVP2INTERSECT()">;
 def HasBF16      : Predicate<"Subtarget->hasBF16()">;
+def HasFP16      : Predicate<"Subtarget->hasFP16()">;
 def HasAVXVNNI : Predicate <"Subtarget->hasAVXVNNI()">;
 def NoVLX_Or_NoVNNI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasVNNI()">;
 
 def HasBITALG    : Predicate<"Subtarget->hasBITALG()">;
 def HasPOPCNT    : Predicate<"Subtarget->hasPOPCNT()">;
 def HasAES       : Predicate<"Subtarget->hasAES()">;
 def HasVAES      : Predicate<"Subtarget->hasVAES()">;
 def NoVLX_Or_NoVAES : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasVAES()">;
def loadi32 : PatFrag<(ops node:$ptr), (i32 (unindexedload node:$ptr)), [{
   if (ExtType == ISD::NON_EXTLOAD)
     return true;
   if (ExtType == ISD::EXTLOAD && EnablePromoteAnyextLoad)
     return LD->getAlignment() >= 4 && LD->isSimple();
   return false;
 }]>;
 
 def loadi64  : PatFrag<(ops node:$ptr), (i64 (load node:$ptr))>;
+def loadf16  : PatFrag<(ops node:$ptr), (f16 (load node:$ptr))>;
 def loadf32  : PatFrag<(ops node:$ptr), (f32 (load node:$ptr))>;
 def loadf64  : PatFrag<(ops node:$ptr), (f64 (load node:$ptr))>;
 def loadf80  : PatFrag<(ops node:$ptr), (f80 (load node:$ptr))>;
 def loadf128 : PatFrag<(ops node:$ptr), (f128 (load node:$ptr))>;
 def alignedloadf128 : PatFrag<(ops node:$ptr), (f128 (load node:$ptr)), [{
   LoadSDNode *Ld = cast<LoadSDNode>(N);
   return Ld->getAlignment() >= Ld->getMemoryVT().getStoreSize();
 }]>;

llvm/lib/Target/X86/X86InstrVecCompiler.td

let Predicates = [NoAVX512] in {
   def : Pat<(f32 (extractelt (v4f32 VR128:$src), (iPTR 0))),
             (COPY_TO_REGCLASS (v4f32 VR128:$src), FR32)>;
   def : Pat<(f64 (extractelt (v2f64 VR128:$src), (iPTR 0))),
             (COPY_TO_REGCLASS (v2f64 VR128:$src), FR64)>;
 }
 
 let Predicates = [HasAVX512] in {
   // A vector extract of the first f32/f64 position is a subregister copy
+  def : Pat<(f16 (extractelt (v8f16 VR128X:$src), (iPTR 0))),
+            (COPY_TO_REGCLASS (v8f16 VR128X:$src), FR16X)>;
   def : Pat<(f32 (extractelt (v4f32 VR128X:$src), (iPTR 0))),
             (COPY_TO_REGCLASS (v4f32 VR128X:$src), FR32X)>;
   def : Pat<(f64 (extractelt (v2f64 VR128X:$src), (iPTR 0))),
             (COPY_TO_REGCLASS (v2f64 VR128X:$src), FR64X)>;
 }
 
 let Predicates = [NoVLX] in {
+  def : Pat<(v8f16 (scalar_to_vector FR16X:$src)),
+            (COPY_TO_REGCLASS FR16X:$src, VR128)>;
   // Implicitly promote a 32-bit scalar to a vector.
   def : Pat<(v4f32 (scalar_to_vector FR32:$src)),
             (COPY_TO_REGCLASS FR32:$src, VR128)>;
   // Implicitly promote a 64-bit scalar to a vector.
   def : Pat<(v2f64 (scalar_to_vector FR64:$src)),
             (COPY_TO_REGCLASS FR64:$src, VR128)>;
 }
 
 let Predicates = [HasVLX] in {
+  def : Pat<(v8f16 (scalar_to_vector FR16X:$src)),
+            (COPY_TO_REGCLASS FR16X:$src, VR128X)>;
   // Implicitly promote a 32-bit scalar to a vector.
   def : Pat<(v4f32 (scalar_to_vector FR32X:$src)),
             (COPY_TO_REGCLASS FR32X:$src, VR128X)>;
   // Implicitly promote a 64-bit scalar to a vector.
   def : Pat<(v2f64 (scalar_to_vector FR64X:$src)),
             (COPY_TO_REGCLASS FR64X:$src, VR128X)>;
 }
 

 // insert to the first 256-bit vector position is a subregister copy that needs
 // no instruction.
 defm : subvector_subreg_lowering<VR128, v4i32, VR256, v8i32,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v4f32, VR256, v8f32,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v2i64, VR256, v4i64,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v2f64, VR256, v4f64,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v8i16, VR256, v16i16, sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v16i8, VR256, v32i8,  sub_xmm>;
+defm : subvector_subreg_lowering<VR128, v8f16, VR256, v16f16, sub_xmm>;
 
 // A 128-bit subvector extract from the first 512-bit vector position is a
 // subregister copy that needs no instruction. Likewise, a 128-bit subvector
 // insert to the first 512-bit vector position is a subregister copy that needs
 // no instruction.
 defm : subvector_subreg_lowering<VR128, v4i32, VR512, v16i32, sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v4f32, VR512, v16f32, sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v2i64, VR512, v8i64,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v2f64, VR512, v8f64,  sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v8i16, VR512, v32i16, sub_xmm>;
 defm : subvector_subreg_lowering<VR128, v16i8, VR512, v64i8,  sub_xmm>;
+defm : subvector_subreg_lowering<VR128, v8f16, VR512, v32f16, sub_xmm>;
 
 // A 128-bit subvector extract from the first 512-bit vector position is a
 // subregister copy that needs no instruction. Likewise, a 128-bit subvector
 // insert to the first 512-bit vector position is a subregister copy that needs
 // no instruction.
 defm : subvector_subreg_lowering<VR256, v8i32,  VR512, v16i32, sub_ymm>;
 defm : subvector_subreg_lowering<VR256, v8f32,  VR512, v16f32, sub_ymm>;
 defm : subvector_subreg_lowering<VR256, v4i64,  VR512, v8i64,  sub_ymm>;
 defm : subvector_subreg_lowering<VR256, v4f64,  VR512, v8f64,  sub_ymm>;
 defm : subvector_subreg_lowering<VR256, v16i16, VR512, v32i16, sub_ymm>;
 defm : subvector_subreg_lowering<VR256, v32i8,  VR512, v64i8,  sub_ymm>;
+defm : subvector_subreg_lowering<VR256, v16f16, VR512, v32f16, sub_ymm>;
 
 
 // If we're inserting into an all zeros vector, just use a plain move which
 // will zero the upper bits. A post-isel hook will take care of removing
 // any moves that we can prove are unnecessary.
 multiclass subvec_zero_lowering<string MoveStr,
                                 RegisterClass RC, ValueType DstTy,
                                 ValueType SrcTy, ValueType ZeroTy,
let Predicates = [HasAVX512, NoVLX] in {
   defm : subvec_zero_lowering<"APDY", VR256, v8f64, v4f64, v16i32, sub_ymm>;
   defm : subvec_zero_lowering<"APSY", VR256, v16f32, v8f32, v16i32, sub_ymm>;
   defm : subvec_zero_lowering<"DQAY", VR256, v8i64, v4i64, v16i32, sub_ymm>;
   defm : subvec_zero_lowering<"DQAY", VR256, v16i32, v8i32, v16i32, sub_ymm>;
   defm : subvec_zero_lowering<"DQAY", VR256, v32i16, v16i16, v16i32, sub_ymm>;
   defm : subvec_zero_lowering<"DQAY", VR256, v64i8, v32i8, v16i32, sub_ymm>;
 }
 
+let Predicates = [HasFP16, HasVLX] in {
+  defm : subvec_zero_lowering<"APSZ128", VR128X, v16f16, v8f16, v8i32, sub_xmm>;
+  defm : subvec_zero_lowering<"APSZ128", VR128X, v32f16, v8f16, v16i32, sub_xmm>;
+  defm : subvec_zero_lowering<"APSZ256", VR256X, v32f16, v16f16, v16i32, sub_ymm>;
+}
+
 class maskzeroupper<ValueType vt, RegisterClass RC> :
   PatLeaf<(vt RC:$src), [{
     return isMaskZeroExtended(N);
   }]>;
 
 def maskzeroupperv1i1  : maskzeroupper<v1i1,  VK1>;
 def maskzeroupperv2i1  : maskzeroupper<v2i1,  VK2>;
 def maskzeroupperv4i1  : maskzeroupper<v4i1,  VK4>;

llvm/lib/Target/X86/X86RegisterInfo.td

 }
 
 // Helper to allow %st to print as %st(0) when its encoded in the instruction.
 def RSTi : RegisterOperand<RST, "printSTiRegOperand">;
 
 // Generic vector registers: VR64 and VR128.
 // Ensure that float types are declared first - only float is legal on SSE1.
 def VR64: RegisterClass<"X86", [x86mmx], 64, (sequence "MM%u", 0, 7)>;
-def VR128 : RegisterClass<"X86", [v4f32, v2f64, v16i8, v8i16, v4i32, v2i64, f128],
+def VR128 : RegisterClass<"X86", [v4f32, v2f64, v8f16, v16i8, v8i16, v4i32, v2i64, f128],
                           128, (add FR32)>;
-def VR256 : RegisterClass<"X86", [v8f32, v4f64, v32i8, v16i16, v8i32, v4i64],
+def VR256 : RegisterClass<"X86", [v8f32, v4f64, v16f16, v32i8, v16i16, v8i32, v4i64],
                           256, (sequence "YMM%u", 0, 15)>;
 
 // Status flags registers.
 def CCR : RegisterClass<"X86", [i32], 32, (add EFLAGS)> {
   let CopyCost = -1;  // Don't allow copying of status registers.
   let isAllocatable = 0;
 }
 def FPCCR : RegisterClass<"X86", [i16], 16, (add FPSW)> {
   let CopyCost = -1;  // Don't allow copying of status registers.
   let isAllocatable = 0;
 }
 def DFCCR : RegisterClass<"X86", [i32], 32, (add DF)> {
   let CopyCost = -1;  // Don't allow copying of status registers.
   let isAllocatable = 0;
 }
 
 // AVX-512 vector/mask registers.
-def VR512 : RegisterClass<"X86", [v16f32, v8f64, v64i8, v32i16, v16i32, v8i64],
+def VR512 : RegisterClass<"X86", [v16f32, v8f64, v32f16, v64i8, v32i16, v16i32, v8i64],
                           512, (sequence "ZMM%u", 0, 31)>;
 
 // Represents the lower 16 registers that have VEX/legacy encodable subregs.
 def VR512_0_15 : RegisterClass<"X86", [v16f32, v8f64, v64i8, v32i16, v16i32, v8i64],
                                512, (sequence "ZMM%u", 0, 15)>;
 
 // Scalar AVX-512 floating point registers.
 def FR32X : RegisterClass<"X86", [f32], 32, (sequence "XMM%u", 0, 31)>;
 
 def FR64X : RegisterClass<"X86", [f64], 64, (add FR32X)>;
 
+def FR16X : RegisterClass<"X86", [f16], 16, (add FR32X)>;
+
 // Extended VR128 and VR256 for AVX-512 instructions
-def VR128X : RegisterClass<"X86", [v4f32, v2f64, v16i8, v8i16, v4i32, v2i64, f128],
+def VR128X : RegisterClass<"X86", [v4f32, v2f64, v8f16, v16i8, v8i16, v4i32, v2i64, f128],
                            128, (add FR32X)>;
-def VR256X : RegisterClass<"X86", [v8f32, v4f64, v32i8, v16i16, v8i32, v4i64],
+def VR256X : RegisterClass<"X86", [v8f32, v4f64, v16f16, v32i8, v16i16, v8i32, v4i64],
                            256, (sequence "YMM%u", 0, 31)>;
 
 // Mask registers
 def VK1     : RegisterClass<"X86", [v1i1],  16,  (sequence "K%u", 0, 7)> {let Size = 16;}
 def VK2     : RegisterClass<"X86", [v2i1],  16,  (add VK1)> {let Size = 16;}
 def VK4     : RegisterClass<"X86", [v4i1],  16,  (add VK2)> {let Size = 16;}
 def VK8     : RegisterClass<"X86", [v8i1],  16,  (add VK4)> {let Size = 16;}
 def VK16    : RegisterClass<"X86", [v16i1], 16, (add VK8)> {let Size = 16;}

llvm/lib/Target/X86/X86Schedule.td

class X86SchedWriteWidths<X86FoldableSchedWrite sScl,
   X86FoldableSchedWrite Scl = sScl; // Scalar float/double operations.
   X86FoldableSchedWrite MMX = sScl; // MMX operations.
   X86FoldableSchedWrite XMM = s128; // XMM operations.
   X86FoldableSchedWrite YMM = s256; // YMM operations.
   X86FoldableSchedWrite ZMM = s512; // ZMM operations.
 }
 
 // Multiclass that wraps X86SchedWriteWidths for each fp vector type.
-class X86SchedWriteSizes<X86SchedWriteWidths sPS,
+class X86SchedWriteSizes<X86SchedWriteWidths sPH,
+                         X86SchedWriteWidths sPS,
                          X86SchedWriteWidths sPD> {
+  X86SchedWriteWidths PH = sPH;
   X86SchedWriteWidths PS = sPS;
   X86SchedWriteWidths PD = sPD;
 }
 
 // Multiclass that wraps move/load/store triple for a vector width.
 class X86SchedWriteMoveLS<SchedWrite MoveRR,
                           SchedWrite LoadRM,
                           SchedWrite StoreMR> {
: X86SchedWriteWidths<WriteVarShuffle, WriteVarShuffleX,
                        WriteVarShuffleY, WriteVarShuffleZ>;
 def SchedWriteBlend
  : X86SchedWriteWidths<WriteBlend, WriteBlend, WriteBlendY, WriteBlendZ>;
 def SchedWriteVarBlend
  : X86SchedWriteWidths<WriteVarBlend, WriteVarBlend,
                        WriteVarBlendY, WriteVarBlendZ>;
 
 // Vector size wrappers.
+// FIXME: Currently PH uses the same schedule method as PS.
+// We may refine them later.
 def SchedWriteFAddSizes
- : X86SchedWriteSizes<SchedWriteFAdd, SchedWriteFAdd64>;
+ : X86SchedWriteSizes<SchedWriteFAdd, SchedWriteFAdd, SchedWriteFAdd64>;
 def SchedWriteFCmpSizes
- : X86SchedWriteSizes<SchedWriteFCmp, SchedWriteFCmp64>;
+ : X86SchedWriteSizes<SchedWriteFCmp, SchedWriteFCmp, SchedWriteFCmp64>;
 def SchedWriteFMulSizes
- : X86SchedWriteSizes<SchedWriteFMul, SchedWriteFMul64>;
+ : X86SchedWriteSizes<SchedWriteFMul, SchedWriteFMul, SchedWriteFMul64>;
 def SchedWriteFDivSizes
- : X86SchedWriteSizes<SchedWriteFDiv, SchedWriteFDiv64>;
+ : X86SchedWriteSizes<SchedWriteFDiv, SchedWriteFDiv, SchedWriteFDiv64>;
 def SchedWriteFSqrtSizes
- : X86SchedWriteSizes<SchedWriteFSqrt, SchedWriteFSqrt64>;
+ : X86SchedWriteSizes<SchedWriteFSqrt, SchedWriteFSqrt, SchedWriteFSqrt64>;
 def SchedWriteFLogicSizes
- : X86SchedWriteSizes<SchedWriteFLogic, SchedWriteFLogic>;
+ : X86SchedWriteSizes<SchedWriteFLogic, SchedWriteFLogic, SchedWriteFLogic>;
 def SchedWriteFShuffleSizes
- : X86SchedWriteSizes<SchedWriteFShuffle, SchedWriteFShuffle>;
+ : X86SchedWriteSizes<SchedWriteFShuffle, SchedWriteFShuffle, SchedWriteFShuffle>;
 
 //===----------------------------------------------------------------------===//
 // Generic Processor Scheduler Models.
 
 // IssueWidth is analogous to the number of decode units. Core and its
 // descendents, including Nehalem and SandyBridge have 4 decoders.
 // Resources beyond the decoder operate on micro-ops and are bufferred
 // so adjacent micro-ops don't directly compete.

llvm/lib/Target/X86/X86Subtarget.h

 //===-- X86Subtarget.h - Define Subtarget for the X86 ----------*- C++ -*--===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
class X86Subtarget final : public X86GenSubtargetInfo {
   bool HasDQI = false;
 
   /// Processor has AVX-512 Byte and Word instructions
   bool HasBWI = false;
 
   /// Processor has AVX-512 Vector Length eXtenstions
   bool HasVLX = false;
 
+  /// Processor has AVX-512 16 bit floating-point extenstions
+  bool HasFP16 = false;
+
   /// Processor has PKU extenstions
   bool HasPKU = false;
 
   /// Processor has AVX-512 Vector Neural Network Instructions
   bool HasVNNI = false;
 
   /// Processor has AVX Vector Neural Network Instructions
   bool HasAVXVNNI = false;
public:
   bool slowIncDec() const { return SlowIncDec; }
   bool hasCDI() const { return HasCDI; }
   bool hasVPOPCNTDQ() const { return HasVPOPCNTDQ; }
   bool hasPFI() const { return HasPFI; }
   bool hasERI() const { return HasERI; }
   bool hasDQI() const { return HasDQI; }
   bool hasBWI() const { return HasBWI; }
   bool hasVLX() const { return HasVLX; }
+  bool hasFP16() const { return HasFP16; }
   bool hasPKU() const { return HasPKU; }
   bool hasVNNI() const { return HasVNNI; }
   bool hasBF16() const { return HasBF16; }
   bool hasVP2INTERSECT() const { return HasVP2INTERSECT; }
   bool hasBITALG() const { return HasBITALG; }
   bool hasSHSTK() const { return HasSHSTK; }
   bool hasCLFLUSHOPT() const { return HasCLFLUSHOPT; }
   bool hasCLWB() const { return HasCLWB; }

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

 //===-- X86TargetTransformInfo.cpp - X86 specific TTI pass ----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
InstructionCost X86TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
   // For 2-input shuffles, we must account for splitting the 2 inputs into many.
   if (Kind == TTI::SK_PermuteTwoSrc && LT.first != 1) {
     // We assume that source and destination have the same vector type.
     InstructionCost NumOfDests = LT.first;
     InstructionCost NumOfShufflesPerDest = LT.first * 2 - 1;
     LT.first = NumOfDests * NumOfShufflesPerDest;
   }
 
+  static const CostTblEntry AVX512FP16ShuffleTbl[] = {
+      {TTI::SK_Broadcast, MVT::v32f16, 1}, // vpbroadcastw
+      {TTI::SK_Broadcast, MVT::v16f16, 1}, // vpbroadcastw
+      {TTI::SK_Broadcast, MVT::v8f16, 1},  // vpbroadcastw
+
+      {TTI::SK_Reverse, MVT::v32f16, 2}, // vpermw
+      {TTI::SK_Reverse, MVT::v16f16, 2}, // vpermw
+      {TTI::SK_Reverse, MVT::v8f16, 1},  // vpshufb
+
+      {TTI::SK_PermuteSingleSrc, MVT::v32f16, 2}, // vpermw
+      {TTI::SK_PermuteSingleSrc, MVT::v16f16, 2}, // vpermw
+      {TTI::SK_PermuteSingleSrc, MVT::v8f16, 1},  // vpshufb
+
+      {TTI::SK_PermuteTwoSrc, MVT::v32f16, 2}, // vpermt2w
+      {TTI::SK_PermuteTwoSrc, MVT::v16f16, 2}, // vpermt2w
+      {TTI::SK_PermuteTwoSrc, MVT::v8f16, 2}   // vpermt2w
+  };
+
+  if (!ST->useSoftFloat() && ST->hasFP16())
+    if (const auto *Entry =
+            CostTableLookup(AVX512FP16ShuffleTbl, Kind, LT.second))
+      return LT.first * Entry->Cost;
+
   static const CostTblEntry AVX512VBMIShuffleTbl[] = {
       {TTI::SK_Reverse, MVT::v64i8, 1}, // vpermb
       {TTI::SK_Reverse, MVT::v32i8, 1}, // vpermb
 
       {TTI::SK_PermuteSingleSrc, MVT::v64i8, 1}, // vpermb
       {TTI::SK_PermuteSingleSrc, MVT::v32i8, 1}, // vpermb
 
       {TTI::SK_PermuteTwoSrc, MVT::v64i8, 2}, // vpermt2b
bool X86TTIImpl::isLegalMaskedLoad(Type *DataTy, Align Alignment) {
   Type *ScalarTy = DataTy->getScalarType();
 
   if (ScalarTy->isPointerTy())
     return true;
 
   if (ScalarTy->isFloatTy() || ScalarTy->isDoubleTy())
     return true;
 
+  if (ScalarTy->isHalfTy() && ST->hasBWI() && ST->hasFP16())
+    return true;
+
   if (!ScalarTy->isIntegerTy())
     return false;
 
   unsigned IntWidth = ScalarTy->getIntegerBitWidth();
   return IntWidth == 32 || IntWidth == 64 ||
          ((IntWidth == 8 || IntWidth == 16) && ST->hasBWI());
 }
 
InstructionCost Cost =
       NumOfMoves;
   return Cost;
 }
 
 InstructionCost X86TTIImpl::getInterleavedMemoryOpCost(
     unsigned Opcode, Type *VecTy, unsigned Factor, ArrayRef<unsigned> Indices,
     Align Alignment, unsigned AddressSpace, TTI::TargetCostKind CostKind,
     bool UseMaskForCond, bool UseMaskForGaps) {
-  auto isSupportedOnAVX512 = [](Type *VecTy, bool HasBW) {
+  auto isSupportedOnAVX512 = [&](Type *VecTy, bool HasBW) {
     Type *EltTy = cast<VectorType>(VecTy)->getElementType();
     if (EltTy->isFloatTy() || EltTy->isDoubleTy() || EltTy->isIntegerTy(64) ||
         EltTy->isIntegerTy(32) || EltTy->isPointerTy())
       return true;
-    if (EltTy->isIntegerTy(16) || EltTy->isIntegerTy(8))
+    if (EltTy->isIntegerTy(16) || EltTy->isIntegerTy(8) ||
+        (!ST->useSoftFloat() && ST->hasFP16() && EltTy->isHalfTy()))
       return HasBW;
     return false;
   };
   if (ST->hasAVX512() && isSupportedOnAVX512(VecTy, ST->hasBWI()))
     return getInterleavedMemoryOpCostAVX512(
         Opcode, cast<FixedVectorType>(VecTy), Factor, Indices, Alignment,
         AddressSpace, CostKind, UseMaskForCond, UseMaskForGaps);
   if (ST->hasAVX2())
     return getInterleavedMemoryOpCostAVX2(
         Opcode, cast<FixedVectorType>(VecTy), Factor, Indices, Alignment,
         AddressSpace, CostKind, UseMaskForCond, UseMaskForGaps);
 
   return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
                                            Alignment, AddressSpace, CostKind,
                                            UseMaskForCond, UseMaskForGaps);
 }

llvm/test/Analysis/CostModel/X86/interleaved-load-half.ll

+; RUN: opt -S -loop-vectorize -debug-only=loop-vectorize -mattr=avx512fp16 %s 2>&1 | FileCheck %s
+target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
+target triple = "i386-unknown-linux-gnu"
+
+@src = common local_unnamed_addr global [120 x half] zeroinitializer, align 4
+@dst = common local_unnamed_addr global [120 x half] zeroinitializer, align 4
+
+; Function Attrs: norecurse nounwind
+define void @stride8(half %k, i32 %width_) {
+entry:
+
+; CHECK: Found an estimated cost of 148 for VF 32 For instruction:   %0 = load half
+
+  %cmp72 = icmp sgt i32 %width_, 0
+  br i1 %cmp72, label %for.body.lr.ph, label %for.cond.cleanup
+
+for.body.lr.ph:                                   ; preds = %entry
+  br label %for.body
+
+for.cond.cleanup.loopexit:                        ; preds = %for.body
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup.loopexit, %entry
+  ret void
+
+for.body:                                         ; preds = %for.body.lr.ph, %for.body
+  %i.073 = phi i32 [ 0, %for.body.lr.ph ], [ %add46, %for.body ]
+  %arrayidx = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %i.073
+  %0 = load half, half* %arrayidx, align 4
+  %mul = fmul fast half %0, %k
+  %arrayidx2 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %i.073
+  %1 = load half, half* %arrayidx2, align 4
+  %add3 = fadd fast half %1, %mul
+  store half %add3, half* %arrayidx2, align 4
+  %add4 = or i32 %i.073, 1
+  %arrayidx5 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add4
+  %2 = load half, half* %arrayidx5, align 4
+  %mul6 = fmul fast half %2, %k
+  %arrayidx8 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add4
+  %3 = load half, half* %arrayidx8, align 4
+  %add9 = fadd fast half %3, %mul6
+  store half %add9, half* %arrayidx8, align 4
+  %add10 = or i32 %i.073, 2
+  %arrayidx11 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add10
+  %4 = load half, half* %arrayidx11, align 4
+  %mul12 = fmul fast half %4, %k
+  %arrayidx14 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add10
+  %5 = load half, half* %arrayidx14, align 4
+  %add15 = fadd fast half %5, %mul12
+  store half %add15, half* %arrayidx14, align 4
+  %add16 = or i32 %i.073, 3
+  %arrayidx17 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add16
+  %6 = load half, half* %arrayidx17, align 4
+  %mul18 = fmul fast half %6, %k
+  %arrayidx20 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add16
+  %7 = load half, half* %arrayidx20, align 4
+  %add21 = fadd fast half %7, %mul18
+  store half %add21, half* %arrayidx20, align 4
+  %add22 = or i32 %i.073, 4
+  %arrayidx23 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add22
+  %8 = load half, half* %arrayidx23, align 4
+  %mul24 = fmul fast half %8, %k
+  %arrayidx26 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add22
+  %9 = load half, half* %arrayidx26, align 4
+  %add27 = fadd fast half %9, %mul24
+  store half %add27, half* %arrayidx26, align 4
+  %add28 = or i32 %i.073, 5
+  %arrayidx29 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add28
+  %10 = load half, half* %arrayidx29, align 4
+  %mul30 = fmul fast half %10, %k
+  %arrayidx32 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add28
+  %11 = load half, half* %arrayidx32, align 4
+  %add33 = fadd fast half %11, %mul30
+  store half %add33, half* %arrayidx32, align 4
+  %add34 = or i32 %i.073, 6
+  %arrayidx35 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add34
+  %12 = load half, half* %arrayidx35, align 4
+  %mul36 = fmul fast half %12, %k
+  %arrayidx38 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add34
+  %13 = load half, half* %arrayidx38, align 4
+  %add39 = fadd fast half %13, %mul36
+  store half %add39, half* %arrayidx38, align 4
+  %add40 = or i32 %i.073, 7
+  %arrayidx41 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add40
+  %14 = load half, half* %arrayidx41, align 4
+  %mul42 = fmul fast half %14, %k
+  %arrayidx44 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add40
+  %15 = load half, half* %arrayidx44, align 4
+  %add45 = fadd fast half %15, %mul42
+  store half %add45, half* %arrayidx44, align 4
+  %add46 = add nuw nsw i32 %i.073, 8
+  %cmp = icmp slt i32 %add46, %width_
+  br i1 %cmp, label %for.body, label %for.cond.cleanup.loopexit
+}
+
+; Function Attrs: norecurse nounwind
+define void @stride3(half %k, i32 %width_) {
+entry:
+
+; CHECK: Found an estimated cost of 18 for VF 32 For instruction:   %0 = load half
+
+  %cmp27 = icmp sgt i32 %width_, 0
+  br i1 %cmp27, label %for.body.lr.ph, label %for.cond.cleanup
+
+for.body.lr.ph:                                   ; preds = %entry
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  ret void
+
+for.body:                                         ; preds = %for.body.lr.ph, %for.body
+  %i.028 = phi i32 [ 0, %for.body.lr.ph ], [ %add16, %for.body ]
+  %arrayidx = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %i.028
+  %0 = load half, half* %arrayidx, align 4
+  %mul = fmul fast half %0, %k
+  %arrayidx2 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %i.028
+  %1 = load half, half* %arrayidx2, align 4
+  %add3 = fadd fast half %1, %mul
+  store half %add3, half* %arrayidx2, align 4
+  %add4 = add nuw nsw i32 %i.028, 1
+  %arrayidx5 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add4
+  %2 = load half, half* %arrayidx5, align 4
+  %mul6 = fmul fast half %2, %k
+  %arrayidx8 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add4
+  %3 = load half, half* %arrayidx8, align 4
+  %add9 = fadd fast half %3, %mul6
+  store half %add9, half* %arrayidx8, align 4
+  %add10 = add nuw nsw i32 %i.028, 2
+  %arrayidx11 = getelementptr inbounds [120 x half], [120 x half]* @src, i32 0, i32 %add10
+  %4 = load half, half* %arrayidx11, align 4
+  %mul12 = fmul fast half %4, %k
+  %arrayidx14 = getelementptr inbounds [120 x half], [120 x half]* @dst, i32 0, i32 %add10
+  %5 = load half, half* %arrayidx14, align 4
+  %add15 = fadd fast half %5, %mul12
+  store half %add15, half* %arrayidx14, align 4
+  %add16 = add nuw nsw i32 %i.028, 3
+  %cmp = icmp slt i32 %add16, %width_
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+}
+

llvm/test/Analysis/CostModel/X86/shuffle-broadcast-fp16.ll

+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; RUN: opt < %s -mtriple=x86_64-unknown-linux-gnu -cost-model -analyze -mattr=+avx512fp16 | FileCheck %s
+
+define void @test_vXf16(<2 x half> %src32, <4 x half> %src64, <8 x half> %src128, <16 x half> %src256, <32 x half> %src512) {
+; CHECK-LABEL: 'test_vXf16'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V32 = shufflevector <2 x half> %src32, <2 x half> undef, <2 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V64 = shufflevector <4 x half> %src64, <4 x half> undef, <4 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> zeroinitializer
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+  %V32  = shufflevector <2 x half> %src32, <2 x half> undef, <2 x i32> zeroinitializer
+  %V64  = shufflevector <4 x half> %src64, <4 x half> undef, <4 x i32> zeroinitializer
+  %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> zeroinitializer
+  %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> zeroinitializer
+  %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> zeroinitializer
+  ret void
+}

llvm/test/Analysis/CostModel/X86/shuffle-reverse-fp16.ll

+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; RUN: opt < %s -mtriple=x86_64-unknown-linux-gnu -cost-model -analyze -mattr=+avx512fp16 | FileCheck %s
+
+define void @test_vXf16(<2 x half> %src32, <4 x half> %src64, <8 x half> %src128, <16 x half> %src256, <32 x half> %src512) {
+; CHECK-LABEL: 'test_vXf16'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V32 = shufflevector <2 x half> %src32, <2 x half> undef, <2 x i32> <i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V64 = shufflevector <4 x half> %src64, <4 x half> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> <i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+  %V32  = shufflevector <2 x half> %src32, <2 x half> undef, <2 x i32> <i32 1, i32 0>
+  %V64  = shufflevector <4 x half> %src64, <4 x half> undef, <4 x i32> <i32 3, i32 2, i32 1, i32 0>
+  %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> <i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  ret void
+}

llvm/test/Analysis/CostModel/X86/shuffle-single-src-fp16.ll

+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; RUN: opt < %s -mtriple=x86_64-unknown-linux-gnu -cost-model -analyze -mattr=+avx512fp16 | FileCheck %s
+
+define void @test_vXf16(<8 x half> %src128, <16 x half> %src256, <32 x half> %src512, <64 x half> %src1024) {
+; CHECK-LABEL: 'test_vXf16'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 1 for instruction: %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 6, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 13, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> <i32 31, i32 30, i32 20, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 11, i32 9, i32 8, i32 7, i32 11, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 4 for instruction: %V1024 = shufflevector <64 x half> %src1024, <64 x half> undef, <64 x i32> <i32 63, i32 62, i32 61, i32 60, i32 59, i32 58, i32 57, i32 56, i32 55, i32 54, i32 53, i32 52, i32 51, i32 50, i32 49, i32 48, i32 47, i32 46, i32 45, i32 44, i32 43, i32 42, i32 41, i32 40, i32 39, i32 38, i32 37, i32 36, i32 35, i32 34, i32 33, i32 32, i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 20, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+  %V128 = shufflevector <8 x half> %src128, <8 x half> undef, <8 x i32> <i32 7, i32 6, i32 6, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V256 = shufflevector <16 x half> %src256, <16 x half> undef, <16 x i32> <i32 15, i32 14, i32 13, i32 13, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V512 = shufflevector <32 x half> %src512, <32 x half> undef, <32 x i32> <i32 31, i32 30, i32 20, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 11, i32 9, i32 8, i32 7, i32 11, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V1024 = shufflevector <64 x half> %src1024, <64 x half> undef, <64 x i32> <i32 63, i32 62, i32 61, i32 60, i32 59, i32 58, i32 57, i32 56, i32 55, i32 54, i32 53, i32 52, i32 51, i32 50, i32 49, i32 48, i32 47, i32 46, i32 45, i32 44, i32 43, i32 42, i32 41, i32 40, i32 39, i32 38, i32 37, i32 36, i32 35, i32 34, i32 33, i32 32, i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 20, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  ret void
+}

llvm/test/Analysis/CostModel/X86/shuffle-two-src-fp16.ll

+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py
+; RUN: opt < %s -mtriple=x86_64-unknown-linux-gnu -cost-model -analyze -mattr=+avx512fp16 | FileCheck %s
+
+define void @test_vXf16(<8 x half> %src128, <16 x half> %src256, <32 x half> %src512, <64 x half> %src1024, <8 x half> %src128_1, <16 x half> %src256_1, <32 x half> %src512_1, <64 x half> %src1024_1) {
+; CHECK-LABEL: 'test_vXf16'
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V128 = shufflevector <8 x half> %src128, <8 x half> %src128_1, <8 x i32> <i32 7, i32 6, i32 6, i32 8, i32 9, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V256 = shufflevector <16 x half> %src256, <16 x half> %src256_1, <16 x i32> <i32 15, i32 14, i32 13, i32 20, i32 21, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 2 for instruction: %V512 = shufflevector <32 x half> %src512, <32 x half> %src512_1, <32 x i32> <i32 31, i32 30, i32 45, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 38, i32 11, i32 11, i32 9, i32 8, i32 7, i32 11, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 12 for instruction: %V1024 = shufflevector <64 x half> %src1024, <64 x half> %src1024_1, <64 x i32> <i32 63, i32 62, i32 71, i32 60, i32 59, i32 58, i32 57, i32 56, i32 55, i32 54, i32 53, i32 52, i32 51, i32 50, i32 49, i32 48, i32 47, i32 46, i32 45, i32 44, i32 43, i32 42, i32 41, i32 40, i32 39, i32 38, i32 37, i32 36, i32 35, i32 34, i32 33, i32 32, i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 20, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 66, i32 2, i32 1, i32 0>
+; CHECK-NEXT:  Cost Model: Found an estimated cost of 0 for instruction: ret void
+;
+  %V128 = shufflevector <8 x half> %src128, <8 x half> %src128_1, <8 x i32> <i32 7, i32 6, i32 6, i32 8, i32 9, i32 2, i32 1, i32 0>
+  %V256 = shufflevector <16 x half> %src256, <16 x half> %src256_1, <16 x i32> <i32 15, i32 14, i32 13, i32 20, i32 21, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V512 = shufflevector <32 x half> %src512, <32 x half> %src512_1, <32 x i32> <i32 31, i32 30, i32 45, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 22, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 38, i32 11, i32 11, i32 9, i32 8, i32 7, i32 11, i32 5, i32 4, i32 3, i32 2, i32 1, i32 0>
+  %V1024 = shufflevector <64 x half> %src1024, <64 x half> %src1024_1, <64 x i32> <i32 63, i32 62, i32 71, i32 60, i32 59, i32 58, i32 57, i32 56, i32 55, i32 54, i32 53, i32 52, i32 51, i32 50, i32 49, i32 48, i32 47, i32 46, i32 45, i32 44, i32 43, i32 42, i32 41, i32 40, i32 39, i32 38, i32 37, i32 36, i32 35, i32 34, i32 33, i32 32, i32 31, i32 30, i32 29, i32 28, i32 27, i32 26, i32 25, i32 24, i32 23, i32 20, i32 21, i32 20, i32 19, i32 18, i32 17, i32 16, i32 15, i32 14, i32 13, i32 12, i32 11, i32 10, i32 9, i32 8, i32 7, i32 6, i32 5, i32 4, i32 66, i32 2, i32 1, i32 0>
+  ret void
+}

llvm/test/CodeGen/MIR/X86/inline-asm-registers.mir

 liveins:
   - { reg: '$rdi' }
   - { reg: '$rsi' }
 body: |
   bb.0.entry:
     liveins: $rdi, $rsi
 
   ; CHECK-LABEL: name: test
-  ; CHECK: INLINEASM &foo, 0 /* attdialect */, 4325386 /* regdef:GR64 */, def $rsi, 4325386 /* regdef:GR64 */, def dead $rdi,
-    INLINEASM &foo, 0, 4325386, def $rsi, 4325386, def dead $rdi, 2147549193, killed $rdi, 2147483657, killed $rsi, 12, implicit-def dead early-clobber $eflags
+  ; CHECK: INLINEASM &foo, 0 /* attdialect */, 4390922 /* regdef:GR64 */, def $rsi, 4390922 /* regdef:GR64 */, def dead $rdi,
+    INLINEASM &foo, 0, 4390922, def $rsi, 4390922, def dead $rdi, 2147549193, killed $rdi, 2147483657, killed $rsi, 12, implicit-def dead early-clobber $eflags
     $rax = MOV64rr killed $rsi
     RETQ killed $rax
 ...
 ---
 name:            test2
 tracksRegLiveness: true
 liveins:
   - { reg: '$rdi' }
   - { reg: '$rsi' }
 body: |
   bb.0.entry:
     liveins: $rdi, $rsi
 
   ; Verify that the register ties are preserved.
   ; CHECK-LABEL: name: test2
-  ; CHECK: INLINEASM &foo, 0 /* attdialect */, 4325386 /* regdef:GR64 */, def $rsi, 4325386 /* regdef:GR64 */, def dead $rdi, 2147549193 /* reguse tiedto:$1 */, killed $rdi(tied-def 5), 2147483657 /* reguse tiedto:$0 */, killed $rsi(tied-def 3), 12 /* clobber */, implicit-def dead early-clobber $eflags
-    INLINEASM &foo, 0, 4325386, def $rsi, 4325386, def dead $rdi, 2147549193, killed $rdi(tied-def 5), 2147483657, killed $rsi(tied-def 3), 12, implicit-def dead early-clobber $eflags
+  ; CHECK: INLINEASM &foo, 0 /* attdialect */, 4390922 /* regdef:GR64 */, def $rsi, 4390922 /* regdef:GR64 */, def dead $rdi, 2147549193 /* reguse tiedto:$1 */, killed $rdi(tied-def 5), 2147483657 /* reguse tiedto:$0 */, killed $rsi(tied-def 3), 12 /* clobber */, implicit-def dead early-clobber $eflags
+    INLINEASM &foo, 0, 4390922, def $rsi, 4390922, def dead $rdi, 2147549193, killed $rdi(tied-def 5), 2147483657, killed $rsi(tied-def 3), 12, implicit-def dead early-clobber $eflags
     $rax = MOV64rr killed $rsi
     RETQ killed $rax
 ...

llvm/test/CodeGen/X86/avx512fp16-insert-extract.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512fp16,+avx512vl | FileCheck %s --check-prefixes=CHECK
+
+define <8 x half> @extract_v16f16_v8f16_0(<16 x half> %x) {
+; CHECK-LABEL: extract_v16f16_v8f16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <16 x half> %x, <16 x half> undef, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  ret <8 x half> %a
+}
+
+define <8 x half> @extract_v16f16_v8f16_1(<16 x half> %x) {
+; CHECK-LABEL: extract_v16f16_v8f16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vextractf128 $1, %ymm0, %xmm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <16 x half> %x, <16 x half> undef, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  ret <8 x half> %a
+}
+
+define <8 x half> @extract_v32f16_v8f16_0(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v8f16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $xmm0 killed $xmm0 killed $zmm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+  ret <8 x half> %a
+}
+
+define <8 x half> @extract_v32f16_v8f16_1(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v8f16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vextractf128 $1, %ymm0, %xmm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <8 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  ret <8 x half> %a
+}
+
+define <8 x half> @extract_v32f16_v8f16_2(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v8f16_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vextractf32x4 $2, %zmm0, %xmm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <8 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23>
+  ret <8 x half> %a
+}
+
+define <8 x half> @extract_v32f16_v8f16_3(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v8f16_3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vextractf32x4 $3, %zmm0, %xmm0
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <8 x i32> <i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <8 x half> %a
+}
+
+define <16 x half> @extract_v32f16_v81616_0(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v81616_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $ymm0 killed $ymm0 killed $zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  ret <16 x half> %a
+}
+
+define <16 x half> @extract_v32f16_v81616_1(<32 x half> %x) {
+; CHECK-LABEL: extract_v32f16_v81616_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vextractf64x4 $1, %zmm0, %ymm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <32 x half> %x, <32 x half> undef, <16 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <16 x half> %a
+}
+
+define <16 x half> @concat_v8f16(<8 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: concat_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $xmm0 killed $xmm0 def $ymm0
+; CHECK-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %x, <8 x half> %y, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  ret <16 x half> %a
+}
+
+define <32 x half> @concat_v16f16(<16 x half> %x, <16 x half> %y) {
+; CHECK-LABEL: concat_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $ymm0 killed $ymm0 def $zmm0
+; CHECK-NEXT:    vinsertf64x4 $1, %ymm1, %zmm0, %zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <16 x half> %x, <16 x half> %y, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x half> %a
+}
+
+define <16 x half> @concat_zero_v8f16(<8 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: concat_zero_v8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm0, %xmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %x, <8 x half> zeroinitializer, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  ret <16 x half> %a
+}
+
+define <32 x half> @concat_zero_v16f16(<16 x half> %x, <16 x half> %y) {
+; CHECK-LABEL: concat_zero_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %ymm0, %ymm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <16 x half> %x, <16 x half> zeroinitializer, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x half> %a
+}
+
+define <32 x half> @insert_v8f16_v32f16_0(<32 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: insert_v8f16_v32f16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vinsertf32x4 $0, %xmm1, %zmm0, %zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %y, <8 x half> undef, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  %b = shufflevector <32 x half> %x, <32 x half> %a, <32 x i32> <i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x half> %b
+}
+
+define <32 x half> @insert_v8f16_v32f16_1(<32 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: insert_v8f16_v32f16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vinsertf32x4 $1, %xmm1, %zmm0, %zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %y, <8 x half> undef, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  %b = shufflevector <32 x half> %x, <32 x half> %a, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x half> %b
+}
+
+define <32 x half> @insert_v8f16_v32f16_2(<32 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: insert_v8f16_v32f16_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vinsertf32x4 $2, %xmm1, %zmm0, %zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %y, <8 x half> undef, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  %b = shufflevector <32 x half> %x, <32 x half> %a, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+  ret <32 x half> %b
+}
+
+define <32 x half> @insert_v8f16_v32f16_3(<32 x half> %x, <8 x half> %y) {
+; CHECK-LABEL: insert_v8f16_v32f16_3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vinsertf32x4 $3, %xmm1, %zmm0, %zmm0
+; CHECK-NEXT:    retq
+  %a = shufflevector <8 x half> %y, <8 x half> undef, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  %b = shufflevector <32 x half> %x, <32 x half> %a, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39>
+  ret <32 x half> %b
+}

llvm/test/CodeGen/X86/avx512fp16-mov.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512fp16 | FileCheck %s --check-prefixes=CHECK,X64
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+avx512fp16 | FileCheck %s --check-prefixes=CHECK,X86
+
+define <8 x half> @broadcastph128(half* %x) {
+; X64-LABEL: broadcastph128:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph128:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpbroadcastw (%eax), %xmm0
+; X86-NEXT:    retl
+  %l1 = load half, half* %x, align 2
+  %vec = insertelement <8 x half> undef, half %l1, i32 0
+  %res = shufflevector <8 x half> %vec, <8 x half> undef, <8 x i32> zeroinitializer
+  ret <8 x half> %res
+}
+
+define <16 x half> @broadcastph256(half* %x) {
+; X64-LABEL: broadcastph256:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw (%rdi), %ymm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph256:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpbroadcastw (%eax), %ymm0
+; X86-NEXT:    retl
+  %l1 = load half, half* %x, align 2
+  %vec = insertelement <16 x half> undef, half %l1, i32 0
+  %res = shufflevector <16 x half> %vec, <16 x half> undef, <16 x i32> zeroinitializer
+  ret <16 x half> %res
+}
+
+define <32 x half> @broadcastph512(half* %x) {
+; X64-LABEL: broadcastph512:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw (%rdi), %zmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph512:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpbroadcastw (%eax), %zmm0
+; X86-NEXT:    retl
+  %l1 = load half, half* %x, align 2
+  %vec = insertelement <32 x half> undef, half %l1, i32 0
+  %res = shufflevector <32 x half> %vec, <32 x half> undef, <32 x i32> zeroinitializer
+  ret <32 x half> %res
+}
+
+define <8 x half> @broadcastph128_scalar(half %x) {
+; X64-LABEL: broadcastph128_scalar:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw %xmm0, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph128_scalar:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+  %vec = insertelement <8 x half> undef, half %x, i32 0
+  %res = shufflevector <8 x half> %vec, <8 x half> undef, <8 x i32> zeroinitializer
+  ret <8 x half> %res
+}
+
+define <16 x half> @broadcastph256_scalar(half %x) {
+; X64-LABEL: broadcastph256_scalar:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw %xmm0, %ymm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph256_scalar:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %ymm0
+; X86-NEXT:    retl
+  %vec = insertelement <16 x half> undef, half %x, i32 0
+  %res = shufflevector <16 x half> %vec, <16 x half> undef, <16 x i32> zeroinitializer
+  ret <16 x half> %res
+}
+
+define <32 x half> @broadcastph512_scalar(half %x) {
+; X64-LABEL: broadcastph512_scalar:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw %xmm0, %zmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: broadcastph512_scalar:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %zmm0
+; X86-NEXT:    retl
+  %vec = insertelement <32 x half> undef, half %x, i32 0
+  %res = shufflevector <32 x half> %vec, <32 x half> undef, <32 x i32> zeroinitializer
+  ret <32 x half> %res
+}
+
+define <8 x half> @broadcastph128_reg(<8 x half> %x) {
+; CHECK-LABEL: broadcastph128_reg:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpbroadcastw %xmm0, %xmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  %res = shufflevector <8 x half> %x, <8 x half> undef, <8 x i32> zeroinitializer
+  ret <8 x half> %res
+}
+
+define <16 x half> @broadcastph256_reg(<16 x half> %x) {
+; CHECK-LABEL: broadcastph256_reg:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpbroadcastw %xmm0, %ymm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  %res = shufflevector <16 x half> %x, <16 x half> undef, <16 x i32> zeroinitializer
+  ret <16 x half> %res
+}
+
+define <32 x half> @broadcastph512_reg(<32 x half> %x) {
+; CHECK-LABEL: broadcastph512_reg:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpbroadcastw %xmm0, %zmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  %res = shufflevector <32 x half> %x, <32 x half> undef, <32 x i32> zeroinitializer
+  ret <32 x half> %res
+}
+
+define i16 @test1(half %x) {
+; X64-LABEL: test1:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %xmm0, %eax
+; X64-NEXT:    # kill: def $ax killed $ax killed $eax
+; X64-NEXT:    retq
+;
+; X86-LABEL: test1:
+; X86:       # %bb.0:
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    retl
+   %res = bitcast half %x to i16
+   ret i16 %res
+}
+
+define <8 x i16> @test2(i16 %x) {
+; X64-LABEL: test2:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovd %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test2:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <8 x i16>undef, i16 %x, i32 0
+   ret <8 x i16>%res
+}
+
+define <8 x i16> @test4(i16* %x) {
+; X64-LABEL: test4:
+; X64:       # %bb.0:
+; X64-NEXT:    vpbroadcastw (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test4:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpbroadcastw (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load i16, i16* %x
+   %res = insertelement <8 x i16>undef, i16 %y, i32 0
+   ret <8 x i16>%res
+}
+
+define void @test5(half %x, half* %y) {
+; X64-LABEL: test5:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: test5:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   store half %x, half* %y, align 2
+   ret void
+}
+
+define half @test7(i16* %x) {
+; X64-LABEL: test7:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test7:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load i16, i16* %x
+   %res = bitcast i16 %y to half
+   ret half %res
+}
+
+define <8 x i16> @test10(i16* %x) {
+; X64-LABEL: test10:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test10:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovw (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load i16, i16* %x, align 2
+   %res = insertelement <8 x i16>zeroinitializer, i16 %y, i32 0
+   ret <8 x i16>%res
+}
+
+define <16 x i16> @test10b(i16* %x) {
+; X64-LABEL: test10b:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test10b:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovw (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load i16, i16* %x, align 2
+   %res = insertelement <16 x i16>zeroinitializer, i16 %y, i32 0
+   ret <16 x i16>%res
+}
+
+define <32 x i16> @test10c(i16* %x) {
+; X64-LABEL: test10c:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test10c:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovw (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load i16, i16* %x, align 2
+   %res = insertelement <32 x i16>zeroinitializer, i16 %y, i32 0
+   ret <32 x i16>%res
+}
+
+define <8 x half> @test11(half* %x) {
+; X64-LABEL: test11:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test11:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load half, half* %x, align 2
+   %res = insertelement <8 x half>zeroinitializer, half %y, i32 0
+   ret <8 x half>%res
+}
+
+define <16 x half> @test11b(half* %x) {
+; X64-LABEL: test11b:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test11b:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load half, half* %x, align 2
+   %res = insertelement <16 x half>zeroinitializer, half %y, i32 0
+   ret <16 x half>%res
+}
+
+define <32 x half> @test11c(half* %x) {
+; X64-LABEL: test11c:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test11c:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh (%eax), %xmm0
+; X86-NEXT:    retl
+   %y = load half, half* %x, align 2
+   %res = insertelement <32 x half>zeroinitializer, half %y, i32 0
+   ret <32 x half>%res
+}
+
+define <8 x half> @test14(half %x) {
+; X64-LABEL: test14:
+; X64:       # %bb.0:
+; X64-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; X64-NEXT:    vmovsh %xmm0, %xmm1, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test14:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <8 x half>zeroinitializer, half %x, i32 0
+   ret <8 x half>%res
+}
+
+define <16 x half> @test14b(half %x) {
+; X64-LABEL: test14b:
+; X64:       # %bb.0:
+; X64-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; X64-NEXT:    vmovsh %xmm0, %xmm1, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test14b:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <16 x half>zeroinitializer, half %x, i32 0
+   ret <16 x half>%res
+}
+
+define <32 x half> @test14c(half %x) {
+; X64-LABEL: test14c:
+; X64:       # %bb.0:
+; X64-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; X64-NEXT:    vmovsh %xmm0, %xmm1, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test14c:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <32 x half>zeroinitializer, half %x, i32 0
+   ret <32 x half>%res
+}
+
+define <8 x i16> @test15(i16 %x) {
+; X64-LABEL: test15:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test15:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <8 x i16>zeroinitializer, i16 %x, i32 0
+   ret <8 x i16>%res
+}
+
+define <16 x i16> @test16(i16 %x) {
+; X64-LABEL: test16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <16 x i16>zeroinitializer, i16 %x, i32 0
+   ret <16 x i16>%res
+}
+
+define <32 x i16> @test17(i16 %x) {
+; X64-LABEL: test17:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test17:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <32 x i16>zeroinitializer, i16 %x, i32 0
+   ret <32 x i16>%res
+}
+
+define <8 x i16> @test18(i16 %x) {
+; X64-LABEL: test18:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovd %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test18:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+   %res = insertelement <8 x i16> undef, i16 %x, i32 0
+   ret <8 x i16>%res
+}
+
+define <16 x i16> @test19(i16 %x) {
+; X64-LABEL: test19:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovd %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test19:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %ymm0
+; X86-NEXT:    retl
+   %res = insertelement <16 x i16> undef, i16 %x, i32 0
+   ret <16 x i16>%res
+}
+
+define <32 x i16> @test20(i16 %x) {
+; X64-LABEL: test20:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovd %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test20:
+; X86:       # %bb.0:
+; X86-NEXT:    vpbroadcastw {{[0-9]+}}(%esp), %zmm0
+; X86-NEXT:    retl
+   %res = insertelement <32 x i16> undef, i16 %x, i32 0
+   ret <32 x i16>%res
+}
+
+@g8f16 = external global <8 x half>
+@g8f16u = external global <8 x half>, align 8
+@g16f16 = external global <16 x half>
+@g16f16u = external global <16 x half>, align 8
+@g32f16 = external global <32 x half>
+@g32f16u = external global <32 x half>, align 8
+
+define <32 x half> @load32f16(<32 x half>* %a) {
+; X64-LABEL: load32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovaps (%rdi), %zmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: load32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovaps (%eax), %zmm0
+; X86-NEXT:    retl
+  %res = load <32 x half>, <32 x half>* %a
+  ret <32 x half> %res
+}
+
+define <32 x half> @load32f16mask(<32 x half>* %a, <32 x half> %b, i32 %c) {
+; X64-LABEL: load32f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load32f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i32 %c to <32 x i1>
+  %res0 = load <32 x half>, <32 x half>* %a
+  %res = select <32 x i1> %msk, <32 x half> %res0, <32 x half> %b
+  ret <32 x half> %res
+}
+
+define <32 x half> @load32f16maskz(<32 x half>* %a, i32 %c) {
+; X64-LABEL: load32f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load32f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i32 %c to <32 x i1>
+  %res0 = load <32 x half>, <32 x half>* %a
+  %res = select <32 x i1> %msk, <32 x half> %res0, <32 x half> zeroinitializer
+  ret <32 x half> %res
+}
+
+define <32 x half> @loadu32f16(<32 x half>* %a) {
+; X64-LABEL: loadu32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovups (%rdi), %zmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovups (%eax), %zmm0
+; X86-NEXT:    retl
+  %res = load <32 x half>, <32 x half>* %a, align 8
+  ret <32 x half> %res
+}
+
+define <32 x half> @loadu32f16mask(<32 x half>* %a, <32 x half> %b, i32 %c) {
+; X64-LABEL: loadu32f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu32f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i32 %c to <32 x i1>
+  %res0 = load <32 x half>, <32 x half>* %a, align 8
+  %res = select <32 x i1> %msk, <32 x half> %res0, <32 x half> %b
+  ret <32 x half> %res
+}
+
+define <32 x half> @loadu32f16maskz(<32 x half>* %a, i32 %c) {
+; X64-LABEL: loadu32f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu32f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i32 %c to <32 x i1>
+  %res0 = load <32 x half>, <32 x half>* %a, align 8
+  %res = select <32 x i1> %msk, <32 x half> %res0, <32 x half> zeroinitializer
+  ret <32 x half> %res
+}
+
+define void @store32f16(<32 x half> %a) {
+; X64-LABEL: store32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g32f16@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovaps %zmm0, (%rax)
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: store32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovaps %zmm0, g32f16
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  store <32 x half> %a, <32 x half>* @g32f16
+  ret void
+}
+
+define void @storeu32f16(<32 x half> %a) {
+; X64-LABEL: storeu32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g32f16u@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovups %zmm0, (%rax)
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovups %zmm0, g32f16u
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  store <32 x half> %a, <32 x half>* @g32f16u, align 8
+  ret void
+}
+
+declare void @llvm.masked.store.v32f16.p0v32f16(<32 x half>, <32 x half>*, i32, <32 x i1>)
+declare <32 x half> @llvm.masked.load.v32f16.p0v32f16(<32 x half>*, i32,  <32 x i1>, <32 x half>)
+
+define void @storeu32f16mask(<32 x i1> %mask, <32 x half>* %addr, <32 x half> %val) {
+; X64-LABEL: storeu32f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X64-NEXT:    vpmovb2m %ymm0, %k1
+; X64-NEXT:    vmovdqu16 %zmm1, (%rdi) {%k1}
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu32f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X86-NEXT:    vpmovb2m %ymm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 %zmm1, (%eax) {%k1}
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  call void @llvm.masked.store.v32f16.p0v32f16(<32 x half> %val, <32 x half>* %addr, i32 4, <32 x i1>%mask)
+  ret void
+}
+
+define <32 x half> @maskloadu32f16(<32 x half>* %addr, <32 x half> %val, <32 x i1> %mask) {
+; X64-LABEL: maskloadu32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %ymm1, %ymm1
+; X64-NEXT:    vpmovb2m %ymm1, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskloadu32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %ymm1, %ymm1
+; X86-NEXT:    vpmovb2m %ymm1, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1}
+; X86-NEXT:    retl
+  %res = call <32 x half> @llvm.masked.load.v32f16.p0v32f16(<32 x half>* %addr, i32 4, <32 x i1> %mask, <32 x half> %val)
+  ret <32 x half> %res
+}
+
+define <32 x half> @maskuloadu32f16(<32 x half>* %addr, <32 x i1> %mask) {
+; X64-LABEL: maskuloadu32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X64-NEXT:    vpmovb2m %ymm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskuloadu32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X86-NEXT:    vpmovb2m %ymm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <32 x half> @llvm.masked.load.v32f16.p0v32f16(<32 x half>* %addr, i32 4, <32 x i1> %mask, <32 x half> undef)
+  ret <32 x half> %res
+}
+
+define <32 x half> @maskzloadu32f16(<32 x half>* %addr, <32 x i1> %mask) {
+; X64-LABEL: maskzloadu32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X64-NEXT:    vpmovb2m %ymm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %zmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskzloadu32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %ymm0, %ymm0
+; X86-NEXT:    vpmovb2m %ymm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %zmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <32 x half> @llvm.masked.load.v32f16.p0v32f16(<32 x half>* %addr, i32 4, <32 x i1> %mask, <32 x half> zeroinitializer)
+  ret <32 x half> %res
+}
+
+define <32 x half> @movrr32f16(<32 x half> %a, <32 x half> %b) {
+; CHECK-LABEL: movrr32f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %zmm1, %zmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  ret <32 x half> %b
+}
+
+define <32 x half> @movrrk32f16(<32 x half> %a, <32 x half> %b, i32 %msk) {
+; X64-LABEL: movrrk32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vpblendmw %zmm0, %zmm1, %zmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrk32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vpblendmw %zmm0, %zmm1, %zmm0 {%k1}
+; X86-NEXT:    retl
+  %mask = bitcast i32 %msk to <32 x i1>
+  %res = select <32 x i1> %mask, <32 x half> %a, <32 x half> %b
+  ret <32 x half> %res
+}
+
+define <32 x half> @movrrkz32f16(<32 x half> %a, i32 %msk) {
+; X64-LABEL: movrrkz32f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vmovdqu16 %zmm0, %zmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrkz32f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 %zmm0, %zmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %mask = bitcast i32 %msk to <32 x i1>
+  %res = select <32 x i1> %mask, <32 x half> %a, <32 x half> zeroinitializer
+  ret <32 x half> %res
+}
+
+define <16 x half> @load16f16(<16 x half>* %a) {
+; X64-LABEL: load16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovaps (%rdi), %ymm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: load16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovaps (%eax), %ymm0
+; X86-NEXT:    retl
+  %res = load <16 x half>, <16 x half>* %a
+  ret <16 x half> %res
+}
+
+define <16 x half> @load16f16mask(<16 x half>* %a, <16 x half> %b, i16 %c) {
+; X64-LABEL: load16f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load16f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i16 %c to <16 x i1>
+  %res0 = load <16 x half>, <16 x half>* %a
+  %res = select <16 x i1> %msk, <16 x half> %res0, <16 x half> %b
+  ret <16 x half> %res
+}
+
+define <16 x half> @load16f16maskz(<16 x half>* %a, i16 %c) {
+; X64-LABEL: load16f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load16f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i16 %c to <16 x i1>
+  %res0 = load <16 x half>, <16 x half>* %a
+  %res = select <16 x i1> %msk, <16 x half> %res0, <16 x half> zeroinitializer
+  ret <16 x half> %res
+}
+
+define <16 x half> @loadu16f16(<16 x half>* %a) {
+; X64-LABEL: loadu16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovups (%rdi), %ymm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovups (%eax), %ymm0
+; X86-NEXT:    retl
+  %res = load <16 x half>, <16 x half>* %a, align 8
+  ret <16 x half> %res
+}
+
+define <16 x half> @loadu16f16mask(<16 x half>* %a, <16 x half> %b, i16 %c) {
+; X64-LABEL: loadu16f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu16f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i16 %c to <16 x i1>
+  %res0 = load <16 x half>, <16 x half>* %a, align 8
+  %res = select <16 x i1> %msk, <16 x half> %res0, <16 x half> %b
+  ret <16 x half> %res
+}
+
+define <16 x half> @loadu16f16maskz(<16 x half>* %a, i16 %c) {
+; X64-LABEL: loadu16f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu16f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i16 %c to <16 x i1>
+  %res0 = load <16 x half>, <16 x half>* %a, align 8
+  %res = select <16 x i1> %msk, <16 x half> %res0, <16 x half> zeroinitializer
+  ret <16 x half> %res
+}
+
+define void @store16f16(<16 x half> %a) {
+; X64-LABEL: store16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g16f16@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovaps %ymm0, (%rax)
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: store16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovaps %ymm0, g16f16
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  store <16 x half> %a, <16 x half>* @g16f16
+  ret void
+}
+
+define void @storeu16f16(<16 x half> %a) {
+; X64-LABEL: storeu16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g16f16u@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovups %ymm0, (%rax)
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovups %ymm0, g16f16u
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  store <16 x half> %a, <16 x half>* @g16f16u, align 8
+  ret void
+}
+
+declare void @llvm.masked.store.v16f16.p0v16f16(<16 x half>, <16 x half>*, i32, <16 x i1>)
+declare <16 x half> @llvm.masked.load.v16f16.p0v16f16(<16 x half>*, i32,  <16 x i1>, <16 x half>)
+
+define void @storeu16f16mask(<16 x i1> %mask, <16 x half>* %addr, <16 x half> %val) {
+; X64-LABEL: storeu16f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X64-NEXT:    vpmovb2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 %ymm1, (%rdi) {%k1}
+; X64-NEXT:    vzeroupper
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu16f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X86-NEXT:    vpmovb2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 %ymm1, (%eax) {%k1}
+; X86-NEXT:    vzeroupper
+; X86-NEXT:    retl
+  call void @llvm.masked.store.v16f16.p0v16f16(<16 x half> %val, <16 x half>* %addr, i32 4, <16 x i1>%mask)
+  ret void
+}
+
+define <16 x half> @maskloadu16f16(<16 x half>* %addr, <16 x half> %val, <16 x i1> %mask) {
+; X64-LABEL: maskloadu16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %xmm1, %xmm1
+; X64-NEXT:    vpmovb2m %xmm1, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskloadu16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %xmm1, %xmm1
+; X86-NEXT:    vpmovb2m %xmm1, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1}
+; X86-NEXT:    retl
+  %res = call <16 x half> @llvm.masked.load.v16f16.p0v16f16(<16 x half>* %addr, i32 4, <16 x i1> %mask, <16 x half> %val)
+  ret <16 x half> %res
+}
+
+define <16 x half> @maskuloadu16f16(<16 x half>* %addr, <16 x i1> %mask) {
+; X64-LABEL: maskuloadu16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X64-NEXT:    vpmovb2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskuloadu16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X86-NEXT:    vpmovb2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <16 x half> @llvm.masked.load.v16f16.p0v16f16(<16 x half>* %addr, i32 4, <16 x i1> %mask, <16 x half> undef)
+  ret <16 x half> %res
+}
+
+define <16 x half> @maskzloadu16f16(<16 x half>* %addr, <16 x i1> %mask) {
+; X64-LABEL: maskzloadu16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X64-NEXT:    vpmovb2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %ymm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskzloadu16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $7, %xmm0, %xmm0
+; X86-NEXT:    vpmovb2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %ymm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <16 x half> @llvm.masked.load.v16f16.p0v16f16(<16 x half>* %addr, i32 4, <16 x i1> %mask, <16 x half> zeroinitializer)
+  ret <16 x half> %res
+}
+
+define <16 x half> @movrr16f16(<16 x half> %a, <16 x half> %b) {
+; CHECK-LABEL: movrr16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %ymm1, %ymm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  ret <16 x half> %b
+}
+
+define <16 x half> @movrrk16f16(<16 x half> %a, <16 x half> %b, i16 %msk) {
+; X64-LABEL: movrrk16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vpblendmw %ymm0, %ymm1, %ymm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrk16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vpblendmw %ymm0, %ymm1, %ymm0 {%k1}
+; X86-NEXT:    retl
+  %mask = bitcast i16 %msk to <16 x i1>
+  %res = select <16 x i1> %mask, <16 x half> %a, <16 x half> %b
+  ret <16 x half> %res
+}
+
+define <16 x half> @movrrkz16f16(<16 x half> %a, i16 %msk) {
+; X64-LABEL: movrrkz16f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrkz16f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; X86-NEXT:    retl
+  %mask = bitcast i16 %msk to <16 x i1>
+  %res = select <16 x i1> %mask, <16 x half> %a, <16 x half> zeroinitializer
+  ret <16 x half> %res
+}
+
+define <8 x half> @load8f16(<8 x half>* %a) {
+; X64-LABEL: load8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovaps (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: load8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovaps (%eax), %xmm0
+; X86-NEXT:    retl
+  %res = load <8 x half>, <8 x half>* %a
+  ret <8 x half> %res
+}
+
+define <8 x half> @load8f16mask(<8 x half>* %a, <8 x half> %b, i8 %c) {
+; X64-LABEL: load8f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load8f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i8 %c to <8 x i1>
+  %res0 = load <8 x half>, <8 x half>* %a
+  %res = select <8 x i1> %msk, <8 x half> %res0, <8 x half> %b
+  ret <8 x half> %res
+}
+
+define <8 x half> @load8f16maskz(<8 x half>* %a, i8 %c) {
+; X64-LABEL: load8f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: load8f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i8 %c to <8 x i1>
+  %res0 = load <8 x half>, <8 x half>* %a
+  %res = select <8 x i1> %msk, <8 x half> %res0, <8 x half> zeroinitializer
+  ret <8 x half> %res
+}
+
+define <8 x half> @loadu8f16(<8 x half>* %a) {
+; X64-LABEL: loadu8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovups (%rdi), %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovups (%eax), %xmm0
+; X86-NEXT:    retl
+  %res = load <8 x half>, <8 x half>* %a, align 8
+  ret <8 x half> %res
+}
+
+define <8 x half> @loadu8f16mask(<8 x half>* %a, <8 x half> %b, i8 %c) {
+; X64-LABEL: loadu8f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu8f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1}
+; X86-NEXT:    retl
+  %msk = bitcast i8 %c to <8 x i1>
+  %res0 = load <8 x half>, <8 x half>* %a, align 8
+  %res = select <8 x i1> %msk, <8 x half> %res0, <8 x half> %b
+  ret <8 x half> %res
+}
+
+define <8 x half> @loadu8f16maskz(<8 x half>* %a, i8 %c) {
+; X64-LABEL: loadu8f16maskz:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %esi, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: loadu8f16maskz:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %msk = bitcast i8 %c to <8 x i1>
+  %res0 = load <8 x half>, <8 x half>* %a, align 8
+  %res = select <8 x i1> %msk, <8 x half> %res0, <8 x half> zeroinitializer
+  ret <8 x half> %res
+}
+
+define void @store8f16(<8 x half> %a) {
+; X64-LABEL: store8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g8f16@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovaps %xmm0, (%rax)
+; X64-NEXT:    retq
+;
+; X86-LABEL: store8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovaps %xmm0, g8f16
+; X86-NEXT:    retl
+  store <8 x half> %a, <8 x half>* @g8f16
+  ret void
+}
+
+define void @storeu8f16(<8 x half> %a) {
+; X64-LABEL: storeu8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    movq g8f16u@GOTPCREL(%rip), %rax
+; X64-NEXT:    vmovups %xmm0, (%rax)
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovups %xmm0, g8f16u
+; X86-NEXT:    retl
+  store <8 x half> %a, <8 x half>* @g8f16u, align 8
+  ret void
+}
+
+declare void @llvm.masked.store.v8f16.p0v8f16(<8 x half>, <8 x half>*, i32, <8 x i1>)
+declare <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>*, i32,  <8 x i1>, <8 x half>)
+
+define void @storeu8f16mask(<8 x i1> %mask, <8 x half>* %addr, <8 x half> %val) {
+; X64-LABEL: storeu8f16mask:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X64-NEXT:    vpmovw2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 %xmm1, (%rdi) {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: storeu8f16mask:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X86-NEXT:    vpmovw2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 %xmm1, (%eax) {%k1}
+; X86-NEXT:    retl
+  call void @llvm.masked.store.v8f16.p0v8f16(<8 x half> %val, <8 x half>* %addr, i32 4, <8 x i1>%mask)
+  ret void
+}
+
+define <8 x half> @maskloadu8f16(<8 x half>* %addr, <8 x half> %val, <8 x i1> %mask) {
+; X64-LABEL: maskloadu8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $15, %xmm1, %xmm1
+; X64-NEXT:    vpmovw2m %xmm1, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskloadu8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $15, %xmm1, %xmm1
+; X86-NEXT:    vpmovw2m %xmm1, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1}
+; X86-NEXT:    retl
+  %res = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %addr, i32 4, <8 x i1> %mask, <8 x half> %val)
+  ret <8 x half> %res
+}
+
+define <8 x half> @maskuloadu8f16(<8 x half>* %addr, <8 x i1> %mask) {
+; X64-LABEL: maskuloadu8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X64-NEXT:    vpmovw2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskuloadu8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X86-NEXT:    vpmovw2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %addr, i32 4, <8 x i1> %mask, <8 x half> undef)
+  ret <8 x half> %res
+}
+
+define <8 x half> @maskzloadu8f16(<8 x half>* %addr, <8 x i1> %mask) {
+; X64-LABEL: maskzloadu8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X64-NEXT:    vpmovw2m %xmm0, %k1
+; X64-NEXT:    vmovdqu16 (%rdi), %xmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: maskzloadu8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    vpsllw $15, %xmm0, %xmm0
+; X86-NEXT:    vpmovw2m %xmm0, %k1
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovdqu16 (%eax), %xmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %res = call <8 x half> @llvm.masked.load.v8f16.p0v8f16(<8 x half>* %addr, i32 4, <8 x i1> %mask, <8 x half> zeroinitializer)
+  ret <8 x half> %res
+}
+
+define <8 x half> @movrr8f16(<8 x half> %a, <8 x half> %b) {
+; CHECK-LABEL: movrr8f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm1, %xmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+  ret <8 x half> %b
+}
+
+define <8 x half> @movrrk8f16(<8 x half> %a, <8 x half> %b, i8 %msk) {
+; X64-LABEL: movrrk8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vpblendmw %xmm0, %xmm1, %xmm0 {%k1}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrk8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vpblendmw %xmm0, %xmm1, %xmm0 {%k1}
+; X86-NEXT:    retl
+  %mask = bitcast i8 %msk to <8 x i1>
+  %res = select <8 x i1> %mask, <8 x half> %a, <8 x half> %b
+  ret <8 x half> %res
+}
+
+define <8 x half> @movrrkz8f16(<8 x half> %a, i8 %msk) {
+; X64-LABEL: movrrkz8f16:
+; X64:       # %bb.0:
+; X64-NEXT:    kmovd %edi, %k1
+; X64-NEXT:    vmovdqu16 %xmm0, %xmm0 {%k1} {z}
+; X64-NEXT:    retq
+;
+; X86-LABEL: movrrkz8f16:
+; X86:       # %bb.0:
+; X86-NEXT:    kmovb {{[0-9]+}}(%esp), %k1
+; X86-NEXT:    vmovdqu16 %xmm0, %xmm0 {%k1} {z}
+; X86-NEXT:    retl
+  %mask = bitcast i8 %msk to <8 x i1>
+  %res = select <8 x i1> %mask, <8 x half> %a, <8 x half> zeroinitializer
+  ret <8 x half> %res
+}
+
+define i16 @test_movw(half %x) {
+; X64-LABEL: test_movw:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %xmm0, %eax
+; X64-NEXT:    # kill: def $ax killed $ax killed $eax
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_movw:
+; X86:       # %bb.0:
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    retl
+  %res = bitcast half %x to i16
+  ret i16 %res
+}
+
+define half @test_movw2(i16 %x) {
+; X64-LABEL: test_movw2:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovw %edi, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_movw2:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    retl
+  %res = bitcast i16 %x to half
+  ret half %res
+}
+
+; sext avoids having a truncate in front of the bitcast input due to calling
+; convention or i16 op promotion.
+define half @test_movw3(i8 %x) {
+; X64-LABEL: test_movw3:
+; X64:       # %bb.0:
+; X64-NEXT:    movsbl %dil, %eax
+; X64-NEXT:    vmovw %eax, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: test_movw3:
+; X86:       # %bb.0:
+; X86-NEXT:    movsbl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovw %eax, %xmm0
+; X86-NEXT:    retl
+  %z = sext i8 %x to i16
+  %a = bitcast i16 %z to half
+  ret half %a
+}
+
+define half @extract_f16_0(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 0
+   ret half %res
+}
+
+define half @extract_f16_1(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpsrld $16, %xmm0, %xmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 1
+   ret half %res
+}
+
+define half @extract_f16_2(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovshdup {{.*#+}} xmm0 = xmm0[1,1,3,3]
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 2
+   ret half %res
+}
+
+define half @extract_f16_3(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpsrlq $48, %xmm0, %xmm0
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 3
+   ret half %res
+}
+
+define half @extract_f16_4(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 4
+   ret half %res
+}
+
+define half @extract_f16_5(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 5
+   ret half %res
+}
+
+define half @extract_f16_6(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 6
+   ret half %res
+}
+
+define half @extract_f16_7(<8 x half> %x) {
+; CHECK-LABEL: extract_f16_7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x half> %x, i32 7
+   ret half %res
+}
+
+define i16 @extract_i16_0(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovw %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 0
+   ret i16 %res
+}
+
+define i16 @extract_i16_1(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $1, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 1
+   ret i16 %res
+}
+
+define i16 @extract_i16_2(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $2, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 2
+   ret i16 %res
+}
+
+define i16 @extract_i16_3(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $3, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 3
+   ret i16 %res
+}
+
+define i16 @extract_i16_4(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $4, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 4
+   ret i16 %res
+}
+
+define i16 @extract_i16_5(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_5:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $5, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 5
+   ret i16 %res
+}
+
+define i16 @extract_i16_6(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_6:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $6, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 6
+   ret i16 %res
+}
+
+define i16 @extract_i16_7(<8 x i16> %x) {
+; CHECK-LABEL: extract_i16_7:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $7, %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 7
+   ret i16 %res
+}
+
+define void @extract_store_f16_0(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_0:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_0:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 0
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_1(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_1:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsrld $16, %xmm0, %xmm0
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_1:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpsrld $16, %xmm0, %xmm0
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 1
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_2(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_2:
+; X64:       # %bb.0:
+; X64-NEXT:    vmovshdup {{.*#+}} xmm0 = xmm0[1,1,3,3]
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_2:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vmovshdup {{.*#+}} xmm0 = xmm0[1,1,3,3]
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 2
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_3(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_3:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsrlq $48, %xmm0, %xmm0
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_3:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpsrlq $48, %xmm0, %xmm0
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 3
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_4(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_4:
+; X64:       # %bb.0:
+; X64-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_4:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpermilpd {{.*#+}} xmm0 = xmm0[1,0]
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 4
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_5(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_5:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_5:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 5
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_6(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_6:
+; X64:       # %bb.0:
+; X64-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_6:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[3,3,3,3]
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 6
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_f16_7(<8 x half> %x, half* %y) {
+; X64-LABEL: extract_store_f16_7:
+; X64:       # %bb.0:
+; X64-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; X64-NEXT:    vmovsh %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_f16_7:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; X86-NEXT:    vmovsh %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x half> %x, i32 7
+   store half %res, half* %y
+   ret void
+}
+
+define void @extract_store_i16_0(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_0:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $0, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_0:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $0, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 0
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_1(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_1:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $1, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_1:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $1, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 1
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_2(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_2:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $2, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_2:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $2, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 2
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_3(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_3:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $3, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_3:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $3, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 3
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_4(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_4:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $4, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_4:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $4, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 4
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_5(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_5:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $5, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_5:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $5, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 5
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_6(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_6:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $6, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_6:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $6, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 6
+   store i16 %res, i16* %y
+   ret void
+}
+
+define void @extract_store_i16_7(<8 x i16> %x, i16* %y) {
+; X64-LABEL: extract_store_i16_7:
+; X64:       # %bb.0:
+; X64-NEXT:    vpextrw $7, %xmm0, (%rdi)
+; X64-NEXT:    retq
+;
+; X86-LABEL: extract_store_i16_7:
+; X86:       # %bb.0:
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    vpextrw $7, %xmm0, (%eax)
+; X86-NEXT:    retl
+   %res = extractelement <8 x i16> %x, i32 7
+   store i16 %res, i16* %y
+   ret void
+}
+
+define i32 @extract_zext_i16_0(<8 x i16> %x) {
+; CHECK-LABEL: extract_zext_i16_0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $0, %xmm0, %eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 0
+   %res2 = zext i16 %res to i32
+   ret i32 %res2
+}
+
+define i32 @extract_zext_i16_1(<8 x i16> %x) {
+; CHECK-LABEL: extract_zext_i16_1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpextrw $1, %xmm0, %eax
+; CHECK-NEXT:    ret{{[l|q]}}
+   %res = extractelement <8 x i16> %x, i32 1
+   %res2 = zext i16 %res to i32
+   ret i32 %res2
+}
+
+define <8 x half> @build_vector_xxxxuuuu(half %a0, half %a1, half %a2, half %a3) {
+; X64-LABEL: build_vector_xxxxuuuu:
+; X64:       # %bb.0:
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X64-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm2[0],zero,zero
+; X64-NEXT:    retq
+;
+; X86-LABEL: build_vector_xxxxuuuu:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm1[0],zero,zero
+; X86-NEXT:    retl
+  %a = insertelement <8 x half> undef, half %a0, i32 0
+  %b = insertelement <8 x half> %a, half %a1, i32 1
+  %c = insertelement <8 x half> %b, half %a2, i32 2
+  %d = insertelement <8 x half> %c, half %a3, i32 3
+  ret <8 x half> %d
+}
+
+define <8 x half> @build_vector_uuuuxxxx(half %a0, half %a1, half %a2, half %a3) {
+; X64-LABEL: build_vector_uuuuxxxx:
+; X64:       # %bb.0:
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X64-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
+; X64-NEXT:    vpbroadcastq %xmm0, %xmm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: build_vector_uuuuxxxx:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X86-NEXT:    vpbroadcastq %xmm0, %xmm0
+; X86-NEXT:    retl
+  %a = insertelement <8 x half> undef, half %a0, i32 4
+  %b = insertelement <8 x half> %a, half %a1, i32 5
+  %c = insertelement <8 x half> %b, half %a2, i32 6
+  %d = insertelement <8 x half> %c, half %a3, i32 7
+  ret <8 x half> %d
+}
+
+define <8 x half> @build_vector_xxxxxxxx(half %a0, half %a1, half %a2, half %a3, half %a4, half %a5, half %a6, half %a7) {
+; X64-LABEL: build_vector_xxxxxxxx:
+; X64:       # %bb.0:
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm6 = xmm6[0],xmm7[0],xmm6[1],xmm7[1],xmm6[2],xmm7[2],xmm6[3],xmm7[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm4 = xmm4[0],xmm5[0],xmm4[1],xmm5[1],xmm4[2],xmm5[2],xmm4[3],xmm5[3]
+; X64-NEXT:    vpunpckldq {{.*#+}} xmm4 = xmm4[0],xmm6[0],xmm4[1],xmm6[1]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X64-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
+; X64-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm4[0]
+; X64-NEXT:    retq
+;
+; X86-LABEL: build_vector_xxxxxxxx:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm3
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X86-NEXT:    vpunpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
+; X86-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; X86-NEXT:    retl
+  %a = insertelement <8 x half> undef, half %a0, i32 0
+  %b = insertelement <8 x half> %a, half %a1, i32 1
+  %c = insertelement <8 x half> %b, half %a2, i32 2
+  %d = insertelement <8 x half> %c, half %a3, i32 3
+  %e = insertelement <8 x half> %d, half %a4, i32 4
+  %f = insertelement <8 x half> %e, half %a5, i32 5
+  %g = insertelement <8 x half> %f, half %a6, i32 6
+  %h = insertelement <8 x half> %g, half %a7, i32 7
+  ret <8 x half> %h
+}
+
+define <16 x half> @build_vector_xxxxuuuuuuuuxxxx(half %a0, half %a1, half %a2, half %a3, half %a4, half %a5, half %a6, half %a7) {
+; X64-LABEL: build_vector_xxxxuuuuuuuuxxxx:
+; X64:       # %bb.0:
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X64-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0],xmm2[0],zero,zero
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm6[0],xmm7[0],xmm6[1],xmm7[1],xmm6[2],xmm7[2],xmm6[3],xmm7[3]
+; X64-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm4[0],xmm5[0],xmm4[1],xmm5[1],xmm4[2],xmm5[2],xmm4[3],xmm5[3]
+; X64-NEXT:    vpunpckldq {{.*#+}} xmm1 = xmm2[0],xmm1[0],xmm2[1],xmm1[1]
+; X64-NEXT:    vpbroadcastq %xmm1, %xmm1
+; X64-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; X64-NEXT:    retq
+;
+; X86-LABEL: build_vector_xxxxuuuuuuuuxxxx:
+; X86:       # %bb.0:
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm0
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm1
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3]
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm2
+; X86-NEXT:    vmovsh {{[0-9]+}}(%esp), %xmm3
+; X86-NEXT:    vpunpcklwd {{.*#+}} xmm2 = xmm2[0],xmm3[0],xmm2[1],xmm3[1],xmm2[2],xmm3[2],xmm2[3],xmm3[3]
+; X86-NEXT:    vinsertps {{.*#+}} xmm1 = xmm1[0],xmm2[0],zero,zero
+; X86-NEXT:    vpbroadcastq %xmm0, %xmm0
+; X86-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
+; X86-NEXT:    retl
+  %a = insertelement <16 x half> undef, half %a0, i32 0
+  %b = insertelement <16 x half> %a, half %a1, i32 1
+  %c = insertelement <16 x half> %b, half %a2, i32 2
+  %d = insertelement <16 x half> %c, half %a3, i32 3
+  %e = insertelement <16 x half> %d, half %a4, i32 12
+  %f = insertelement <16 x half> %e, half %a5, i32 13
+  %g = insertelement <16 x half> %f, half %a6, i32 14
+  %h = insertelement <16 x half> %g, half %a7, i32 15
+  ret <16 x half> %h
+}
+

llvm/test/CodeGen/X86/avx512fp16-mscatter.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unkown-unkown -mattr=+avx512bw -mattr=+avx512vl -mattr=+avx512fp16 | FileCheck %s
+
+define void @test_mscatter_v16f16(half* %base, <16 x i32> %index, <16 x half> %val)
+; CHECK-LABEL: test_mscatter_v16f16:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vpbroadcastq %rdi, %zmm3
+; CHECK-NEXT:    vextracti64x4 $1, %zmm0, %ymm2
+; CHECK-NEXT:    vpmovsxdq %ymm2, %zmm2
+; CHECK-NEXT:    vpaddq %zmm3, %zmm2, %zmm4
+; CHECK-NEXT:    vpaddq %zmm4, %zmm2, %zmm2
+; CHECK-NEXT:    vpmovsxdq %ymm0, %zmm0
+; CHECK-NEXT:    vpaddq %zmm3, %zmm0, %zmm3
+; CHECK-NEXT:    vpaddq %zmm3, %zmm0, %zmm0
+; CHECK-NEXT:    vmovq %xmm0, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpsrld $16, %xmm1, %xmm3
+; CHECK-NEXT:    vpextrq $1, %xmm0, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vmovshdup {{.*#+}} xmm3 = xmm1[1,1,3,3]
+; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm4
+; CHECK-NEXT:    vmovq %xmm4, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vpsrlq $48, %xmm1, %xmm3
+; CHECK-NEXT:    vpextrq $1, %xmm4, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vpermilpd {{.*#+}} xmm3 = xmm1[1,0]
+; CHECK-NEXT:    vextracti32x4 $2, %zmm0, %xmm4
+; CHECK-NEXT:    vmovq %xmm4, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm3 = xmm1[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    vpextrq $1, %xmm4, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vpermilps {{.*#+}} xmm3 = xmm1[3,3,3,3]
+; CHECK-NEXT:    vextracti32x4 $3, %zmm0, %xmm0
+; CHECK-NEXT:    vmovq %xmm0, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm3 = xmm1[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    vpextrq $1, %xmm0, %rax
+; CHECK-NEXT:    vmovsh %xmm3, (%rax)
+; CHECK-NEXT:    vextractf128 $1, %ymm1, %xmm0
+; CHECK-NEXT:    vmovq %xmm2, %rax
+; CHECK-NEXT:    vmovsh %xmm0, (%rax)
+; CHECK-NEXT:    vpsrld $16, %xmm0, %xmm1
+; CHECK-NEXT:    vpextrq $1, %xmm2, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vmovshdup {{.*#+}} xmm1 = xmm0[1,1,3,3]
+; CHECK-NEXT:    vextracti128 $1, %ymm2, %xmm3
+; CHECK-NEXT:    vmovq %xmm3, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpsrlq $48, %xmm0, %xmm1
+; CHECK-NEXT:    vpextrq $1, %xmm3, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpermilpd {{.*#+}} xmm1 = xmm0[1,0]
+; CHECK-NEXT:    vextracti32x4 $2, %zmm2, %xmm3
+; CHECK-NEXT:    vmovq %xmm3, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm1 = xmm0[10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    vpextrq $1, %xmm3, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpermilps {{.*#+}} xmm1 = xmm0[3,3,3,3]
+; CHECK-NEXT:    vextracti32x4 $3, %zmm2, %xmm2
+; CHECK-NEXT:    vmovq %xmm2, %rax
+; CHECK-NEXT:    vmovsh %xmm1, (%rax)
+; CHECK-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; CHECK-NEXT:    vpextrq $1, %xmm2, %rax
+; CHECK-NEXT:    vmovsh %xmm0, (%rax)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+{
+  %gep = getelementptr half, half* %base, <16 x i32> %index
+  call void @llvm.masked.scatter.v16f16.v16p0f16(<16 x half> %val, <16 x half*> %gep, i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
+  ret void
+}
+declare void @llvm.masked.scatter.v16f16.v16p0f16(<16 x half> , <16 x half*> , i32 , <16 x i1>)

llvm/test/CodeGen/X86/avx512fp16-subv-broadcast-fp16.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=skx -mattr=+avx512fp16 | FileCheck %s
+
+define dso_local void @test_v8f16_v32f16(<8 x half>* %x_addr, <32 x half>* %y_addr) {
+; CHECK-LABEL: test_v8f16_v32f16:
+; CHECK:       ## %bb.0: ## %entry
+; CHECK-NEXT:    vbroadcastf32x4 {{.*#+}} zmm0 = mem[0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3]
+; CHECK-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+entry:
+  %0 = load <8 x half>, <8 x half>* %x_addr, align 16
+  %shuffle.i58 = shufflevector <8 x half> %0, <8 x half> %0, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  store <32 x half> %shuffle.i58, <32 x half>* %y_addr, align 64
+  ret void
+}
+
+define dso_local void @test_v8f16_v16f16(<8 x half>* %x_addr, <16 x half>* %y_addr) {
+; CHECK-LABEL: test_v8f16_v16f16:
+; CHECK:       ## %bb.0: ## %entry
+; CHECK-NEXT:    vbroadcastf128 {{.*#+}} ymm0 = mem[0,1,0,1]
+; CHECK-NEXT:    vmovdqa %ymm0, (%rsi)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+entry:
+  %0 = load <8 x half>, <8 x half>* %x_addr, align 16
+  %shuffle.i58 = shufflevector <8 x half> %0, <8 x half> %0, <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  store <16 x half> %shuffle.i58, <16 x half>* %y_addr, align 64
+  ret void
+}
+
+define dso_local void @test_v16f16_v32f16(<16 x half>* %x_addr, <32 x half>* %y_addr) {
+; CHECK-LABEL: test_v16f16_v32f16:
+; CHECK:       ## %bb.0: ## %entry
+; CHECK-NEXT:    vbroadcastf64x4 {{.*#+}} zmm0 = mem[0,1,2,3,0,1,2,3]
+; CHECK-NEXT:    vmovdqa64 %zmm0, (%rsi)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+entry:
+  %0 = load <16 x half>, <16 x half>* %x_addr, align 16
+  %shuffle.i58 = shufflevector <16 x half> %0, <16 x half> %0, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+  store <32 x half> %shuffle.i58, <32 x half>* %y_addr, align 64
+  ret void
+}

llvm/test/CodeGen/X86/avx512fp16vl-intrinsics.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512vl -mattr=+avx512fp16 | FileCheck %s
+
+define signext i16 @test_mm_cvtsi128_si16(<2 x i64> %A) local_unnamed_addr #0 {
+; CHECK-LABEL: test_mm_cvtsi128_si16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vmovw %xmm0, %eax
+; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
+; CHECK-NEXT:    retq
+entry:
+  %0 = bitcast <2 x i64> %A to <8 x i16>
+  %vecext.i = extractelement <8 x i16> %0, i32 0
+  ret i16 %vecext.i
+}
+
+define <2 x i64> @test_mm_cvtsi16_si128(i16 signext %A) local_unnamed_addr #0 {
+; CHECK-LABEL: test_mm_cvtsi16_si128:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    vmovw %edi, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %vecinit7.i = insertelement <8 x i16> <i16 undef, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0>, i16 %A, i32 0
+  %0 = bitcast <8 x i16> %vecinit7.i to <2 x i64>
+  ret <2 x i64> %0
+}

llvm/test/CodeGen/X86/fp128-cast-strict.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-linux-android -mattr=+sse | FileCheck %s --check-prefixes=X64,X64-SSE
 ; RUN: llc < %s -mtriple=x86_64-linux-gnu -mattr=+sse | FileCheck %s --check-prefixes=X64,X64-SSE
 ; RUN: llc < %s -mtriple=x86_64-linux-android -mattr=+avx | FileCheck %s --check-prefixes=X64,X64-AVX
 ; RUN: llc < %s -mtriple=x86_64-linux-gnu -mattr=+avx | FileCheck %s --check-prefixes=X64,X64-AVX
-; RUN: llc < %s -mtriple=x86_64-linux-android -mattr=+avx512f | FileCheck %s --check-prefixes=X64,X64-AVX
-; RUN: llc < %s -mtriple=x86_64-linux-gnu -mattr=+avx512f | FileCheck %s --check-prefixes=X64,X64-AVX
+; RUN: llc < %s -mtriple=x86_64-linux-android -mattr=+avx512fp16 | FileCheck %s --check-prefixes=X64,X64-AVX,X64-AVX512
+; RUN: llc < %s -mtriple=x86_64-linux-gnu -mattr=+avx512fp16 | FileCheck %s --check-prefixes=X64,X64-AVX,X64-AVX512
 ; RUN: llc < %s -mtriple=i686-linux-gnu -mattr=-sse | FileCheck %s --check-prefixes=X86
 
 ; Check soft floating point conversion function calls.
 
+@vf16 = common dso_local global half 0.000000e+00, align 2
 @vf32 = common dso_local global float 0.000000e+00, align 4
 @vf64 = common dso_local global double 0.000000e+00, align 8
 @vf80 = common dso_local global x86_fp80 0xK00000000000000000000, align 8
 @vf128 = common dso_local global fp128 0xL00000000000000000000000000000000, align 16
 
+define dso_local void @TestFPExtF16_F128() nounwind strictfp {
+; X64-SSE-LABEL: TestFPExtF16_F128:
+; X64-SSE:       # %bb.0: # %entry
+; X64-SSE-NEXT:    pushq %rax
+; X64-SSE-NEXT:    movzwl vf16(%rip), %edi
+; X64-SSE-NEXT:    callq __gnu_h2f_ieee@PLT
+; X64-SSE-NEXT:    callq __extendsftf2@PLT
+; X64-SSE-NEXT:    movaps %xmm0, vf128(%rip)
+; X64-SSE-NEXT:    popq %rax
+; X64-SSE-NEXT:    retq
+;
+; X64-AVX512-LABEL: TestFPExtF16_F128:
+; X64-AVX512:       # %bb.0: # %entry
+; X64-AVX512-NEXT:    pushq %rax
+; X64-AVX512-NEXT:    vmovsh vf16(%rip), %xmm0
+; X64-AVX512-NEXT:    callq __extendhftf2@PLT
+; X64-AVX512-NEXT:    vmovaps %xmm0, vf128(%rip)
+; X64-AVX512-NEXT:    popq %rax
+; X64-AVX512-NEXT:    retq
+;
+; X86-LABEL: TestFPExtF16_F128:
+; X86:       # %bb.0: # %entry
+; X86-NEXT:    pushl %esi
+; X86-NEXT:    subl $24, %esp
+; X86-NEXT:    movzwl vf16, %eax
+; X86-NEXT:    movl %eax, (%esp)
+; X86-NEXT:    calll __gnu_h2f_ieee
+; X86-NEXT:    fstps {{[0-9]+}}(%esp)
+; X86-NEXT:    wait
+; X86-NEXT:    leal {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl %eax, (%esp)
+; X86-NEXT:    calll __extendsftf2
+; X86-NEXT:    subl $4, %esp
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; X86-NEXT:    movl %esi, vf128+12
+; X86-NEXT:    movl %edx, vf128+8
+; X86-NEXT:    movl %ecx, vf128+4
+; X86-NEXT:    movl %eax, vf128
+; X86-NEXT:    addl $24, %esp
+; X86-NEXT:    popl %esi
+; X86-NEXT:    retl
+entry:
+  %0 = load half, half* @vf16, align 2
+  %conv = call fp128 @llvm.experimental.constrained.fpext.f128.f16(half %0, metadata !"fpexcept.strict") #0
+  store fp128 %conv, fp128* @vf128, align 16
+  ret void
+}
+
 define dso_local void @TestFPExtF32_F128() nounwind strictfp {
 ; X64-SSE-LABEL: TestFPExtF32_F128:
 ; X64-SSE:       # %bb.0: # %entry
 ; X64-SSE-NEXT:    pushq %rax
 ; X64-SSE-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X64-SSE-NEXT:    callq __extendsftf2@PLT
 ; X64-SSE-NEXT:    movaps %xmm0, vf128(%rip)
 ; X64-SSE-NEXT:    popq %rax

 ; X86-NEXT:    retl
 entry:
   %0 = load x86_fp80, x86_fp80* @vf80, align 8
   %conv = call fp128 @llvm.experimental.constrained.fpext.f128.f80(x86_fp80 %0, metadata !"fpexcept.strict") #0
   store fp128 %conv, fp128* @vf128, align 16
   ret void
 }
 
+define dso_local void @TestFPTruncF128_F16() nounwind strictfp {
+; X64-SSE-LABEL: TestFPTruncF128_F16:
+; X64-SSE:       # %bb.0: # %entry
+; X64-SSE-NEXT:    pushq %rax
+; X64-SSE-NEXT:    movaps vf128(%rip), %xmm0
+; X64-SSE-NEXT:    callq __trunctfhf2@PLT
+; X64-SSE-NEXT:    movw %ax, vf16(%rip)
+; X64-SSE-NEXT:    popq %rax
+; X64-SSE-NEXT:    retq
+;
+; X64-AVX512-LABEL: TestFPTruncF128_F16:
+; X64-AVX512:       # %bb.0: # %entry
+; X64-AVX512-NEXT:    pushq %rax
+; X64-AVX512-NEXT:    vmovaps vf128(%rip), %xmm0
+; X64-AVX512-NEXT:    callq __trunctfhf2@PLT
+; X64-AVX512-NEXT:    vmovsh %xmm0, vf16(%rip)
+; X64-AVX512-NEXT:    popq %rax
+; X64-AVX512-NEXT:    retq
+;
+; X86-LABEL: TestFPTruncF128_F16:
+; X86:       # %bb.0: # %entry
+; X86-NEXT:    subl $12, %esp
+; X86-NEXT:    pushl vf128+12
+; X86-NEXT:    pushl vf128+8
+; X86-NEXT:    pushl vf128+4
+; X86-NEXT:    pushl vf128
+; X86-NEXT:    calll __trunctfhf2
+; X86-NEXT:    addl $16, %esp
+; X86-NEXT:    movw %ax, vf16
+; X86-NEXT:    addl $12, %esp
+; X86-NEXT:    retl
+entry:
+  %0 = load fp128, fp128* @vf128, align 16
+  %conv = call half @llvm.experimental.constrained.fptrunc.f16.f128(fp128 %0, metadata !"round.dynamic", metadata !"fpexcept.strict") #0
+  store half %conv, half* @vf16, align 2
+  ret void
+}
+
 define dso_local void @TestFPTruncF128_F32() nounwind strictfp {
 ; X64-SSE-LABEL: TestFPTruncF128_F32:
 ; X64-SSE:       # %bb.0: # %entry
 ; X64-SSE-NEXT:    pushq %rax
 ; X64-SSE-NEXT:    movaps vf128(%rip), %xmm0
 ; X64-SSE-NEXT:    callq __trunctfsf2@PLT
 ; X64-SSE-NEXT:    movss %xmm0, vf32(%rip)
 ; X64-SSE-NEXT:    popq %rax

 ; X86-NEXT:    retl $4
 entry:
   %conv = call fp128 @llvm.experimental.constrained.uitofp.f128.i128(i128 %x, metadata !"round.dynamic", metadata !"fpexcept.strict") #0
   ret fp128 %conv
 }
 
 attributes #0 = { strictfp }
 
+declare half @llvm.experimental.constrained.fptrunc.f16.f128(fp128, metadata, metadata)
 declare float @llvm.experimental.constrained.fptrunc.f32.f128(fp128, metadata, metadata)
 declare double @llvm.experimental.constrained.fptrunc.f64.f128(fp128, metadata, metadata)
 declare x86_fp80 @llvm.experimental.constrained.fptrunc.f80.f128(fp128, metadata, metadata)
+declare fp128 @llvm.experimental.constrained.fpext.f128.f16(half, metadata)
 declare fp128 @llvm.experimental.constrained.fpext.f128.f32(float, metadata)
 declare fp128 @llvm.experimental.constrained.fpext.f128.f64(double, metadata)
 declare fp128 @llvm.experimental.constrained.fpext.f128.f80(x86_fp80, metadata)
 declare i8 @llvm.experimental.constrained.fptosi.i8.f128(fp128, metadata)
 declare i16 @llvm.experimental.constrained.fptosi.i16.f128(fp128, metadata)
 declare i32 @llvm.experimental.constrained.fptosi.i32.f128(fp128, metadata)
 declare i64 @llvm.experimental.constrained.fptosi.i64.f128(fp128, metadata)
 declare i128 @llvm.experimental.constrained.fptosi.i128.f128(fp128, metadata)

llvm/test/CodeGen/X86/pseudo_cmov_lower-fp16.ll

+; RUN: llc < %s -mtriple=i386-linux-gnu -mattr=+avx512fp16 -mattr=+avx512vl -o - | FileCheck %s
+
+; This test checks that only a single jne gets generated in the final code
+; for lowering the CMOV pseudos that get created for this IR.
+define dso_local <32 x half> @foo3(<32 x half> %a, <32 x half> %b, i1 zeroext %sign) local_unnamed_addr #0 {
+; CHECK-LABEL: foo3:
+; CHECK: jne
+; CHECK-NOT: jne
+entry:
+  %spec.select = select i1 %sign, <32 x half> %a, <32 x half> %b
+  ret <32 x half> %spec.select
+}
+
+; This test checks that only a single jne gets generated in the final code
+; for lowering the CMOV pseudos that get created for this IR.
+define dso_local <16 x half> @foo4(<16 x half> %a, <16 x half> %b, i1 zeroext %sign) local_unnamed_addr #0 {
+; CHECK-LABEL: foo4:
+; CHECK: jne
+; CHECK-NOT: jne
+entry:
+  %spec.select = select i1 %sign, <16 x half> %a, <16 x half> %b
+  ret <16 x half> %spec.select
+}
+
+; This test checks that only a single jne gets generated in the final code
+; for lowering the CMOV pseudos that get created for this IR.
+define dso_local <8 x half> @foo5(<8 x half> %a, <8 x half> %b, i1 zeroext %sign) local_unnamed_addr #0 {
+; CHECK-LABEL: foo5:
+; CHECK: jne
+; CHECK-NOT: jne
+entry:
+  %spec.select = select i1 %sign, <8 x half> %a, <8 x half> %b
+  ret <8 x half> %spec.select
+}

llvm/test/CodeGen/X86/statepoint-invoke-ra-enter-at-end.mir

body:             |
   ; CHECK: bb.7.bb33:
   ; CHECK:   successors: %bb.7(0x80000000)
   ; CHECK:   [[INC64r:%[0-9]+]]:gr64_with_sub_8bit = nuw nsw INC64r [[INC64r]], implicit-def dead $eflags
   ; CHECK:   [[MOV64rm2:%[0-9]+]]:gr64 = MOV64rm undef %59:gr64, 1, $noreg, 0, $noreg :: (load unordered 8 from `i8 addrspace(1)* addrspace(1)* undef`, addrspace 1)
   ; CHECK:   [[NOT64r2:%[0-9]+]]:gr64 = NOT64r [[NOT64r2]]
   ; CHECK:   CMP64rr [[NOT64r2]], [[COPY6]], implicit-def $eflags
   ; CHECK:   undef %102.sub_32bit:gr64_with_sub_8bit = MOV32ri 0
   ; CHECK:   [[CMOV64rr:%[0-9]+]]:gr64 = CMOV64rr [[CMOV64rr]], %102, 4, implicit killed $eflags
-  ; CHECK:   INLINEASM &"lock btsq $0,($1)", 1 /* sideeffect attdialect */, 4325385 /* reguse:GR64 */, %102, 4325385 /* reguse:GR64 */, undef %56:gr64, 12 /* clobber */, implicit-def dead early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def dead early-clobber $eflags
+  ; CHECK:   INLINEASM &"lock btsq $0,($1)", 1 /* sideeffect attdialect */, 4390921 /* reguse:{{.*}} */, %102, 4390921 /* reguse:{{.*}} */, undef %56:gr64, 12 /* clobber */, implicit-def dead early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def dead early-clobber $eflags
   ; CHECK:   LCMPXCHG32 undef %67:gr64, 1, $noreg, 0, $noreg, [[COPY5]], implicit-def dead $eax, implicit-def dead $eflags, implicit undef $eax :: (load store acquire monotonic 4 on `i32 addrspace(1)* undef`, addrspace 1)
   ; CHECK:   ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
   ; CHECK:   $rdi = COPY [[COPY4]]
   ; CHECK:   CALL64pcrel32 target-flags(x86-plt) @wobble, csr_64, implicit $rsp, implicit $ssp, implicit $rdi, implicit-def $rsp, implicit-def $ssp
   ; CHECK:   ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
   ; CHECK:   [[MOV64rm2:%[0-9]+]]:gr64 = MOV64rm $rip, 1, $noreg, target-flags(x86-gotpcrel) @global, $noreg :: (load 8 from got)
   ; CHECK:   [[MOV64rm3:%[0-9]+]]:gr64 = MOV64rm [[MOV64rm2]], 1, $noreg, 0, $noreg :: (dereferenceable load unordered 8 from @global)
   ; CHECK:   [[NOT64r2:%[0-9]+]]:gr64 = NOT64r [[NOT64r2]]
body:             |
   bb.7.bb33:
     successors: %bb.7(0x80000000)
 
     %81:gr64_with_sub_8bit = nuw nsw INC64r %81, implicit-def dead $eflags
     %63:gr64 = MOV64rm undef %59:gr64, 1, $noreg, 0, $noreg :: (load unordered 8 from `i8 addrspace(1)* addrspace(1)* undef`, addrspace 1)
     %63:gr64 = NOT64r %63
     CMP64rr %63, %31, implicit-def $eflags
     %63:gr64 = CMOV64rr %63, %53, 4, implicit killed $eflags
-    INLINEASM &"lock btsq $0,($1)", 1 /* sideeffect attdialect */, 4325385 /* reguse:GR64 */, %53, 4325385 /* reguse:GR64 */, undef %56:gr64, 12 /* clobber */, implicit-def dead early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def dead early-clobber $eflags
+    INLINEASM &"lock btsq $0,($1)", 1 /* sideeffect attdialect */, 4390921 /* reguse:GR64 */, %53, 4390921 /* reguse:GR64 */, undef %56:gr64, 12 /* clobber */, implicit-def dead early-clobber $df, 12 /* clobber */, implicit-def early-clobber $fpsw, 12 /* clobber */, implicit-def dead early-clobber $eflags
     LCMPXCHG32 undef %67:gr64, 1, $noreg, 0, $noreg, %65, implicit-def dead $eax, implicit-def dead $eflags, implicit undef $eax :: (load store acquire monotonic 4 on `i32 addrspace(1)* undef`, addrspace 1)
     ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
     $rdi = COPY %64
     CALL64pcrel32 target-flags(x86-plt) @wobble, csr_64, implicit $rsp, implicit $ssp, implicit killed $rdi, implicit-def $rsp, implicit-def $ssp
     ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
     %71:gr64 = MOV64rm %46, 1, $noreg, 0, $noreg :: (dereferenceable load unordered 8 from @global)
     %71:gr64 = NOT64r %71
     ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp

llvm/test/CodeGen/X86/vector-reduce-fmax-nnan.ll

; AVX512-NEXT:    retq
   %1 = call nnan double @llvm.vector.reduce.fmax.v16f64(<16 x double> %a0)
   ret double %1
 }
 
 define half @test_v2f16(<2 x half> %a0) nounwind {
 ; SSE-LABEL: test_v2f16:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    pushq %rbp
-; SSE-NEXT:    pushq %r14
 ; SSE-NEXT:    pushq %rbx
-; SSE-NEXT:    subq $16, %rsp
+; SSE-NEXT:    pushq %rax
 ; SSE-NEXT:    movl %esi, %ebx
-; SSE-NEXT:    movl %edi, %r14d
-; SSE-NEXT:    movzwl %bx, %ebp
-; SSE-NEXT:    movl %ebp, %edi
+; SSE-NEXT:    movl %edi, %ebp
+; SSE-NEXT:    movzwl %bx, %edi
 ; SSE-NEXT:    callq __gnu_h2f_ieee@PLT
 ; SSE-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; SSE-NEXT:    movzwl %r14w, %edi
+; SSE-NEXT:    movzwl %bp, %edi
 ; SSE-NEXT:    callq __gnu_h2f_ieee@PLT
+; SSE-NEXT:    xorl %eax, %eax
 ; SSE-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
-; SSE-NEXT:    movw %bp, {{[0-9]+}}(%rsp)
-; SSE-NEXT:    cmoval %r14d, %ebx
-; SSE-NEXT:    movw %bx, (%rsp)
-; SSE-NEXT:    movl (%rsp), %eax
-; SSE-NEXT:    # kill: def $ax killed $ax killed $eax
-; SSE-NEXT:    addq $16, %rsp
+; SSE-NEXT:    seta %al
+; SSE-NEXT:    negl %eax
+; SSE-NEXT:    testb $1, %al
+; SSE-NEXT:    cmovnel %ebp, %ebx
+; SSE-NEXT:    movl %ebx, %eax
+; SSE-NEXT:    addq $8, %rsp
 ; SSE-NEXT:    popq %rbx
-; SSE-NEXT:    popq %r14
 ; SSE-NEXT:    popq %rbp
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: test_v2f16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
-; AVX-NEXT:    pushq %r14
 ; AVX-NEXT:    pushq %rbx
-; AVX-NEXT:    subq $16, %rsp
+; AVX-NEXT:    pushq %rax
 ; AVX-NEXT:    movl %esi, %ebx
-; AVX-NEXT:    movl %edi, %r14d
-; AVX-NEXT:    movzwl %bx, %ebp
-; AVX-NEXT:    movl %ebp, %edi
+; AVX-NEXT:    movl %edi, %ebp
+; AVX-NEXT:    movzwl %bx, %edi
 ; AVX-NEXT:    callq __gnu_h2f_ieee@PLT
 ; AVX-NEXT:    vmovss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; AVX-NEXT:    movzwl %r14w, %edi
+; AVX-NEXT:    movzwl %bp, %edi
 ; AVX-NEXT:    callq __gnu_h2f_ieee@PLT
+; AVX-NEXT:    xorl %eax, %eax
 ; AVX-NEXT:    vucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
-; AVX-NEXT:    movw %bp, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    cmoval %r14d, %ebx
-; AVX-NEXT:    movw %bx, (%rsp)
-; AVX-NEXT:    movl (%rsp), %eax
-; AVX-NEXT:    # kill: def $ax killed $ax killed $eax
-; AVX-NEXT:    addq $16, %rsp
+; AVX-NEXT:    seta %al
+; AVX-NEXT:    negl %eax
+; AVX-NEXT:    testb $1, %al
+; AVX-NEXT:    cmovnel %ebp, %ebx
+; AVX-NEXT:    movl %ebx, %eax
+; AVX-NEXT:    addq $8, %rsp
 ; AVX-NEXT:    popq %rbx
-; AVX-NEXT:    popq %r14
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v2f16:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    movzwl %si, %eax
-; AVX512-NEXT:    vmovd %eax, %xmm0
+; AVX512-NEXT:    movl %esi, %eax
+; AVX512-NEXT:    movzwl %ax, %ecx
+; AVX512-NEXT:    vmovd %ecx, %xmm0
 ; AVX512-NEXT:    vcvtph2ps %xmm0, %xmm0
 ; AVX512-NEXT:    movzwl %di, %ecx
 ; AVX512-NEXT:    vmovd %ecx, %xmm1
 ; AVX512-NEXT:    vcvtph2ps %xmm1, %xmm1
 ; AVX512-NEXT:    vucomiss %xmm0, %xmm1
-; AVX512-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; AVX512-NEXT:    cmoval %edi, %esi
-; AVX512-NEXT:    movw %si, -{{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movl -{{[0-9]+}}(%rsp), %eax
+; AVX512-NEXT:    cmoval %edi, %eax
 ; AVX512-NEXT:    # kill: def $ax killed $ax killed $eax
 ; AVX512-NEXT:    retq
   %1 = call nnan half @llvm.vector.reduce.fmax.v2f16(<2 x half> %a0)
   ret half %1
 }
 declare float @llvm.vector.reduce.fmax.v1f32(<1 x float>)
 declare float @llvm.vector.reduce.fmax.v2f32(<2 x float>)
 declare float @llvm.vector.reduce.fmax.v4f32(<4 x float>)

llvm/test/CodeGen/X86/vector-reduce-fmin-nnan.ll

; AVX512-NEXT:    retq
   %1 = call nnan double @llvm.vector.reduce.fmin.v16f64(<16 x double> %a0)
   ret double %1
 }
 
 define half @test_v2f16(<2 x half> %a0) nounwind {
 ; SSE-LABEL: test_v2f16:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    pushq %rbp
-; SSE-NEXT:    pushq %r14
 ; SSE-NEXT:    pushq %rbx
-; SSE-NEXT:    subq $16, %rsp
+; SSE-NEXT:    pushq %rax
 ; SSE-NEXT:    movl %esi, %ebx
-; SSE-NEXT:    movl %edi, %r14d
-; SSE-NEXT:    movzwl %bx, %ebp
-; SSE-NEXT:    movl %ebp, %edi
+; SSE-NEXT:    movl %edi, %ebp
+; SSE-NEXT:    movzwl %bx, %edi
 ; SSE-NEXT:    callq __gnu_h2f_ieee@PLT
 ; SSE-NEXT:    movss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; SSE-NEXT:    movzwl %r14w, %edi
+; SSE-NEXT:    movzwl %bp, %edi
 ; SSE-NEXT:    callq __gnu_h2f_ieee@PLT
 ; SSE-NEXT:    ucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
-; SSE-NEXT:    movw %bp, {{[0-9]+}}(%rsp)
-; SSE-NEXT:    cmovbl %r14d, %ebx
-; SSE-NEXT:    movw %bx, (%rsp)
-; SSE-NEXT:    movl (%rsp), %eax
-; SSE-NEXT:    # kill: def $ax killed $ax killed $eax
-; SSE-NEXT:    addq $16, %rsp
+; SSE-NEXT:    cmovbl %ebp, %ebx
+; SSE-NEXT:    movl %ebx, %eax
+; SSE-NEXT:    addq $8, %rsp
 ; SSE-NEXT:    popq %rbx
-; SSE-NEXT:    popq %r14
 ; SSE-NEXT:    popq %rbp
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: test_v2f16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    pushq %rbp
-; AVX-NEXT:    pushq %r14
 ; AVX-NEXT:    pushq %rbx
-; AVX-NEXT:    subq $16, %rsp
+; AVX-NEXT:    pushq %rax
 ; AVX-NEXT:    movl %esi, %ebx
-; AVX-NEXT:    movl %edi, %r14d
-; AVX-NEXT:    movzwl %bx, %ebp
-; AVX-NEXT:    movl %ebp, %edi
+; AVX-NEXT:    movl %edi, %ebp
+; AVX-NEXT:    movzwl %bx, %edi
 ; AVX-NEXT:    callq __gnu_h2f_ieee@PLT
 ; AVX-NEXT:    vmovss %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
-; AVX-NEXT:    movzwl %r14w, %edi
+; AVX-NEXT:    movzwl %bp, %edi
 ; AVX-NEXT:    callq __gnu_h2f_ieee@PLT
 ; AVX-NEXT:    vucomiss {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 4-byte Folded Reload
-; AVX-NEXT:    movw %bp, {{[0-9]+}}(%rsp)
-; AVX-NEXT:    cmovbl %r14d, %ebx
-; AVX-NEXT:    movw %bx, (%rsp)
-; AVX-NEXT:    movl (%rsp), %eax
-; AVX-NEXT:    # kill: def $ax killed $ax killed $eax
-; AVX-NEXT:    addq $16, %rsp
+; AVX-NEXT:    cmovbl %ebp, %ebx
+; AVX-NEXT:    movl %ebx, %eax
+; AVX-NEXT:    addq $8, %rsp
 ; AVX-NEXT:    popq %rbx
-; AVX-NEXT:    popq %r14
 ; AVX-NEXT:    popq %rbp
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: test_v2f16:
 ; AVX512:       # %bb.0:
-; AVX512-NEXT:    movzwl %si, %eax
-; AVX512-NEXT:    vmovd %eax, %xmm0
+; AVX512-NEXT:    movl %esi, %eax
+; AVX512-NEXT:    movzwl %ax, %ecx
+; AVX512-NEXT:    vmovd %ecx, %xmm0
 ; AVX512-NEXT:    vcvtph2ps %xmm0, %xmm0
 ; AVX512-NEXT:    movzwl %di, %ecx
 ; AVX512-NEXT:    vmovd %ecx, %xmm1
 ; AVX512-NEXT:    vcvtph2ps %xmm1, %xmm1
 ; AVX512-NEXT:    vucomiss %xmm0, %xmm1
-; AVX512-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
-; AVX512-NEXT:    cmovbl %edi, %esi
-; AVX512-NEXT:    movw %si, -{{[0-9]+}}(%rsp)
-; AVX512-NEXT:    movl -{{[0-9]+}}(%rsp), %eax
+; AVX512-NEXT:    cmovbl %edi, %eax
 ; AVX512-NEXT:    # kill: def $ax killed $ax killed $eax
 ; AVX512-NEXT:    retq
   %1 = call nnan half @llvm.vector.reduce.fmin.v2f16(<2 x half> %a0)
   ret half %1
 }
 
 declare float @llvm.vector.reduce.fmin.v1f32(<1 x float>)
 declare float @llvm.vector.reduce.fmin.v2f32(<2 x float>)

llvm/test/MC/Disassembler/X86/avx512fp16.txt

+# RUN: llvm-mc --disassemble %s -triple=x86_64 | FileCheck %s --check-prefixes=ATT
+# RUN: llvm-mc --disassemble %s -triple=x86_64 -x86-asm-syntax=intel --output-asm-variant=1 | FileCheck %s --check-prefixes=INTEL
+
+# ATT:   vmovsh %xmm28, %xmm29, %xmm30
+# INTEL: vmovsh xmm30, xmm29, xmm28
+0x62,0x05,0x16,0x00,0x10,0xf4
+
+# ATT:   vmovsh 268435456(%rbp,%r14,8), %xmm30 {%k7}
+# INTEL: vmovsh xmm30 {k7}, word ptr [rbp + 8*r14 + 268435456]
+0x62,0x25,0x7e,0x0f,0x10,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# ATT:   vmovsh (%r9), %xmm30
+# INTEL: vmovsh xmm30, word ptr [r9]
+0x62,0x45,0x7e,0x08,0x10,0x31
+
+# ATT:   vmovsh 254(%rcx), %xmm30
+# INTEL: vmovsh xmm30, word ptr [rcx + 254]
+0x62,0x65,0x7e,0x08,0x10,0x71,0x7f
+
+# ATT:   vmovsh -256(%rdx), %xmm30 {%k7} {z}
+# INTEL: vmovsh xmm30 {k7} {z}, word ptr [rdx - 256]
+0x62,0x65,0x7e,0x8f,0x10,0x72,0x80
+
+# ATT:   vmovsh %xmm30, 268435456(%rbp,%r14,8) {%k7}
+# INTEL: vmovsh word ptr [rbp + 8*r14 + 268435456] {k7}, xmm30
+0x62,0x25,0x7e,0x0f,0x11,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# ATT:   vmovsh %xmm30, (%r9)
+# INTEL: vmovsh word ptr [r9], xmm30
+0x62,0x45,0x7e,0x08,0x11,0x31
+
+# ATT:   vmovsh %xmm30, 254(%rcx)
+# INTEL: vmovsh word ptr [rcx + 254], xmm30
+0x62,0x65,0x7e,0x08,0x11,0x71,0x7f
+
+# ATT:   vmovsh %xmm30, -256(%rdx) {%k7}
+# INTEL: vmovsh word ptr [rdx - 256] {k7}, xmm30
+0x62,0x65,0x7e,0x0f,0x11,0x72,0x80
+
+# ATT:   vmovw %r12d, %xmm30
+# INTEL: vmovw xmm30, r12d
+0x62,0x45,0x7d,0x08,0x6e,0xf4
+
+# ATT:   vmovw %xmm30, %r12d
+# INTEL: vmovw r12d, xmm30
+0x62,0x45,0x7d,0x08,0x7e,0xf4
+
+# ATT:   vmovw 268435456(%rbp,%r14,8), %xmm30
+# INTEL: vmovw xmm30, word ptr [rbp + 8*r14 + 268435456]
+0x62,0x25,0x7d,0x08,0x6e,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# ATT:   vmovw (%r9), %xmm30
+# INTEL: vmovw xmm30, word ptr [r9]
+0x62,0x45,0x7d,0x08,0x6e,0x31
+
+# ATT:   vmovw 254(%rcx), %xmm30
+# INTEL: vmovw xmm30, word ptr [rcx + 254]
+0x62,0x65,0x7d,0x08,0x6e,0x71,0x7f
+
+# ATT:   vmovw -256(%rdx), %xmm30
+# INTEL: vmovw xmm30, word ptr [rdx - 256]
+0x62,0x65,0x7d,0x08,0x6e,0x72,0x80
+
+# ATT:   vmovw %xmm30, 268435456(%rbp,%r14,8)
+# INTEL: vmovw word ptr [rbp + 8*r14 + 268435456], xmm30
+0x62,0x25,0x7d,0x08,0x7e,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# ATT:   vmovw %xmm30, (%r9)
+# INTEL: vmovw word ptr [r9], xmm30
+0x62,0x45,0x7d,0x08,0x7e,0x31
+
+# ATT:   vmovw %xmm30, 254(%rcx)
+# INTEL: vmovw word ptr [rcx + 254], xmm30
+0x62,0x65,0x7d,0x08,0x7e,0x71,0x7f
+
+# ATT:   vmovw %xmm30, -256(%rdx)
+# INTEL: vmovw word ptr [rdx - 256], xmm30
+0x62,0x65,0x7d,0x08,0x7e,0x72,0x80

llvm/test/MC/X86/avx512fp16.s

+// RUN: llvm-mc -triple x86_64-unknown-unknown --show-encoding < %s  | FileCheck %s
+
+// CHECK: vmovsh %xmm28, %xmm29, %xmm30
+// CHECK: encoding: [0x62,0x05,0x16,0x00,0x10,0xf4]
+          vmovsh %xmm28, %xmm29, %xmm30
+
+// CHECK: vmovsh 268435456(%rbp,%r14,8), %xmm30 {%k7}
+// CHECK: encoding: [0x62,0x25,0x7e,0x0f,0x10,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          vmovsh 268435456(%rbp,%r14,8), %xmm30 {%k7}
+
+// CHECK: vmovsh (%r9), %xmm30
+// CHECK: encoding: [0x62,0x45,0x7e,0x08,0x10,0x31]
+          vmovsh (%r9), %xmm30
+
+// CHECK: vmovsh 254(%rcx), %xmm30
+// CHECK: encoding: [0x62,0x65,0x7e,0x08,0x10,0x71,0x7f]
+          vmovsh 254(%rcx), %xmm30
+
+// CHECK: vmovsh -256(%rdx), %xmm30 {%k7} {z}
+// CHECK: encoding: [0x62,0x65,0x7e,0x8f,0x10,0x72,0x80]
+          vmovsh -256(%rdx), %xmm30 {%k7} {z}
+
+// CHECK: vmovsh %xmm30, 268435456(%rbp,%r14,8) {%k7}
+// CHECK: encoding: [0x62,0x25,0x7e,0x0f,0x11,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          vmovsh %xmm30, 268435456(%rbp,%r14,8) {%k7}
+
+// CHECK: vmovsh %xmm30, (%r9)
+// CHECK: encoding: [0x62,0x45,0x7e,0x08,0x11,0x31]
+          vmovsh %xmm30, (%r9)
+
+// CHECK: vmovsh %xmm30, 254(%rcx)
+// CHECK: encoding: [0x62,0x65,0x7e,0x08,0x11,0x71,0x7f]
+          vmovsh %xmm30, 254(%rcx)
+
+// CHECK: vmovsh %xmm30, -256(%rdx) {%k7}
+// CHECK: encoding: [0x62,0x65,0x7e,0x0f,0x11,0x72,0x80]
+          vmovsh %xmm30, -256(%rdx) {%k7}
+
+// CHECK: vmovw %r12d, %xmm30
+// CHECK: encoding: [0x62,0x45,0x7d,0x08,0x6e,0xf4]
+          vmovw %r12d, %xmm30
+
+// CHECK: vmovw %xmm30, %r12d
+// CHECK: encoding: [0x62,0x45,0x7d,0x08,0x7e,0xf4]
+          vmovw %xmm30, %r12d
+
+// CHECK: vmovw 268435456(%rbp,%r14,8), %xmm30
+// CHECK: encoding: [0x62,0x25,0x7d,0x08,0x6e,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          vmovw 268435456(%rbp,%r14,8), %xmm30
+
+// CHECK: vmovw (%r9), %xmm30
+// CHECK: encoding: [0x62,0x45,0x7d,0x08,0x6e,0x31]
+          vmovw (%r9), %xmm30
+
+// CHECK: vmovw 254(%rcx), %xmm30
+// CHECK: encoding: [0x62,0x65,0x7d,0x08,0x6e,0x71,0x7f]
+          vmovw 254(%rcx), %xmm30
+
+// CHECK: vmovw -256(%rdx), %xmm30
+// CHECK: encoding: [0x62,0x65,0x7d,0x08,0x6e,0x72,0x80]
+          vmovw -256(%rdx), %xmm30
+
+// CHECK: vmovw %xmm30, 268435456(%rbp,%r14,8)
+// CHECK: encoding: [0x62,0x25,0x7d,0x08,0x7e,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          vmovw %xmm30, 268435456(%rbp,%r14,8)
+
+// CHECK: vmovw %xmm30, (%r9)
+// CHECK: encoding: [0x62,0x45,0x7d,0x08,0x7e,0x31]
+          vmovw %xmm30, (%r9)
+
+// CHECK: vmovw %xmm30, 254(%rcx)
+// CHECK: encoding: [0x62,0x65,0x7d,0x08,0x7e,0x71,0x7f]
+          vmovw %xmm30, 254(%rcx)
+
+// CHECK: vmovw %xmm30, -256(%rdx)
+// CHECK: encoding: [0x62,0x65,0x7d,0x08,0x7e,0x72,0x80]
+          vmovw %xmm30, -256(%rdx)

llvm/test/MC/X86/intel-syntax-avx512fp16.s

+// RUN: llvm-mc -triple i686-unknown-unknown -x86-asm-syntax=intel -output-asm-variant=1 --show-encoding %s | FileCheck %s
+
+// CHECK: vmovsh xmm6, xmm5, xmm4
+// CHECK: encoding: [0x62,0xf5,0x56,0x08,0x10,0xf4]
+          vmovsh xmm6, xmm5, xmm4
+
+// CHECK: vmovsh xmm6 {k7}, word ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0x62,0xf5,0x7e,0x0f,0x10,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          vmovsh xmm6 {k7}, word ptr [esp + 8*esi + 268435456]
+
+// CHECK: vmovsh xmm6, word ptr [ecx]
+// CHECK: encoding: [0x62,0xf5,0x7e,0x08,0x10,0x31]
+          vmovsh xmm6, word ptr [ecx]
+
+// CHECK: vmovsh xmm6, word ptr [ecx + 254]
+// CHECK: encoding: [0x62,0xf5,0x7e,0x08,0x10,0x71,0x7f]
+          vmovsh xmm6, word ptr [ecx + 254]
+
+// CHECK: vmovsh xmm6 {k7} {z}, word ptr [edx - 256]
+// CHECK: encoding: [0x62,0xf5,0x7e,0x8f,0x10,0x72,0x80]
+          vmovsh xmm6 {k7} {z}, word ptr [edx - 256]
+
+// CHECK: vmovsh word ptr [esp + 8*esi + 268435456] {k7}, xmm6
+// CHECK: encoding: [0x62,0xf5,0x7e,0x0f,0x11,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          vmovsh word ptr [esp + 8*esi + 268435456] {k7}, xmm6
+
+// CHECK: vmovsh word ptr [ecx], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7e,0x08,0x11,0x31]
+          vmovsh word ptr [ecx], xmm6
+
+// CHECK: vmovsh word ptr [ecx + 254], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7e,0x08,0x11,0x71,0x7f]
+          vmovsh word ptr [ecx + 254], xmm6
+
+// CHECK: vmovsh word ptr [edx - 256] {k7}, xmm6
+// CHECK: encoding: [0x62,0xf5,0x7e,0x0f,0x11,0x72,0x80]
+          vmovsh word ptr [edx - 256] {k7}, xmm6
+
+// CHECK: vmovw xmm6, edx
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x6e,0xf2]
+          vmovw xmm6, edx
+
+// CHECK: vmovw edx, xmm6
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x7e,0xf2]
+          vmovw edx, xmm6
+
+// CHECK: vmovw xmm6, word ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x6e,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          vmovw xmm6, word ptr [esp + 8*esi + 268435456]
+
+// CHECK: vmovw xmm6, word ptr [ecx]
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x6e,0x31]
+          vmovw xmm6, word ptr [ecx]
+
+// CHECK: vmovw xmm6, word ptr [ecx + 254]
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x6e,0x71,0x7f]
+          vmovw xmm6, word ptr [ecx + 254]
+
+// CHECK: vmovw xmm6, word ptr [edx - 256]
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x6e,0x72,0x80]
+          vmovw xmm6, word ptr [edx - 256]
+
+// CHECK: vmovw word ptr [esp + 8*esi + 268435456], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x7e,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          vmovw word ptr [esp + 8*esi + 268435456], xmm6
+
+// CHECK: vmovw word ptr [ecx], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x7e,0x31]
+          vmovw word ptr [ecx], xmm6
+
+// CHECK: vmovw word ptr [ecx + 254], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x7e,0x71,0x7f]
+          vmovw word ptr [ecx + 254], xmm6
+
+// CHECK: vmovw word ptr [edx - 256], xmm6
+// CHECK: encoding: [0x62,0xf5,0x7d,0x08,0x7e,0x72,0x80]
+          vmovw word ptr [edx - 256], xmm6

llvm/test/MachineVerifier/test_copy_physregs_x86.mir

 
 ---
 name:            test_invalid_copies
 tracksRegLiveness: true
 body:             |
   bb.0:
     liveins: $xmm0, $xmm1, $xmm2, $xmm3
 
-    ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
+    ; FP16 reg is sub_reg of xmm
     %0:_(s16) = COPY $xmm0
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     %1:_(<4 x s16>) = COPY $xmm1
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     %2:_(s256) = COPY $xmm2
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     %3:_(<8 x s32>) = COPY $xmm3
 
-    ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
+    ; FP16 reg is sub_reg of xmm
     $xmm0 = COPY %0
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     $xmm1 = COPY %1
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     $xmm2 = COPY %2
 
     ; CHECK: *** Bad machine code: Copy Instruction is illegal with mismatching sizes ***
     $xmm3 = COPY %3
 ...

llvm/utils/TableGen/X86DisassemblerTables.h

 //===- X86DisassemblerTables.h - Disassembler tables ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
private:
   /// [0] one-byte opcodes
   /// [1] two-byte opcodes of the form 0f __
   /// [2] three-byte opcodes of the form 0f 38 __
   /// [3] three-byte opcodes of the form 0f 3a __
   /// [4] XOP8 map opcode
   /// [5] XOP9 map opcode
   /// [6] XOPA map opcode
   /// [7] 3dnow map opcode
-  std::unique_ptr<ContextDecision> Tables[8];
+  /// [8] fixed length MAP5 opcode
+  /// [9] fixed length MAP6 opcode
+  std::unique_ptr<ContextDecision> Tables[10];
 
   // Table of ModRM encodings.
   typedef std::map<std::vector<unsigned>, unsigned> ModRMMapTy;
   mutable ModRMMapTy ModRMTable;
 
   /// The instruction information table
   std::vector<InstructionSpecifier> InstructionSpecifiers;
 

llvm/utils/TableGen/X86DisassemblerTables.cpp

 //===- X86DisassemblerTables.cpp - Disassembler tables ----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
void DisassemblerTables::emitContextDecisions(raw_ostream &o1, raw_ostream &o2,
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[0], ONEBYTE_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[1], TWOBYTE_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[2], THREEBYTE38_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[3], THREEBYTE3A_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[4], XOP8_MAP_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[5], XOP9_MAP_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[6], XOPA_MAP_STR);
   emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[7], THREEDNOW_MAP_STR);
+  emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[8], MAP5_STR);
+  emitContextDecision(o1, o2, i1, i2, ModRMTableNum, *Tables[9], MAP6_STR);
 }
 
 void DisassemblerTables::emit(raw_ostream &o) const {
   unsigned i1 = 0;
   unsigned i2 = 0;
 
   std::string s1;
   std::string s2;

llvm/utils/TableGen/X86RecognizableInstr.h

 //===- X86RecognizableInstr.h - Disassembler instruction spec ----*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
enum {
     MRM0X = 56, MRM1X = 57, MRM2X = 58, MRM3X = 59,
     MRM4X = 60, MRM5X = 61, MRM6X = 62, MRM7X = 63,
 #define MAP(from, to) MRM_##from = to,
     X86_INSTR_MRM_MAPPING
 #undef MAP
   };
 
   enum {
-    OB = 0, TB = 1, T8 = 2, TA = 3, XOP8 = 4, XOP9 = 5, XOPA = 6, ThreeDNow = 7
+    OB = 0, TB = 1, T8 = 2, TA = 3, XOP8 = 4, XOP9 = 5, XOPA = 6, ThreeDNow = 7,
+    T_MAP5 = 8, T_MAP6 = 9
   };
 
   enum {
     PD = 1, XS = 2, XD = 3, PS = 4
   };
 
   enum {
     VEX = 1, XOP = 2, EVEX = 3

llvm/utils/TableGen/X86RecognizableInstr.cpp

 //===- X86RecognizableInstr.cpp - Disassembler instruction spec --*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
#define MAP(from, to)                     \
   case X86Local::OB:        opcodeType = ONEBYTE;       break;
   case X86Local::TB:        opcodeType = TWOBYTE;       break;
   case X86Local::T8:        opcodeType = THREEBYTE_38;  break;
   case X86Local::TA:        opcodeType = THREEBYTE_3A;  break;
   case X86Local::XOP8:      opcodeType = XOP8_MAP;      break;
   case X86Local::XOP9:      opcodeType = XOP9_MAP;      break;
   case X86Local::XOPA:      opcodeType = XOPA_MAP;      break;
   case X86Local::ThreeDNow: opcodeType = THREEDNOW_MAP; break;
+  case X86Local::T_MAP5:    opcodeType = MAP5;          break;
+  case X86Local::T_MAP6:    opcodeType = MAP6;          break;
   }
 
   std::unique_ptr<ModRMFilter> filter;
   switch (Form) {
   default: llvm_unreachable("Invalid form!");
   case X86Local::Pseudo: llvm_unreachable("Pseudo should not be emitted!");
   case X86Local::RawFrm:
   case X86Local::AddRegFrm:
OperandType RecognizableInstr::typeFromString(const std::string &s,
   TYPE("f128mem",             TYPE_M)
   TYPE("f256mem",             TYPE_M)
   TYPE("f512mem",             TYPE_M)
   TYPE("FR128",               TYPE_XMM)
   TYPE("FR64",                TYPE_XMM)
   TYPE("FR64X",               TYPE_XMM)
   TYPE("f64mem",              TYPE_M)
   TYPE("sdmem",               TYPE_M)
+  TYPE("FR16X",               TYPE_XMM)
   TYPE("FR32",                TYPE_XMM)
   TYPE("FR32X",               TYPE_XMM)
   TYPE("f32mem",              TYPE_M)
+  TYPE("f16mem",              TYPE_M)
   TYPE("ssmem",               TYPE_M)
+  TYPE("shmem",               TYPE_M)
   TYPE("RST",                 TYPE_ST)
   TYPE("RSTi",                TYPE_ST)
   TYPE("i128mem",             TYPE_M)
   TYPE("i256mem",             TYPE_M)
   TYPE("i512mem",             TYPE_M)
   TYPE("i64i32imm_brtarget",  TYPE_REL)
   TYPE("i16imm_brtarget",     TYPE_REL)
   TYPE("i32imm_brtarget",     TYPE_REL)
RecognizableInstr::immediateEncodingFromString(const std::string &s,
   ENCODING("i64u8imm",        ENCODING_IB)
   // This is not a typo.  Instructions like BLENDVPD put
   // register IDs in 8-bit immediates nowadays.
   ENCODING("FR32",            ENCODING_IB)
   ENCODING("FR64",            ENCODING_IB)
   ENCODING("FR128",           ENCODING_IB)
   ENCODING("VR128",           ENCODING_IB)
   ENCODING("VR256",           ENCODING_IB)
+  ENCODING("FR16X",           ENCODING_IB)
   ENCODING("FR32X",           ENCODING_IB)
   ENCODING("FR64X",           ENCODING_IB)
   ENCODING("VR128X",          ENCODING_IB)
   ENCODING("VR256X",          ENCODING_IB)
   ENCODING("VR512",           ENCODING_IB)
   ENCODING("TILE",            ENCODING_IB)
   errs() << "Unhandled immediate encoding " << s << "\n";
   llvm_unreachable("Unhandled immediate encoding");
RecognizableInstr::rmRegisterEncodingFromString(const std::string &s,
   ENCODING("GR8",             ENCODING_RM)
   ENCODING("VR128",           ENCODING_RM)
   ENCODING("VR128X",          ENCODING_RM)
   ENCODING("FR128",           ENCODING_RM)
   ENCODING("FR64",            ENCODING_RM)
   ENCODING("FR32",            ENCODING_RM)
   ENCODING("FR64X",           ENCODING_RM)
   ENCODING("FR32X",           ENCODING_RM)
+  ENCODING("FR16X",           ENCODING_RM)
   ENCODING("VR64",            ENCODING_RM)
   ENCODING("VR256",           ENCODING_RM)
   ENCODING("VR256X",          ENCODING_RM)
   ENCODING("VR512",           ENCODING_RM)
   ENCODING("VK1",             ENCODING_RM)
   ENCODING("VK2",             ENCODING_RM)
   ENCODING("VK4",             ENCODING_RM)
   ENCODING("VK8",             ENCODING_RM)
RecognizableInstr::roRegisterEncodingFromString(const std::string &s,
   ENCODING("SEGMENT_REG",     ENCODING_REG)
   ENCODING("DEBUG_REG",       ENCODING_REG)
   ENCODING("CONTROL_REG",     ENCODING_REG)
   ENCODING("VR256",           ENCODING_REG)
   ENCODING("VR256X",          ENCODING_REG)
   ENCODING("VR128X",          ENCODING_REG)
   ENCODING("FR64X",           ENCODING_REG)
   ENCODING("FR32X",           ENCODING_REG)
+  ENCODING("FR16X",           ENCODING_REG)
   ENCODING("VR512",           ENCODING_REG)
   ENCODING("VK1",             ENCODING_REG)
   ENCODING("VK2",             ENCODING_REG)
   ENCODING("VK4",             ENCODING_REG)
   ENCODING("VK8",             ENCODING_REG)
   ENCODING("VK16",            ENCODING_REG)
   ENCODING("VK32",            ENCODING_REG)
   ENCODING("VK64",            ENCODING_REG)
RecognizableInstr::vvvvRegisterEncodingFromString(const std::string &s,
                                                   uint8_t OpSize) {
   ENCODING("GR32",            ENCODING_VVVV)
   ENCODING("GR64",            ENCODING_VVVV)
   ENCODING("FR32",            ENCODING_VVVV)
   ENCODING("FR128",           ENCODING_VVVV)
   ENCODING("FR64",            ENCODING_VVVV)
   ENCODING("VR128",           ENCODING_VVVV)
   ENCODING("VR256",           ENCODING_VVVV)
+  ENCODING("FR16X",           ENCODING_VVVV)
   ENCODING("FR32X",           ENCODING_VVVV)
   ENCODING("FR64X",           ENCODING_VVVV)
   ENCODING("VR128X",          ENCODING_VVVV)
   ENCODING("VR256X",          ENCODING_VVVV)
   ENCODING("VR512",           ENCODING_VVVV)
   ENCODING("VK1",             ENCODING_VVVV)
   ENCODING("VK2",             ENCODING_VVVV)
   ENCODING("VK4",             ENCODING_VVVV)

 
 OperandEncoding
 RecognizableInstr::memoryEncodingFromString(const std::string &s,
                                             uint8_t OpSize) {
   ENCODING("i16mem",          ENCODING_RM)
   ENCODING("i32mem",          ENCODING_RM)
   ENCODING("i64mem",          ENCODING_RM)
   ENCODING("i8mem",           ENCODING_RM)
+  ENCODING("shmem",           ENCODING_RM)
   ENCODING("ssmem",           ENCODING_RM)
   ENCODING("sdmem",           ENCODING_RM)
   ENCODING("f128mem",         ENCODING_RM)
   ENCODING("f256mem",         ENCODING_RM)
   ENCODING("f512mem",         ENCODING_RM)
   ENCODING("f64mem",          ENCODING_RM)
   ENCODING("f32mem",          ENCODING_RM)
+  ENCODING("f16mem",          ENCODING_RM)
   ENCODING("i128mem",         ENCODING_RM)
   ENCODING("i256mem",         ENCODING_RM)
   ENCODING("i512mem",         ENCODING_RM)
   ENCODING("f80mem",          ENCODING_RM)
   ENCODING("lea64_32mem",     ENCODING_RM)
   ENCODING("lea64mem",        ENCODING_RM)
   ENCODING("anymem",          ENCODING_RM)
   ENCODING("opaquemem",       ENCODING_RM)