[X86]Support options -mno-gather -mno-scatter

[XinWang10]

Summary

Gather instructions could lead to security issues, details please refer to https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/gather-data-sampling.html.
This supported options -mno-gather and -mno-scatter, which could avoid generating gather/scatter instructions in backend except using intrinsics or inline asms.

[XinWang10]

• restore blank line

[wangpc]

clang/lib/Driver/ToolChains/Clang.cpp

7527 ↗

(On Diff #549268)

Put these in clang/lib/Driver/ToolChains/Arch/X86.cpp:getX86TargetFeatures? They are target-specific.

[XinWang10]

• mv driver implement to x86 specific code

[pengfei]

clang/include/clang/Driver/Options.td

903–906

Move under "// X86 feature flags"

clang/lib/Driver/ToolChains/Arch/X86.cpp

272–277

No need parentheses

llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll

10–11

Don't need these and is duplicated with RUN line triple.

[MaskRay]

clang/include/clang/Driver/Options.td

903–906

Also remove NoXarchOption

https://maskray.me/blog/2021-03-28-compiler-driver-and-cross-compilation#misc

If an option has the NoXarchOption flag, ClangDriver will emit an error if the option is used after -Xarch_* (originally for universal macOS binary, reused by offloading purposes -Xarch_host/etc). The error checking only applies to a small set of options (e.g. -o) and is not very useful for most options, but NoXarchOption was improperly named DriverOption (commit aabb0b11a3c1d8a6bb859db80400cffdcc9b336f) and lured some contributors to add NoXarchOption to options that should not have the flag.

[skan]

llvm/lib/Target/X86/X86.td

437

Does "Prefer no gather instructions" sound better?

I think these two should be put under "X86 Subtarget Tuning features"?

[XinWang10]

• resolve comments

[XinWang10]

llvm/lib/Target/X86/X86.td

437

I think the two options are to mitigate security issues. Could refer to link in summary.

[pengfei]

llvm/lib/Target/X86/X86.td

437

It depends on if the micro code was applied. We should assume user care of this option should have applied the micro code. So it's more like a performance turning rather than mitigation. And you cannot disable all gather/scatter instructions with these options.

[XinWang10]

llvm/lib/Target/X86/X86.td

437

Micro code applied? You mean we should keep eye on the byte code generated.
And what's the essential difference between performance turning and mitigation? Do mitigation for no-gather means we could not emit gather whenever but performance turning could exist some gather considering the performance?
Second, I think the intention for this option is to mitigate the security issue but not tune the performance.

[XinWang10]

• mov definition to tuning features

[XinWang10]

llvm/lib/Target/X86/X86.td

437

Talked offline and agreed to move to performance tuning.

[pengfei]

LGTM.

[MaskRay]

clang/test/Driver/x86-no-gather-no-scatter.cpp

4

Each RUN line makes testsuite execution slower and we should be careful adding more lines.

We don't generally additionally test %clangxx when %clang suffices. For non-linking, %clang is mostly identical to %clangxx.

17

No newline at end of file

[XinWang10]

• update fe test

[thakis]

clang/test/Driver/x86-no-gather-no-scatter.cpp

4

FYI, %clang_cl always must use -- before %s, else tests fail on macOS. I fixed this test in 547ee1c81fceaabcb.

clang/include/clang/Driver/Options.td

 def Xpreprocessor : Separate<["-"], "Xpreprocessor">, Group<Preprocessor_Group>,
   HelpText<"Pass <arg> to the preprocessor">, MetaVarName<"<arg>">;
 def X_Flag : Flag<["-"], "X">, Group<Link_Group>;
 // Used by some macOS projects. IgnoredGCCCompat is a misnomer since GCC doesn't allow it.
 def : Flag<["-"], "Xparser">, IgnoredGCCCompat;
 // FIXME -Xcompiler is misused by some ChromeOS packages. Remove it after a while.
 def : Flag<["-"], "Xcompiler">, IgnoredGCCCompat;
 def Z_Flag : Flag<["-"], "Z">, Group<Link_Group>;
 def all__load : Flag<["-"], "all_load">;
 def allowable__client : Separate<["-"], "allowable_client">;
 def ansi : Flag<["-", "--"], "ansi">, Group<CompileOnly_Group>;
 def arch__errors__fatal : Flag<["-"], "arch_errors_fatal">;
 def arch : Separate<["-"], "arch">, Flags<[NoXarchOption]>;
 def arch__only : Separate<["-"], "arch_only">;
 def autocomplete : Joined<["--"], "autocomplete=">;
 def bind__at__load : Flag<["-"], "bind_at_load">;
 def bundle__loader : Separate<["-"], "bundle_loader">;
 def bundle : Flag<["-"], "bundle">;
 def b : JoinedOrSeparate<["-"], "b">, Flags<[LinkerInput]>,
   HelpText<"Pass -b <arg> to the linker on AIX">, MetaVarName<"<arg>">,

 def mxsaves : Flag<["-"], "mxsaves">, Group<m_x86_Features_Group>;
 def mno_xsaves : Flag<["-"], "mno-xsaves">, Group<m_x86_Features_Group>;
 def mshstk : Flag<["-"], "mshstk">, Group<m_x86_Features_Group>;
 def mno_shstk : Flag<["-"], "mno-shstk">, Group<m_x86_Features_Group>;
 def mretpoline_external_thunk : Flag<["-"], "mretpoline-external-thunk">, Group<m_x86_Features_Group>;
 def mno_retpoline_external_thunk : Flag<["-"], "mno-retpoline-external-thunk">, Group<m_x86_Features_Group>;
 def mvzeroupper : Flag<["-"], "mvzeroupper">, Group<m_x86_Features_Group>;
 def mno_vzeroupper : Flag<["-"], "mno-vzeroupper">, Group<m_x86_Features_Group>;
+def mno_gather : Flag<["-"], "mno-gather">, Group<m_x86_Features_Group>,
+                 HelpText<"Disable generation of gather instructions in auto-vectorization(x86 only)">;
+def mno_scatter : Flag<["-"], "mno-scatter">, Group<m_x86_Features_Group>,
+                  HelpText<"Disable generation of scatter instructions in auto-vectorization(x86 only)">;
 } // let Flags = [TargetSpecific]
 
 // These are legacy user-facing driver-level option spellings. They are always
 // aliases for options that are spelled using the more common Unix / GNU flag
 // style of double-dash and equals-joined flags.
 def target_legacy_spelling : Separate<["-"], "target">,
                              Alias<target>,
                              Flags<[CoreOption]>;

 def _SLASH_tune : CLCompileJoined<"tune:">,
   HelpText<"Set CPU for optimization without affecting instruction set">,
   Alias<mtune_EQ>;
 def _SLASH_QIntel_jcc_erratum : CLFlag<"QIntel-jcc-erratum">,
   HelpText<"Align branches within 32-byte boundaries to mitigate the performance impact of the Intel JCC erratum.">,
   Alias<mbranches_within_32B_boundaries>;
 def _SLASH_arm64EC : CLFlag<"arm64EC">,
   HelpText<"Set build target to arm64ec">;
+def : CLFlag<"Qgather-">, Alias<mno_gather>,
+      HelpText<"Disable generation of gather instructions in auto-vectorization(x86 only)">;
+def : CLFlag<"Qscatter-">, Alias<mno_scatter>,
+      HelpText<"Disable generation of scatter instructions in auto-vectorization(x86 only)">;
 
 // Non-aliases:
 
 def _SLASH_arch : CLCompileJoined<"arch:">,
   HelpText<"Set architecture for code generation">;
 
 def _SLASH_M_Group : OptionGroup<"</M group>">, Group<cl_compile_Group>;
 def _SLASH_volatile_Group : OptionGroup<"</volatile group>">,

clang/lib/Driver/ToolChains/Arch/X86.cpp

if (Scope == "all") {
       Features.push_back("+harden-sls-ret");
     } else if (Scope == "indirect-jmp") {
       Features.push_back("+harden-sls-ijmp");
     } else if (Scope != "none") {
       D.Diag(diag::err_drv_unsupported_option_argument)
           << A->getSpelling() << Scope;
     }
   }
+
+  // -mno-gather, -mno-scatter support
+  if (Args.hasArg(options::OPT_mno_gather))
+    Features.push_back("+prefer-no-gather");
+  if (Args.hasArg(options::OPT_mno_scatter))
+    Features.push_back("+prefer-no-scatter");
 }

clang/test/Driver/x86-no-gather-no-scatter.cpp

+/// Tests -mno-gather and -mno-scatter
+// RUN: %clang -c -mno-gather -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOGATHER %s
+// RUN: %clangxx -c -mno-gather -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOGATHER %s
+// RUN: %clang_cl -c /Qgather- -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOGATHER %s
+// NOGATHER: "-target-feature" "+prefer-no-gather"
+
+// RUN: %clang -c -mno-scatter -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOSCATTER %s
+// RUN: %clangxx -c -mno-scatter -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOSCATTER %s
+// RUN: %clang_cl -c /Qscatter- -### %s 2>&1 \
+// RUN:  | FileCheck --check-prefix=NOSCATTER %s
+// NOSCATTER: "-target-feature" "+prefer-no-scatter"
 No newline at end of file

llvm/lib/Target/X86/X86.td

 
 def FeatureHardenSlsIJmp
     : SubtargetFeature<
           "harden-sls-ijmp", "HardenSlsIJmp", "true",
           "Harden against straight line speculation across indirect JMP instructions.">;
 
 //===----------------------------------------------------------------------===//
 // X86 Subtarget Tuning features
 //===----------------------------------------------------------------------===//
 def TuningPreferMovmskOverVTest : SubtargetFeature<"prefer-movmsk-over-vtest",
                                        "PreferMovmskOverVTest", "true",
                                        "Prefer movmsk over vtest instruction">;
 
 def TuningSlowSHLD : SubtargetFeature<"slow-shld", "IsSHLDSlow", "true",
                                        "SHLD instruction is slow">;
 
 def TuningSlowPMULLD : SubtargetFeature<"slow-pmulld", "IsPMULLDSlow", "true",

 // Gather is available since Haswell (AVX2 set). So technically, we can
 // generate Gathers on all AVX2 processors. But the overhead on HSW is high.
 // Skylake Client processor has faster Gathers than HSW and performance is
 // similar to Skylake Server (AVX-512).
 def TuningFastGather
     : SubtargetFeature<"fast-gather", "HasFastGather", "true",
                        "Indicates if gather is reasonably fast (this is true for Skylake client and all AVX-512 CPUs)">;
 
+def TuningPreferNoGather
+    : SubtargetFeature<"prefer-no-gather", "PreferGather", "false",
+                       "Prefer no gather instructions">;
+def TuningPreferNoScatter
+    : SubtargetFeature<"prefer-no-scatter", "PreferScatter", "false",
+                       "Prefer no scatter instructions">;
+
 def TuningPrefer128Bit
     : SubtargetFeature<"prefer-128-bit", "Prefer128Bit", "true",
                        "Prefer 128-bit AVX instructions">;
 
 def TuningPrefer256Bit
     : SubtargetFeature<"prefer-256-bit", "Prefer256Bit", "true",
                        "Prefer 256-bit AVX instructions">;
 

llvm/lib/Target/X86/X86TargetTransformInfo.h

public:
   bool isLegalMaskedStore(Type *DataType, Align Alignment);
   bool isLegalNTLoad(Type *DataType, Align Alignment);
   bool isLegalNTStore(Type *DataType, Align Alignment);
   bool isLegalBroadcastLoad(Type *ElementTy, ElementCount NumElements) const;
   bool forceScalarizeMaskedGather(VectorType *VTy, Align Alignment);
   bool forceScalarizeMaskedScatter(VectorType *VTy, Align Alignment) {
     return forceScalarizeMaskedGather(VTy, Alignment);
   }
+  bool isLegalMaskedGatherScatter(Type *DataType, Align Alignment);
   bool isLegalMaskedGather(Type *DataType, Align Alignment);
   bool isLegalMaskedScatter(Type *DataType, Align Alignment);
   bool isLegalMaskedExpandLoad(Type *DataType);
   bool isLegalMaskedCompressStore(Type *DataType);
   bool isLegalAltInstr(VectorType *VecTy, unsigned Opcode0, unsigned Opcode1,
                        const SmallBitVector &OpcodeMask) const;
   bool hasDivRemOp(Type *DataType, bool IsSigned);
   bool isExpensiveToSpeculativelyExecute(const Instruction *I);

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

bool X86TTIImpl::forceScalarizeMaskedGather(VectorType *VTy, Align Alignment) {
   // instructions. Right now we give the scalar cost of vector-4 for KNL. TODO:
   // Check, maybe the gather/scatter instruction is better in the VariableMask
   // case.
   unsigned NumElts = cast<FixedVectorType>(VTy)->getNumElements();
   return NumElts == 1 ||
          (ST->hasAVX512() && (NumElts == 2 || (NumElts == 4 && !ST->hasVLX())));
 }
 
-bool X86TTIImpl::isLegalMaskedGather(Type *DataTy, Align Alignment) {
-  if (!supportsGather())
-    return false;
+bool X86TTIImpl::isLegalMaskedGatherScatter(Type *DataTy, Align Alignment) {
   Type *ScalarTy = DataTy->getScalarType();
   if (ScalarTy->isPointerTy())
     return true;
 
   if (ScalarTy->isFloatTy() || ScalarTy->isDoubleTy())
     return true;
 
   if (!ScalarTy->isIntegerTy())
     return false;
 
   unsigned IntWidth = ScalarTy->getIntegerBitWidth();
   return IntWidth == 32 || IntWidth == 64;
 }
 
+bool X86TTIImpl::isLegalMaskedGather(Type *DataTy, Align Alignment) {
+  if (!supportsGather() || !ST->preferGather())
+    return false;
+  return isLegalMaskedGatherScatter(DataTy, Alignment);
+}
+
 bool X86TTIImpl::isLegalAltInstr(VectorType *VecTy, unsigned Opcode0,
                                  unsigned Opcode1,
                                  const SmallBitVector &OpcodeMask) const {
   // ADDSUBPS  4xf32 SSE3
   // VADDSUBPS 4xf32 AVX
   // VADDSUBPS 8xf32 AVX2
   // ADDSUBPD  2xf64 SSE3
   // VADDSUBPD 2xf64 AVX
if (ElemTy->isFloatTy())
     return ST->hasSSE3() && NumElements % 4 == 0;
   if (ElemTy->isDoubleTy())
     return ST->hasSSE3() && NumElements % 2 == 0;
   return false;
 }
 
 bool X86TTIImpl::isLegalMaskedScatter(Type *DataType, Align Alignment) {
   // AVX2 doesn't support scatter
-  if (!ST->hasAVX512())
+  if (!ST->hasAVX512() || !ST->preferScatter())
     return false;
-  return isLegalMaskedGather(DataType, Alignment);
+  return isLegalMaskedGatherScatter(DataType, Alignment);
 }
 
 bool X86TTIImpl::hasDivRemOp(Type *DataType, bool IsSigned) {
   EVT VT = TLI->getValueType(DL, DataType);
   return TLI->isOperationLegal(IsSigned ? ISD::SDIVREM : ISD::UDIVREM, VT);
 }
 
 bool X86TTIImpl::isExpensiveToSpeculativelyExecute(const Instruction* I) {

llvm/test/CodeGen/X86/x86-prefer-no-gather-no-scatter.ll

+; Check that if option prefer-no-gather/scatter can disable gather/scatter instructions.
+; RUN: llc -mattr=+avx2,+fast-gather %s -o - | FileCheck %s --check-prefixes=GATHER
+; RUN: llc -mattr=+avx2,+fast-gather,+prefer-no-gather %s -o - | FileCheck %s --check-prefixes=NO-GATHER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  -mattr=+avx512vl,+avx512dq < %s | FileCheck %s --check-prefix=SCATTER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  -mattr=+avx512vl,+avx512dq,+prefer-no-gather < %s | FileCheck %s --check-prefix=SCATTER-NO-GATHER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  -mattr=+avx512vl,+avx512dq,+prefer-no-scatter < %s | FileCheck %s --check-prefix=GATHER-NO-SCATTER
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  -mattr=+avx512vl,+avx512dq,+prefer-no-gather,+prefer-no-scatter < %s | FileCheck %s --check-prefix=NO-SCATTER-GATHER
+
+@A = global [1024 x i8] zeroinitializer, align 128
+@B = global [1024 x i64] zeroinitializer, align 128
+@C = global [1024 x i64] zeroinitializer, align 128
+
+; This tests the function that if prefer-no-gather can disable lowerMGather
+define void @test() #0 {
+; GATHER-LABEL: test:
+; GATHER: vpgatherdq
+;
+; NO-GATHER-LABEL: test:
+; NO-GATHER-NOT: vpgatherdq
+;
+; GATHER-NO-SCATTER-LABEL: test:
+; GATHER-NO-SCATTER: vpgatherdq
+;
+; NO-SCATTER-GATHER-LABEL: test:
+; NO-SCATTER-GATHER-NOT: vpgatherdq
+iter.check:
+  br i1 false, label %vec.epilog.scalar.ph, label %vector.main.loop.iter.check
+
+vector.main.loop.iter.check:                      ; preds = %iter.check
+  br i1 false, label %vec.epilog.ph, label %vector.ph
+
+vector.ph:                                        ; preds = %vector.main.loop.iter.check
+  br label %vector.body
+
+vector.body:                                      ; preds = %vector.body, %vector.ph
+  %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
+  %0 = add i64 %index, 0
+  %1 = getelementptr inbounds [1024 x i8], ptr @A, i64 0, i64 %0
+  %2 = getelementptr inbounds i8, ptr %1, i32 0
+  %wide.load = load <32 x i8>, ptr %2, align 1
+  %3 = sext <32 x i8> %wide.load to <32 x i64>
+  %4 = getelementptr inbounds [1024 x i64], ptr @B, i64 0, <32 x i64> %3
+  %wide.masked.gather = call <32 x i64> @llvm.masked.gather.v32i64.v32p0(<32 x ptr> %4, i32 8, <32 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <32 x i64> poison)
+  %5 = getelementptr inbounds [1024 x i64], ptr @C, i64 0, i64 %0
+  %6 = getelementptr inbounds i64, ptr %5, i32 0
+  store <32 x i64> %wide.masked.gather, ptr %6, align 8
+  %index.next = add nuw i64 %index, 32
+  %7 = icmp eq i64 %index.next, 1024
+  br i1 %7, label %middle.block, label %vector.body, !llvm.loop !0
+
+middle.block:                                     ; preds = %vector.body
+  %cmp.n = icmp eq i64 1024, 1024
+  br i1 %cmp.n, label %for.cond.cleanup, label %vec.epilog.iter.check
+
+vec.epilog.iter.check:                            ; preds = %middle.block
+  br i1 true, label %vec.epilog.scalar.ph, label %vec.epilog.ph
+
+vec.epilog.ph:                                    ; preds = %vector.main.loop.iter.check, %vec.epilog.iter.check
+  %vec.epilog.resume.val = phi i64 [ 1024, %vec.epilog.iter.check ], [ 0, %vector.main.loop.iter.check ]
+  br label %vec.epilog.vector.body
+
+vec.epilog.vector.body:                           ; preds = %vec.epilog.vector.body, %vec.epilog.ph
+  %index2 = phi i64 [ %vec.epilog.resume.val, %vec.epilog.ph ], [ %index.next5, %vec.epilog.vector.body ]
+  %8 = add i64 %index2, 0
+  %9 = getelementptr inbounds [1024 x i8], ptr @A, i64 0, i64 %8
+  %10 = getelementptr inbounds i8, ptr %9, i32 0
+  %wide.load3 = load <16 x i8>, ptr %10, align 1
+  %11 = sext <16 x i8> %wide.load3 to <16 x i64>
+  %12 = getelementptr inbounds [1024 x i64], ptr @B, i64 0, <16 x i64> %11
+  %wide.masked.gather4 = call <16 x i64> @llvm.masked.gather.v16i64.v16p0(<16 x ptr> %12, i32 8, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <16 x i64> poison)
+  %13 = getelementptr inbounds [1024 x i64], ptr @C, i64 0, i64 %8
+  %14 = getelementptr inbounds i64, ptr %13, i32 0
+  store <16 x i64> %wide.masked.gather4, ptr %14, align 8
+  %index.next5 = add nuw i64 %index2, 16
+  %15 = icmp eq i64 %index.next5, 1024
+  br i1 %15, label %vec.epilog.middle.block, label %vec.epilog.vector.body, !llvm.loop !2
+
+vec.epilog.middle.block:                          ; preds = %vec.epilog.vector.body
+  %cmp.n1 = icmp eq i64 1024, 1024
+  br i1 %cmp.n1, label %for.cond.cleanup, label %vec.epilog.scalar.ph
+
+vec.epilog.scalar.ph:                             ; preds = %iter.check, %vec.epilog.iter.check, %vec.epilog.middle.block
+  %bc.resume.val = phi i64 [ 1024, %vec.epilog.middle.block ], [ 1024, %vec.epilog.iter.check ], [ 0, %iter.check ]
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %vec.epilog.scalar.ph
+  %iv = phi i64 [ %bc.resume.val, %vec.epilog.scalar.ph ], [ %iv.next, %for.body ]
+  %inA = getelementptr inbounds [1024 x i8], ptr @A, i64 0, i64 %iv
+  %valA = load i8, ptr %inA, align 1
+  %valA.ext = sext i8 %valA to i64
+  %inB = getelementptr inbounds [1024 x i64], ptr @B, i64 0, i64 %valA.ext
+  %valB = load i64, ptr %inB, align 8
+  %out = getelementptr inbounds [1024 x i64], ptr @C, i64 0, i64 %iv
+  store i64 %valB, ptr %out, align 8
+  %iv.next = add nuw nsw i64 %iv, 1
+  %cmp = icmp ult i64 %iv.next, 1024
+  br i1 %cmp, label %for.body, label %for.cond.cleanup, !llvm.loop !4
+
+for.cond.cleanup:                                 ; preds = %vec.epilog.middle.block, %middle.block, %for.body
+  ret void
+}
+
+declare <32 x i64> @llvm.masked.gather.v32i64.v32p0(<32 x ptr>, i32 immarg, <32 x i1>, <32 x i64>) #1
+
+declare <16 x i64> @llvm.masked.gather.v16i64.v16p0(<16 x ptr>, i32 immarg, <16 x i1>, <16 x i64>) #1
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.isvectorized", i32 1}
+!2 = distinct !{!2, !1, !3}
+!3 = !{!"llvm.loop.unroll.runtime.disable"}
+!4 = distinct !{!4, !3, !1}
+
+; This tests the function that if prefer-no-gather can disable ScalarizeMaskedGather
+define <4 x float> @gather_v4f32_ptr_v4i32(<4 x ptr> %ptr, <4 x i32> %trigger, <4 x float> %passthru) {
+; GATHER-LABEL: gather_v4f32_ptr_v4i32:
+; GATHER: vgatherqps
+;
+; NO-GATHER-LABEL: gather_v4f32_ptr_v4i32:
+; NO-GATHER-NOT: vgatherqps
+;
+; GATHER-NO-SCATTER-LABEL: gather_v4f32_ptr_v4i32:
+; GATHER-NO-SCATTER: vgatherqps
+;
+; NO-SCATTER-GATHER-LABEL: gather_v4f32_ptr_v4i32:
+; NO-SCATTER-GATHER-NOT: vgatherqps
+  %mask = icmp eq <4 x i32> %trigger, zeroinitializer
+  %res = call <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr> %ptr, i32 4, <4 x i1> %mask, <4 x float> %passthru)
+  ret <4 x float> %res
+}
+
+declare <4 x float> @llvm.masked.gather.v4f32.v4p0(<4 x ptr>, i32, <4 x i1>, <4 x float>)
+
+%struct.a = type { [4 x i32], [4 x i8], %struct.b, i32 }
+%struct.b = type { i32, i32 }
+@c = external dso_local global %struct.a, align 4
+
+; This tests the function that if prefer-no-gather can disable ScalarizeMaskedGather
+define <8 x i32> @gather_v8i32_v8i32(<8 x i32> %trigger) {
+; GATHER-LABEL: gather_v8i32_v8i32:
+; GATHER: vpgatherdd
+;
+; NO-GATHER-LABEL: gather_v8i32_v8i32:
+; NO-GATHER-NOT: vpgatherdd
+;
+; NO-SCATTER-GATHER-LABEL: gather_v8i32_v8i32:
+; NO-SCATTER-GATHER-NOT: vpgatherdd
+  %1 = icmp eq <8 x i32> %trigger, zeroinitializer
+  %2 = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> getelementptr (%struct.a, <8 x ptr> <ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c>, <8 x i64> zeroinitializer, i32 0, <8 x i64> <i64 3, i64 3, i64 3, i64 3, i64 3, i64 3, i64 3, i64 3>), i32 4, <8 x i1> %1, <8 x i32> undef)
+  %3 = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> getelementptr (%struct.a, <8 x ptr> <ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c>, <8 x i64> zeroinitializer, i32 3), i32 4, <8 x i1> %1, <8 x i32> undef)
+  %4 = add <8 x i32> %2, %3
+  %5 = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> getelementptr (%struct.a, <8 x ptr> <ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c, ptr @c>, <8 x i64> zeroinitializer, i32 3), i32 4, <8 x i1> %1, <8 x i32> undef)
+  %6 = add <8 x i32> %4, %5
+  ret <8 x i32> %6
+}
+
+declare <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr>, i32, <8 x i1>, <8 x i32>)
+
+; scatter test cases 
+define void @scatter_test1(ptr %base, <16 x i32> %ind, i16 %mask, <16 x i32>%val) {
+; SCATTER-LABEL: scatter_test1:
+; SCATTER: vpscatterdd
+;
+; SCATTER-NO-GATHER-LABEL: scatter_test1:
+; SCATTER-NO-GATHER: vpscatterdd
+;
+; GATHER-NO-SCATTER-LABEL: scatter_test1:
+; GATHER-NO-SCATTER-NOT: vpscatterdd
+;
+; NO-SCATTER-GATHER-LABEL: scatter_test1:
+; NO-SCATTER-GATHER-NOT: vpscatterdd
+  %broadcast.splatinsert = insertelement <16 x ptr> undef, ptr %base, i32 0
+  %broadcast.splat = shufflevector <16 x ptr> %broadcast.splatinsert, <16 x ptr> undef, <16 x i32> zeroinitializer
+
+  %gep.random = getelementptr i32, <16 x ptr> %broadcast.splat, <16 x i32> %ind
+  %imask = bitcast i16 %mask to <16 x i1>
+  call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32>%val, <16 x ptr> %gep.random, i32 4, <16 x i1> %imask)
+  call void @llvm.masked.scatter.v16i32.v16p0(<16 x i32>%val, <16 x ptr> %gep.random, i32 4, <16 x i1> %imask)
+  ret void
+}
+
+declare void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> , <8 x ptr> , i32 , <8 x i1> )
+declare void @llvm.masked.scatter.v16i32.v16p0(<16 x i32> , <16 x ptr> , i32 , <16 x i1> )
+
+define <8 x i32> @scatter_test2(<8 x i32>%a1, <8 x ptr> %ptr) {
+; SCATTER-LABEL: scatter_test2:
+; SCATTER: vpscatterqd
+;
+; SCATTER-NO-GATHER-LABEL: scatter_test2:
+; SCATTER-NO-GATHER: vpscatterqd
+;
+; GATHER-NO-SCATTER-LABEL: scatter_test2:
+; GATHER-NO-SCATTER-NOT: vpscatterqd
+;
+; NO-SCATTER-GATHER-LABEL: scatter_test2:
+; NO-SCATTER-GATHER-NOT: vpscatterqd
+  %a = call <8 x i32> @llvm.masked.gather.v8i32.v8p0(<8 x ptr> %ptr, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> undef)
+
+  call void @llvm.masked.scatter.v8i32.v8p0(<8 x i32> %a1, <8 x ptr> %ptr, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
+  ret <8 x i32>%a
+}