[ARM] [AARCH64] Add CodeGen IR tests for {VS}QRDML{AS}H v8.1a intrinsics.

[labrinea]

Summary

The existing tests are checking back-end generated assembly. Instead, we want to check front-end generated IR.

[echristo]

Please remove the asm tests here. As I stated in the original review thread there's no reason for them to be here.

Thanks.

-eric

[labrinea]

ASM tests have been removed.

[echristo]

One inline comment, thanks!

-eric

test/CodeGen/aarch64-v8.1a-neon-intrinsics.c

4

Why do you need to enable the optimizers?

[labrinea]

test/CodeGen/aarch64-v8.1a-neon-intrinsics.c

4

Our intention with these tests is to check that we are generating a sequence of {v/s}qrdmulh, {v/s}q{add/sub}{s}, shufflevector, {insert/extract}element IR instructions. Using -O1 promotes memory to registers, combines instructions, and therefore decreases the context of IR that we need to check.

[echristo]

Should be pretty easy to either use CHECK-DAG or pick out the particular instructions you want to check here. Otherwise you're just checking how the optimizer runs. That, in particular, also sounds like a good backend check.

[jmolloy]

Eric is reviewing this; resigning myself.

[labrinea]

Hi Eric,

The main optimization I feel is useful is mem2reg. Without that, if I want to properly check the right values go to the right operands of the intrinsic calls I have to write FileCheck matchers that match stores and their relevant loads, plus bitcasts. This not only looks more obfuscated than matching the mem2reg output, but it is also less resilient to changes in the way clang code generates.

The generated IR for each intrinsic is around 50 lines. I can just pick out the particular instructions I want to check, as you suggested, but they won't we connected by the flow of values. In my opinion such a test will be less valuable.

I can do this both ways but my preferred way is to run the bare minimum of optimization to de-cruft the output and make the test robust and readable. If you feel however that you don't want the optimizers run I will make a best effort at writing a test that doesn't use them.

[echristo]

I understand the conflicting priorities here for sure. You'd like a test that's as minimal as possible, without having to depend on external (to clang) libraries here. I really would appreciate it if you'd make the test not rely on mem2reg etc so we can be sure that clang's code generation is the thing tested here and not the optimizer. Making sure that the unoptimized output reduces properly would be a great opt test for the backend though.

Thanks!

-eric

[labrinea]

Disabled optimizers.

[echristo]

LGTM, and thanks for all of the iteration.

-eric

test/CodeGen/aarch64-v8.1a-neon-intrinsics.c

 // REQUIRES: aarch64-registered-target
 
 // RUN: %clang_cc1 -triple aarch64-linux-gnu -target-feature +neon \
-// RUN:  -target-feature +v8.1a -O3 -S -o - %s \
-// RUN:  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-AARCH64
+// RUN:  -target-feature +v8.1a -O1 -S -emit-llvm -o - %s | FileCheck %s
 
  #include <arm_neon.h>
 
-// CHECK-AARCH64-LABEL: test_vqrdmlah_laneq_s16
+// CHECK-LABEL: test_vqrdmlah_laneq_s16
 int16x4_t test_vqrdmlah_laneq_s16(int16x4_t a, int16x4_t b, int16x8_t v) {
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.h[7]
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %v, <8 x i16> undef, <4 x i32> <i32 7, i32 7, i32 7, i32 7>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK: tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlah_laneq_s16(a, b, v, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlah_laneq_s32
+// CHECK-LABEL: test_vqrdmlah_laneq_s32
 int32x2_t test_vqrdmlah_laneq_s32(int32x2_t a, int32x2_t b, int32x4_t v) {
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.s[3]
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i32> %v, <4 x i32> undef, <2 x i32> <i32 3, i32 3>
+// CHECK: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK: tail call <2 x i32> @llvm.aarch64.neon.sqadd.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlah_laneq_s32(a, b, v, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahq_laneq_s16
+// CHECK-LABEL: test_vqrdmlahq_laneq_s16
 int16x8_t test_vqrdmlahq_laneq_s16(int16x8_t a, int16x8_t b, int16x8_t v) {
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.h[7]
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %v, <8 x i16> undef, <8 x i32> <i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7>
+// CHECK: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK: tail call <8 x i16> @llvm.aarch64.neon.sqadd.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlahq_laneq_s16(a, b, v, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahq_laneq_s32
+// CHECK-LABEL: test_vqrdmlahq_laneq_s32
 int32x4_t test_vqrdmlahq_laneq_s32(int32x4_t a, int32x4_t b, int32x4_t v) {
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.s[3]
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i32> %v, <4 x i32> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK: tail call <4 x i32> @llvm.aarch64.neon.sqadd.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlahq_laneq_s32(a, b, v, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahh_s16
+// CHECK-LABEL: test_vqrdmlahh_s16
 int16_t test_vqrdmlahh_s16(int16_t a, int16_t b, int16_t c) {
-// CHECK-AARCH64: sqrdmlah {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[insc:%.*]] = insertelement <4 x i16> undef, i16 %c, i64 0
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[insc]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[add:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[add]], i64 0
   return vqrdmlahh_s16(a, b, c);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahs_s32
+// CHECK-LABEL: test_vqrdmlahs_s32
 int32_t test_vqrdmlahs_s32(int32_t a, int32_t b, int32_t c) {
-// CHECK-AARCH64: sqrdmlah {{s[0-9]+}}, {{s[0-9]+}}, {{s[0-9]+}}
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 %c)
+// CHECK: tail call i32 @llvm.aarch64.neon.sqadd.i32(i32 %a, i32 [[mul]])
   return vqrdmlahs_s32(a, b, c);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahh_lane_s16
+// CHECK-LABEL: test_vqrdmlahh_lane_s16
 int16_t test_vqrdmlahh_lane_s16(int16_t a, int16_t b, int16x4_t c) {
-// CHECK-AARCH64: sqrdmlah {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{v[0-9]+}}.h[3]
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 undef, i32 undef, i32 undef>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[shuffle]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[add:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[add]], i64 0
   return vqrdmlahh_lane_s16(a, b, c, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahs_lane_s32
+// CHECK-LABEL: test_vqrdmlahs_lane_s32
 int32_t test_vqrdmlahs_lane_s32(int32_t a, int32_t b, int32x2_t c) {
-// CHECK-AARCH64: sqrdmlah {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
+// CHECK: [[extc:%.*]] = extractelement <2 x i32> %c, i32 1
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 [[extc]])
+// CHECK: tail call i32 @llvm.aarch64.neon.sqadd.i32(i32 %a, i32 [[mul]])
   return vqrdmlahs_lane_s32(a, b, c, 1);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahh_laneq_s16
+// CHECK-LABEL: test_vqrdmlahh_laneq_s16
 int16_t test_vqrdmlahh_laneq_s16(int16_t a, int16_t b, int16x8_t c) {
-// CHECK-AARCH64: sqrdmlah {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{v[0-9]+}}.h[7]
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %c, <8 x i16> undef, <4 x i32> <i32 7, i32 undef, i32 undef, i32 undef>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[shuffle]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[add:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[add]], i64 0
   return vqrdmlahh_laneq_s16(a, b, c, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlahs_laneq_s32
+// CHECK-LABEL: test_vqrdmlahs_laneq_s32
 int32_t test_vqrdmlahs_laneq_s32(int32_t a, int32_t b, int32x4_t c) {
-// CHECK-AARCH64: sqrdmlah {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
+// CHECK: [[extc:%.*]] = extractelement <4 x i32> %c, i32 3
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 [[extc]])
+// CHECK: tail call i32 @llvm.aarch64.neon.sqadd.i32(i32 %a, i32 [[mul]])
   return vqrdmlahs_laneq_s32(a, b, c, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlsh_laneq_s16
+// CHECK-LABEL: test_vqrdmlsh_laneq_s16
 int16x4_t test_vqrdmlsh_laneq_s16(int16x4_t a, int16x4_t b, int16x8_t v) {
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.h[7]
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %v, <8 x i16> undef, <4 x i32> <i32 7, i32 7, i32 7, i32 7>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK: tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlsh_laneq_s16(a, b, v, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlsh_laneq_s32
+// CHECK-LABEL: test_vqrdmlsh_laneq_s32
 int32x2_t test_vqrdmlsh_laneq_s32(int32x2_t a, int32x2_t b, int32x4_t v) {
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.s[3]
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i32> %v, <4 x i32> undef, <2 x i32> <i32 3, i32 3>
+// CHECK: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK: tail call <2 x i32> @llvm.aarch64.neon.sqsub.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlsh_laneq_s32(a, b, v, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshq_laneq_s16
+// CHECK-LABEL: test_vqrdmlshq_laneq_s16
 int16x8_t test_vqrdmlshq_laneq_s16(int16x8_t a, int16x8_t b, int16x8_t v) {
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.h[7]
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %v, <8 x i16> undef, <8 x i32> <i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7, i32 7>
+// CHECK: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK: tail call <8 x i16> @llvm.aarch64.neon.sqsub.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlshq_laneq_s16(a, b, v, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshq_laneq_s32
+// CHECK-LABEL: test_vqrdmlshq_laneq_s32
 int32x4_t test_vqrdmlshq_laneq_s32(int32x4_t a, int32x4_t b, int32x4_t v) {
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.s[3]
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i32> %v, <4 x i32> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK: tail call <4 x i32> @llvm.aarch64.neon.sqsub.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlshq_laneq_s32(a, b, v, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshh_s16
+// CHECK-LABEL: test_vqrdmlshh_s16
 int16_t test_vqrdmlshh_s16(int16_t a, int16_t b, int16_t c) {
-// CHECK-AARCH64: sqrdmlsh {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[insc:%.*]] = insertelement <4 x i16> undef, i16 %c, i64 0
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[insc]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[sub:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[sub]], i64 0
   return vqrdmlshh_s16(a, b, c);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshs_s32
+// CHECK-LABEL: test_vqrdmlshs_s32
 int32_t test_vqrdmlshs_s32(int32_t a, int32_t b, int32_t c) {
-// CHECK-AARCH64: sqrdmlsh {{s[0-9]+}}, {{s[0-9]+}}, {{s[0-9]+}}
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 %c)
+// CHECK: tail call i32 @llvm.aarch64.neon.sqsub.i32(i32 %a, i32 [[mul]])
   return vqrdmlshs_s32(a, b, c);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshh_lane_s16
+// CHECK-LABEL: test_vqrdmlshh_lane_s16
 int16_t test_vqrdmlshh_lane_s16(int16_t a, int16_t b, int16x4_t c) {
-// CHECK-AARCH64: sqrdmlsh {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{v[0-9]+}}.h[3]
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 undef, i32 undef, i32 undef>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[shuffle]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[sub:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[sub]], i64 0
   return vqrdmlshh_lane_s16(a, b, c, 3);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshs_lane_s32
+// CHECK-LABEL: test_vqrdmlshs_lane_s32
 int32_t test_vqrdmlshs_lane_s32(int32_t a, int32_t b, int32x2_t c) {
-// CHECK-AARCH64: sqrdmlsh {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[1]
+// CHECK: [[extc:%.*]] = extractelement <2 x i32> %c, i32 1
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 [[extc]])
+// CHECK: tail call i32 @llvm.aarch64.neon.sqsub.i32(i32 %a, i32 [[mul]])
   return vqrdmlshs_lane_s32(a, b, c, 1);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshh_laneq_s16
+// CHECK-LABEL: test_vqrdmlshh_laneq_s16
 int16_t test_vqrdmlshh_laneq_s16(int16_t a, int16_t b, int16x8_t c) {
-// CHECK-AARCH64: sqrdmlsh {{h[0-9]+|v[0-9]+.4h}}, {{h[0-9]+|v[0-9]+.4h}}, {{v[0-9]+}}.h[7]
+// CHECK: [[insb:%.*]] = insertelement <4 x i16> undef, i16 %b, i64 0
+// CHECK: [[shuffle:%.*]] = shufflevector <8 x i16> %c, <8 x i16> undef, <4 x i32> <i32 7, i32 undef, i32 undef, i32 undef>
+// CHECK: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> [[insb]], <4 x i16> [[shuffle]])
+// CHECK: [[insa:%.*]] = insertelement <4 x i16> undef, i16 %a, i64 0
+// CHECK: [[sub:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> [[insa]], <4 x i16> [[mul]])
+// CHECK: extractelement <4 x i16> [[sub]], i64 0
   return vqrdmlshh_laneq_s16(a, b, c, 7);
 }
 
-// CHECK-AARCH64-LABEL: test_vqrdmlshs_laneq_s32
+// CHECK-LABEL: test_vqrdmlshs_laneq_s32
 int32_t test_vqrdmlshs_laneq_s32(int32_t a, int32_t b, int32x4_t c) {
-// CHECK-AARCH64: sqrdmlsh {{s[0-9]+}}, {{s[0-9]+}}, {{v[0-9]+}}.s[3]
+// CHECK: [[extc:%.*]] = extractelement <4 x i32> %c, i32 3
+// CHECK: [[mul:%.*]] = tail call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %b, i32 [[extc]])
+// CHECK: tail call i32 @llvm.aarch64.neon.sqsub.i32(i32 %a, i32 [[mul]])
   return vqrdmlshs_laneq_s32(a, b, c, 3);
 }
 

test/CodeGen/arm-v8.1a-neon-intrinsics.c

 // RUN: %clang_cc1 -triple armv8.1a-linux-gnu -target-feature +neon \
-// RUN:  -O3 -S -o - %s \
+// RUN:  -O1 -S -emit-llvm -o - %s \
 // RUN:  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-ARM
+
 // RUN: %clang_cc1 -triple aarch64-linux-gnu -target-feature +neon \
-// RUN:  -target-feature +v8.1a -O3 -S -o - %s \
+// RUN:  -target-feature +v8.1a -O1 -S -emit-llvm -o - %s \
 // RUN:  | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-AARCH64
+
 // REQUIRES: arm-registered-target,aarch64-registered-target
 
 #include <arm_neon.h>
 
 // CHECK-LABEL: test_vqrdmlah_s16
 int16x4_t test_vqrdmlah_s16(int16x4_t a, int16x4_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlah.s16 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.4h
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i16> @llvm.arm.neon.vqrdmulh.v4i16(<4 x i16> %b, <4 x i16> %c)
+// CHECK-ARM: tail call <4 x i16> @llvm.arm.neon.vqadds.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> %c)
+// CHECK-AARCH64: tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlah_s16(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlah_s32
 int32x2_t test_vqrdmlah_s32(int32x2_t a, int32x2_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlah.s32 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.2s
+// CHECK-ARM: [[mul:%.*]] = tail call <2 x i32> @llvm.arm.neon.vqrdmulh.v2i32(<2 x i32> %b, <2 x i32> %c)
+// CHECK-ARM: tail call <2 x i32> @llvm.arm.neon.vqadds.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> %c)
+// CHECK-AARCH64: tail call <2 x i32> @llvm.aarch64.neon.sqadd.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlah_s32(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlahq_s16
 int16x8_t test_vqrdmlahq_s16(int16x8_t a, int16x8_t b, int16x8_t c) {
-// CHECK-ARM: vqrdmlah.s16 q{{[0-9]+}}, q{{[0-9]+}}, q{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.8h
+// CHECK-ARM: [[mul:%.*]] = tail call <8 x i16> @llvm.arm.neon.vqrdmulh.v8i16(<8 x i16> %b, <8 x i16> %c)
+// CHECK-ARM: tail call <8 x i16> @llvm.arm.neon.vqadds.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> %c)
+// CHECK-AARCH64: tail call <8 x i16> @llvm.aarch64.neon.sqadd.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlahq_s16(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlahq_s32
 int32x4_t test_vqrdmlahq_s32(int32x4_t a, int32x4_t b, int32x4_t c) {
-// CHECK-ARM: vqrdmlah.s32 q{{[0-9]+}}, q{{[0-9]+}}, q{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.4s
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i32> @llvm.arm.neon.vqrdmulh.v4i32(<4 x i32> %b, <4 x i32> %c)
+// CHECK-ARM: tail call <4 x i32> @llvm.arm.neon.vqadds.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> %c)
+// CHECK-AARCH64: tail call <4 x i32> @llvm.aarch64.neon.sqadd.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlahq_s32(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlah_lane_s16
 int16x4_t test_vqrdmlah_lane_s16(int16x4_t a, int16x4_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlah.s16 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}[3]
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.h[3]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i16> @llvm.arm.neon.vqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK-ARM: tail call <4 x i16> @llvm.arm.neon.vqadds.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK-AARCH64: tail call <4 x i16> @llvm.aarch64.neon.sqadd.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlah_lane_s16(a, b, c, 3);
 }
 
 // CHECK-LABEL: test_vqrdmlah_lane_s32
 int32x2_t test_vqrdmlah_lane_s32(int32x2_t a, int32x2_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlah.s32 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}[1]
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.s[1]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <2 x i32> <i32 1, i32 1>
+// CHECK-ARM: [[mul:%.*]] = tail call <2 x i32> @llvm.arm.neon.vqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK-ARM: tail call <2 x i32> @llvm.arm.neon.vqadds.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <2 x i32> <i32 1, i32 1>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK-AARCH64: tail call <2 x i32> @llvm.aarch64.neon.sqadd.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlah_lane_s32(a, b, c, 1);
 }
 
 // CHECK-LABEL: test_vqrdmlahq_lane_s16
 int16x8_t test_vqrdmlahq_lane_s16(int16x8_t a, int16x8_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlah.s16 q{{[0-9]+}}, q{{[0-9]+}}, d{{[0-9]+}}[3]
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.h[3]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <8 x i32> <i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3>
+// CHECK-ARM: [[mul:%.*]] = tail call <8 x i16> @llvm.arm.neon.vqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK-ARM: tail call <8 x i16> @llvm.arm.neon.vqadds.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <8 x i32> <i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK-AARCH64: tail call <8 x i16> @llvm.aarch64.neon.sqadd.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlahq_lane_s16(a, b, c, 3);
 }
 
 // CHECK-LABEL: test_vqrdmlahq_lane_s32
 int32x4_t test_vqrdmlahq_lane_s32(int32x4_t a, int32x4_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlah.s32 q{{[0-9]+}}, q{{[0-9]+}}, d{{[0-9]+}}[1]
-// CHECK-AARCH64: sqrdmlah {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.s[1]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i32> @llvm.arm.neon.vqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK-ARM: tail call <4 x i32> @llvm.arm.neon.vqadds.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK-AARCH64: tail call <4 x i32> @llvm.aarch64.neon.sqadd.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlahq_lane_s32(a, b, c, 1);
 }
 
 // CHECK-LABEL: test_vqrdmlsh_s16
 int16x4_t test_vqrdmlsh_s16(int16x4_t a, int16x4_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlsh.s16 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.4h
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i16> @llvm.arm.neon.vqrdmulh.v4i16(<4 x i16> %b, <4 x i16> %c)
+// CHECK-ARM: tail call <4 x i16> @llvm.arm.neon.vqsubs.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> %c)
+// CHECK-AARCH64: tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlsh_s16(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlsh_s32
 int32x2_t test_vqrdmlsh_s32(int32x2_t a, int32x2_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlsh.s32 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.2s
+// CHECK-ARM: [[mul:%.*]] = tail call <2 x i32> @llvm.arm.neon.vqrdmulh.v2i32(<2 x i32> %b, <2 x i32> %c)
+// CHECK-ARM: tail call <2 x i32> @llvm.arm.neon.vqsubs.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> %c)
+// CHECK-AARCH64: tail call <2 x i32> @llvm.aarch64.neon.sqsub.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlsh_s32(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlshq_s16
 int16x8_t test_vqrdmlshq_s16(int16x8_t a, int16x8_t b, int16x8_t c) {
-// CHECK-ARM: vqrdmlsh.s16 q{{[0-9]+}}, q{{[0-9]+}}, q{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.8h
+// CHECK-ARM: [[mul:%.*]] = tail call <8 x i16> @llvm.arm.neon.vqrdmulh.v8i16(<8 x i16> %b, <8 x i16> %c)
+// CHECK-ARM: tail call <8 x i16> @llvm.arm.neon.vqsubs.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> %c)
+// CHECK-AARCH64: tail call <8 x i16> @llvm.aarch64.neon.sqsub.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlshq_s16(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlshq_s32
 int32x4_t test_vqrdmlshq_s32(int32x4_t a, int32x4_t b, int32x4_t c) {
-// CHECK-ARM: vqrdmlsh.s32 q{{[0-9]+}}, q{{[0-9]+}}, q{{[0-9]+}}
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.4s
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i32> @llvm.arm.neon.vqrdmulh.v4i32(<4 x i32> %b, <4 x i32> %c)
+// CHECK-ARM: tail call <4 x i32> @llvm.arm.neon.vqsubs.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
+
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> %c)
+// CHECK-AARCH64: tail call <4 x i32> @llvm.aarch64.neon.sqsub.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlshq_s32(a, b, c);
 }
 
 // CHECK-LABEL: test_vqrdmlsh_lane_s16
 int16x4_t test_vqrdmlsh_lane_s16(int16x4_t a, int16x4_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlsh.s16 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}[3]
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4h, {{v[0-9]+}}.4h, {{v[0-9]+}}.h[3]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i16> @llvm.arm.neon.vqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK-ARM: tail call <4 x i16> @llvm.arm.neon.vqsubs.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <4 x i32> <i32 3, i32 3, i32 3, i32 3>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i16> @llvm.aarch64.neon.sqrdmulh.v4i16(<4 x i16> %b, <4 x i16> [[shuffle]])
+// CHECK-AARCH64: tail call <4 x i16> @llvm.aarch64.neon.sqsub.v4i16(<4 x i16> %a, <4 x i16> [[mul]])
   return vqrdmlsh_lane_s16(a, b, c, 3);
 }
 
 // CHECK-LABEL: test_vqrdmlsh_lane_s32
 int32x2_t test_vqrdmlsh_lane_s32(int32x2_t a, int32x2_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlsh.s32 d{{[0-9]+}}, d{{[0-9]+}}, d{{[0-9]+}}[1]
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.2s, {{v[0-9]+}}.2s, {{v[0-9]+}}.s[1]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <2 x i32> <i32 1, i32 1>
+// CHECK-ARM: [[mul:%.*]] = tail call <2 x i32> @llvm.arm.neon.vqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK-ARM: tail call <2 x i32> @llvm.arm.neon.vqsubs.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <2 x i32> <i32 1, i32 1>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <2 x i32> @llvm.aarch64.neon.sqrdmulh.v2i32(<2 x i32> %b, <2 x i32> [[shuffle]])
+// CHECK-AARCH64: tail call <2 x i32> @llvm.aarch64.neon.sqsub.v2i32(<2 x i32> %a, <2 x i32> [[mul]])
   return vqrdmlsh_lane_s32(a, b, c, 1);
 }
 
 // CHECK-LABEL: test_vqrdmlshq_lane_s16
 int16x8_t test_vqrdmlshq_lane_s16(int16x8_t a, int16x8_t b, int16x4_t c) {
-// CHECK-ARM: vqrdmlsh.s16 q{{[0-9]+}}, q{{[0-9]+}}, d{{[0-9]+}}[3]
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.8h, {{v[0-9]+}}.8h, {{v[0-9]+}}.h[3]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <8 x i32> <i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3>
+// CHECK-ARM: [[mul:%.*]] = tail call <8 x i16> @llvm.arm.neon.vqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK-ARM: tail call <8 x i16> @llvm.arm.neon.vqsubs.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <4 x i16> %c, <4 x i16> undef, <8 x i32> <i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <8 x i16> @llvm.aarch64.neon.sqrdmulh.v8i16(<8 x i16> %b, <8 x i16> [[shuffle]])
+// CHECK-AARCH64: tail call <8 x i16> @llvm.aarch64.neon.sqsub.v8i16(<8 x i16> %a, <8 x i16> [[mul]])
   return vqrdmlshq_lane_s16(a, b, c, 3);
 }
 
 // CHECK-LABEL: test_vqrdmlshq_lane_s32
 int32x4_t test_vqrdmlshq_lane_s32(int32x4_t a, int32x4_t b, int32x2_t c) {
-// CHECK-ARM: vqrdmlsh.s32 q{{[0-9]+}}, q{{[0-9]+}}, d{{[0-9]+}}[1]
-// CHECK-AARCH64: sqrdmlsh {{v[0-9]+}}.4s, {{v[0-9]+}}.4s, {{v[0-9]+}}.s[1]
+// CHECK-ARM: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-ARM: [[mul:%.*]] = tail call <4 x i32> @llvm.arm.neon.vqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK-ARM: tail call <4 x i32> @llvm.arm.neon.vqsubs.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
+
+// CHECK-AARCH64: [[shuffle:%.*]] = shufflevector <2 x i32> %c, <2 x i32> undef, <4 x i32> <i32 1, i32 1, i32 1, i32 1>
+// CHECK-AARCH64: [[mul:%.*]] = tail call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %b, <4 x i32> [[shuffle]])
+// CHECK-AARCH64: tail call <4 x i32> @llvm.aarch64.neon.sqsub.v4i32(<4 x i32> %a, <4 x i32> [[mul]])
   return vqrdmlshq_lane_s32(a, b, c, 1);
 }