[ARM] Assembler support for the ARMv8.2a dot product instructions

[SjoerdMeijer]

Summary

Commit r310480 added the AArch64 ARMv8.2a dot product instructions, this adds the AArch32 instructions.

[SjoerdMeijer]

Oops, I now see that I forgot to add the disassmbler tests.

[SjoerdMeijer]

Added the disassembler tests.

Note that the AArch32 instructions were added to the an instruction encoding space that has been reclaimed from what was originally allocated for the unused “Coprocessor 13”.
That's why I created a DecoderNamespace CoProc and assigned the coprocessor isntructions to it, otherwise the dot product instructions would get disassembled to coprocessor instructions.

[samparker]

Hi Sjoerd,

Shouldn't the feature be predicated on HasV8_2aOps, not just NEON? It seems weird, from the tests, that we can target arm or thumbv7 but still have access to these instructions.

cheers,
sam

[t.p.northover]

Shouldn't the feature be predicated on HasV8_2aOps, not just NEON? It seems weird, from the tests, that we can target arm or thumbv7 but still have access to these instructions.

They're optional in v8.2a (actually, until I saw the XML I thought they were v8.4) so they need a separate feature. I think those implies fields mostly make sense for the grander features like whole architecture levels; Sjoerd's choice is at least consistent with the FeatureCrypto above.

[SjoerdMeijer]

Hi Sam,
Thanks for reviewing. I agree it looks odd, but the argument of not doing this, is that this is not user-facing. I.e., they are internal options and it's Clang's responsibilities to parse options and set the target features. Other optional instructions/extensions are also not predicated on e.g. v8.1a or v8.2a.

Cheers,
Sjoerd.

[samparker]

Ok, all fair points. LGTM then.

lib/Target/ARM/ARM.td

 def FeatureCrypto         : SubtargetFeature<"crypto", "HasCrypto", "true",
                                              "Enable support for "
                                              "Cryptography extensions",
                                              [FeatureNEON]>;
 
 def FeatureCRC            : SubtargetFeature<"crc", "HasCRC", "true",
                                              "Enable support for CRC instructions">;
 
+def FeatureDotProd        : SubtargetFeature<"dotprod", "HasDotProd", "true",
+                                             "Enable support for dot product instructions",
+                                             [FeatureNEON]>;
 
 // Not to be confused with FeatureHasRetAddrStack (return address stack)
 def FeatureRAS            : SubtargetFeature<"ras", "HasRAS", "true",
                                              "Enable Reliability, Availability "
                                              "and Serviceability extensions">;
 
 // Fast computation of non-negative address offsets
 def FeatureFPAO           : SubtargetFeature<"fpao", "HasFPAO", "true",

lib/Target/ARM/ARMInstrInfo.td

def HasDPVFP         : Predicate<"!Subtarget->isFPOnlySP()">,
                                  AssemblerPredicate<"!FeatureVFPOnlySP",
                                                     "double precision VFP">;
 def HasFPARMv8       : Predicate<"Subtarget->hasFPARMv8()">,
                                  AssemblerPredicate<"FeatureFPARMv8", "FPARMv8">;
 def HasNEON          : Predicate<"Subtarget->hasNEON()">,
                                  AssemblerPredicate<"FeatureNEON", "NEON">;
 def HasCrypto        : Predicate<"Subtarget->hasCrypto()">,
                                  AssemblerPredicate<"FeatureCrypto", "crypto">;
+def HasDotProd       : Predicate<"Subtarget->hasDotProd()">,
+                                 AssemblerPredicate<"FeatureDotProd", "dotprod">;
 def HasCRC           : Predicate<"Subtarget->hasCRC()">,
                                  AssemblerPredicate<"FeatureCRC", "crc">;
 def HasRAS           : Predicate<"Subtarget->hasRAS()">,
                                  AssemblerPredicate<"FeatureRAS", "ras">;
 def HasFP16          : Predicate<"Subtarget->hasFP16()">,
                                  AssemblerPredicate<"FeatureFP16","half-float conversions">;
 def HasFullFP16      : Predicate<"Subtarget->hasFullFP16()">,
                                  AssemblerPredicate<"FeatureFullFP16","full half-float">;

lib/Target/ARM/ARMInstrNEON.td

def : Pat<(v4f32 (fma QPR:$Vn, QPR:$Vm, QPR:$src1)),
           Requires<[HasVFP4]>;
 def : Pat<(v2f32 (fma (fneg DPR:$Vn), DPR:$Vm, DPR:$src1)),
           (VFMSfd DPR:$src1, DPR:$Vn, DPR:$Vm)>,
       Requires<[HasVFP4]>;
 def : Pat<(v4f32 (fma (fneg QPR:$Vn), QPR:$Vm, QPR:$src1)),
           (VFMSfq QPR:$src1, QPR:$Vn, QPR:$Vm)>,
       Requires<[HasVFP4]>;
 
+// ARMv8.2a dot product instructions.
+// We put them in the VFPV8 decoder namespace because the ARM and Thumb
+// encodings are the same and thus no further bit twiddling is necessary
+// in the disassembler.
+let Predicates = [HasDotProd], DecoderNamespace = "VFPV8" in {
+
+def VUDOTD : N3Vnp<0b11000, 0b10, 0b1101, 0b0, 0b1,
+                  (outs DPR:$Vd), (ins DPR:$Vn, DPR:$Vm),
+                  N3RegFrm, IIC_VDOTPROD, "vudot", "u8", []>;
+def VSDOTD : N3Vnp<0b11000, 0b10, 0b1101, 0b0, 0b0,
+                  (outs DPR:$Vd), (ins DPR:$Vn, DPR:$Vm),
+                  N3RegFrm, IIC_VDOTPROD, "vsdot", "s8", []>;
+def VUDOTQ : N3Vnp<0b11000, 0b10, 0b1101, 0b1, 0b1,
+                  (outs QPR:$Vd), (ins QPR:$Vn, QPR:$Vm),
+                  N3RegFrm, IIC_VDOTPROD, "vudot", "u8", []>;
+def VSDOTQ : N3Vnp<0b11000, 0b10, 0b1101, 0b1, 0b0,
+                  (outs QPR:$Vd), (ins QPR:$Vn, QPR:$Vm),
+                  N3RegFrm, IIC_VDOTPROD, "vsdot", "s8", []>;
+
+// Indexed dot product instructions:
+class DOTI<string opc, string dt, bit Q, bit U, RegisterClass Ty> :
+  N3Vnp<0b11100, 0b10, 0b1101, Q, U,
+       (outs Ty:$Vd), (ins Ty:$Vn, DPR:$Vm, VectorIndex32:$lane),
+       N3RegFrm, IIC_VDOTPROD, opc, dt, []> {
+  bit lane;
+  let Inst{5} = lane;
+  let AsmString = !strconcat(opc, ".", dt, "\t$Vd, $Vn, $Vm$lane");
+}
+
+def VUDOTDI : DOTI<"vudot", "u8", 0b0, 0b1, DPR>;
+def VSDOTDI : DOTI<"vsdot", "s8", 0b0, 0b0, DPR>;
+def VUDOTQI : DOTI<"vudot", "u8", 0b1, 0b1, QPR>;
+def VSDOTQI : DOTI<"vsdot", "s8", 0b1, 0b0, QPR>;
+
+}  // HasDotProd
+
 // Vector Subtract Operations.
 
 //   VSUB     : Vector Subtract (integer and floating-point)
 defm VSUB     : N3V_QHSD<1, 0, 0b1000, 0, IIC_VSUBiD, IIC_VSUBiQ,
                          "vsub", "i", sub, 0>;
 def  VSUBfd   : N3VD<0, 0, 0b10, 0b1101, 0, IIC_VBIND, "vsub", "f32",
                      v2f32, v2f32, fsub, 0>;
 def  VSUBfq   : N3VQ<0, 0, 0b10, 0b1101, 0, IIC_VBINQ, "vsub", "f32",

lib/Target/ARM/ARMSchedule.td

 def IIC_VTB1       : InstrItinClass;
 def IIC_VTB2       : InstrItinClass;
 def IIC_VTB3       : InstrItinClass;
 def IIC_VTB4       : InstrItinClass;
 def IIC_VTBX1      : InstrItinClass;
 def IIC_VTBX2      : InstrItinClass;
 def IIC_VTBX3      : InstrItinClass;
 def IIC_VTBX4      : InstrItinClass;
+def IIC_VDOTPROD   : InstrItinClass;
 
 //===----------------------------------------------------------------------===//
 // Processor instruction itineraries.
 
 include "ARMScheduleV6.td"
 include "ARMScheduleA8.td"
 include "ARMScheduleA9.td"
 include "ARMScheduleSwift.td"
 include "ARMScheduleR52.td"
 include "ARMScheduleA57.td"
 include "ARMScheduleM3.td"

lib/Target/ARM/ARMSubtarget.h

protected:
   /// HasVFPv2, HasVFPv3, HasVFPv4, HasFPARMv8, HasNEON - Specify what
   /// floating point ISAs are supported.
   bool HasVFPv2 = false;
   bool HasVFPv3 = false;
   bool HasVFPv4 = false;
   bool HasFPARMv8 = false;
   bool HasNEON = false;
 
+  /// HasDotProd - True if the ARMv8.2A dot product instructions are supported.
+  bool HasDotProd = false;
+
   /// UseNEONForSinglePrecisionFP - if the NEONFP attribute has been
   /// specified. Use the method useNEONForSinglePrecisionFP() to
   /// determine if NEON should actually be used.
   bool UseNEONForSinglePrecisionFP = false;
 
   /// UseMulOps - True if non-microcoded fused integer multiply-add and
   /// multiply-subtract instructions should be used.
   bool UseMulOps = false;
public:
   bool hasARMOps() const { return !NoARM; }
 
   bool hasVFP2() const { return HasVFPv2; }
   bool hasVFP3() const { return HasVFPv3; }
   bool hasVFP4() const { return HasVFPv4; }
   bool hasFPARMv8() const { return HasFPARMv8; }
   bool hasNEON() const { return HasNEON;  }
   bool hasCrypto() const { return HasCrypto; }
+  bool hasDotProd() const { return HasDotProd; }
   bool hasCRC() const { return HasCRC; }
   bool hasRAS() const { return HasRAS; }
   bool hasVirtualization() const { return HasVirtualization; }
 
   bool useNEONForSinglePrecisionFP() const {
     return hasNEON() && UseNEONForSinglePrecisionFP;
   }
 

lib/Target/ARM/AsmParser/ARMAsmParser.cpp

if ((Mnemonic == "movs" && isThumb()) ||
       Mnemonic == "vcgt"  || Mnemonic == "vcle"   || Mnemonic == "smlal" ||
       Mnemonic == "umaal" || Mnemonic == "umlal"  || Mnemonic == "vabal" ||
       Mnemonic == "vmlal" || Mnemonic == "vpadal" || Mnemonic == "vqdmlal" ||
       Mnemonic == "fmuls" || Mnemonic == "vmaxnm" || Mnemonic == "vminnm" ||
       Mnemonic == "vcvta" || Mnemonic == "vcvtn"  || Mnemonic == "vcvtp" ||
       Mnemonic == "vcvtm" || Mnemonic == "vrinta" || Mnemonic == "vrintn" ||
       Mnemonic == "vrintp" || Mnemonic == "vrintm" || Mnemonic == "hvc" ||
       Mnemonic.startswith("vsel") || Mnemonic == "vins" || Mnemonic == "vmovx" ||
-      Mnemonic == "bxns"  || Mnemonic == "blxns")
+      Mnemonic == "bxns"  || Mnemonic == "blxns" ||
+      Mnemonic == "vudot" || Mnemonic == "vsdot")
     return Mnemonic;
 
   // First, split out any predication code. Ignore mnemonics we know aren't
   // predicated but do have a carry-set and so weren't caught above.
   if (Mnemonic != "adcs" && Mnemonic != "bics" && Mnemonic != "movs" &&
       Mnemonic != "muls" && Mnemonic != "smlals" && Mnemonic != "smulls" &&
       Mnemonic != "umlals" && Mnemonic != "umulls" && Mnemonic != "lsls" &&
       Mnemonic != "sbcs" && Mnemonic != "rscs") {
if (Mnemonic == "bkpt" || Mnemonic == "cbnz" || Mnemonic == "setend" ||
       Mnemonic.startswith("crc32") || Mnemonic.startswith("cps") ||
       Mnemonic.startswith("vsel") || Mnemonic == "vmaxnm" ||
       Mnemonic == "vminnm" || Mnemonic == "vcvta" || Mnemonic == "vcvtn" ||
       Mnemonic == "vcvtp" || Mnemonic == "vcvtm" || Mnemonic == "vrinta" ||
       Mnemonic == "vrintn" || Mnemonic == "vrintp" || Mnemonic == "vrintm" ||
       Mnemonic.startswith("aes") || Mnemonic == "hvc" || Mnemonic == "setpan" ||
       Mnemonic.startswith("sha1") || Mnemonic.startswith("sha256") ||
       (FullInst.startswith("vmull") && FullInst.endswith(".p64")) ||
-      Mnemonic == "vmovx" || Mnemonic == "vins") {
+      Mnemonic == "vmovx" || Mnemonic == "vins" ||
+      Mnemonic == "vudot" || Mnemonic == "vsdot") {
     // These mnemonics are never predicable
     CanAcceptPredicationCode = false;
   } else if (!isThumb()) {
     // Some instructions are only predicable in Thumb mode
     CanAcceptPredicationCode =
         Mnemonic != "cdp2" && Mnemonic != "clrex" && Mnemonic != "mcr2" &&
         Mnemonic != "mcrr2" && Mnemonic != "mrc2" && Mnemonic != "mrrc2" &&
         Mnemonic != "dmb" && Mnemonic != "dsb" && Mnemonic != "isb" &&

test/MC/ARM/armv8.2a-dotprod-a32.s

+// RUN: llvm-mc -triple arm -mattr=+dotprod -show-encoding < %s | FileCheck %s  --check-prefix=CHECK
+
+// RUN: not llvm-mc -triple arm -mattr=-dotprod -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-NO-DOTPROD < %t %s
+// RUN: not llvm-mc -triple arm -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-NO-DOTPROD < %t %s
+// RUN: not llvm-mc -triple arm -mattr=+v8.1a -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-NO-DOTPROD < %t %s
+// RUN: not llvm-mc -triple arm -mattr=+v8.2a -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-NO-DOTPROD < %t %s
+
+vudot.u8 d0, d1, d2
+vsdot.s8 d0, d1, d2
+vudot.u8 q0, q1, q4
+vsdot.s8 q0, q1, q4
+vudot.u8 d0, d1, d2[0]
+vsdot.s8 d0, d1, d2[1]
+vudot.u8 q0, q1, d4[0]
+vsdot.s8 q0, q1, d4[1]
+
+// CHECK: vudot.u8  d0, d1, d2      @ encoding: [0x12,0x0d,0x21,0xfc]
+// CHECK: vsdot.s8  d0, d1, d2      @ encoding: [0x02,0x0d,0x21,0xfc]
+// CHECK: vudot.u8  q0, q1, q4      @ encoding: [0x58,0x0d,0x22,0xfc]
+// CHECK: vsdot.s8  q0, q1, q4      @ encoding: [0x48,0x0d,0x22,0xfc]
+// CHECK: vudot.u8  d0, d1, d2[0]   @ encoding: [0x12,0x0d,0x21,0xfe]
+// CHECK: vsdot.s8  d0, d1, d2[1]   @ encoding: [0x22,0x0d,0x21,0xfe]
+// CHECK: vudot.u8  q0, q1, d4[0]   @ encoding: [0x54,0x0d,0x22,0xfe]
+// CHECK: vsdot.s8  q0, q1, d4[1]   @ encoding: [0x64,0x0d,0x22,0xfe]
+
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod
+// CHECK-NO-DOTPROD: error: instruction requires: dotprod

test/MC/ARM/armv8.2a-dotprod-error.s

+// RUN: not llvm-mc -triple arm -mattr=+dotprod -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+// RUN: not llvm-mc -triple thumb -mattr=+dotprod -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+
+vudot.u8 d0, d1, d2[2]
+vsdot.s8 d0, d1, d2[2]
+vudot.u8 q0, q1, d4[2]
+vsdot.s8 q0, q1, d4[2]
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction

test/MC/ARM/armv8.2a-dotprod-t32.s

+// RUN: llvm-mc -triple thumb -mattr=+dotprod -show-encoding < %s | FileCheck %s  --check-prefix=CHECK
+
+// RUN: not llvm-mc -triple thumb -mattr=-dotprod -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+// RUN: not llvm-mc -triple thumb -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+// RUN: not llvm-mc -triple thumb -mattr=+v8.1a -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+// RUN: not llvm-mc -triple thumb -mattr=+v8.2a -show-encoding < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-ERROR < %t %s
+
+  vudot.u8  d0, d1, d2
+  vsdot.s8  d0, d1, d2
+  vudot.u8  q0, q1, q4
+  vsdot.s8  q0, q1, q4
+  vudot.u8  d0, d1, d2[0]
+  vsdot.s8  d0, d1, d2[1]
+  vudot.u8  q0, q1, d4[0]
+  vsdot.s8  q0, q1, d4[1]
+
+//CHECK:  vudot.u8  d0, d1, d2      @ encoding: [0x21,0xfc,0x12,0x0d]
+//CHECK:  vsdot.s8  d0, d1, d2      @ encoding: [0x21,0xfc,0x02,0x0d]
+//CHECK:  vudot.u8  q0, q1, q4      @ encoding: [0x22,0xfc,0x58,0x0d]
+//CHECK:  vsdot.s8  q0, q1, q4      @ encoding: [0x22,0xfc,0x48,0x0d]
+//CHECK:  vudot.u8  d0, d1, d2[0]   @ encoding: [0x21,0xfe,0x12,0x0d]
+//CHECK:  vsdot.s8  d0, d1, d2[1]   @ encoding: [0x21,0xfe,0x22,0x0d]
+//CHECK:  vudot.u8  q0, q1, d4[0]   @ encoding: [0x22,0xfe,0x54,0x0d]
+//CHECK:  vsdot.s8  q0, q1, d4[1]   @ encoding: [0x22,0xfe,0x64,0x0d]
+
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+//CHECK-ERROR: error: instruction requires: dotprod
+