[SVE][Inline-Asm] Add constraints for SVE predicate registers

[kmclaughlin]

Summary

Adds the following inline asm constraints for SVE:

• Upl: One of the low eight SVE predicate registers, P0 to P7 inclusive
• Upa: SVE predicate register with full range, P0 to P15

[sdesmalen]

lib/IR/InlineAsm.cpp

188 ↗

(On Diff #216351)

"Expected a digit" seems more appropriate, since this code is only testing a single character.

192 ↗

(On Diff #216351)

StringRef can be used here instead of std::string.

lib/Target/AArch64/AArch64ISelLowering.cpp

5803 ↗

(On Diff #216351)

This is missing a check that Constraint.size() == 3. Or perhaps it is even better to split this code out into a separate function, because this patch writes out the condition three times. Maybe something like PredicateConstraint isPredicateConstraint(StringRef S), where PredicateConstraint is an enum.

[kmclaughlin]

• Added isPredicateConstraint function to AArch64ISelLowering.cpp, which returns Upl, Upa or Invalid. This is used to replace some repeated checks of the predicate type
• Minor changes to InlineAsm.cpp

[rovka]

Just some drive-by suggestions :)

lib/Target/AArch64/AArch64ISelLowering.cpp

5747 ↗

(On Diff #218071)

Nit: I think get- or parsePredicateConstraint reads better, since this doesn't return a simple yes/no answer.

test/CodeGen/AArch64/aarch64-sve-asm.ll

50 ↗

(On Diff #218071)

Nit: I would be a bit pedantic here and also check that they are used for the inline asm.

[greened]

docs/LangRef.rst

3818 ↗

(On Diff #218071)

What do these names mean? "<something> predicate lower|all?" I see they are the names used in gcc, so I guess it makes sense to use them here. Are these names used in the SVE architecture manual somewhere? I cannot find them.

lib/Target/AArch64/AArch64ISelLowering.cpp

5747 ↗

(On Diff #218071)

+1.

[kmclaughlin]

• Renamed the isPredicateConstraint function to parsePredicateConstraint
• Added more thorough checks to the tests in aarch64-sve-asm.ll

[kmclaughlin]

docs/LangRef.rst

3818 ↗

(On Diff #218071)

The length of the constraint depends on the first letter, and for AArch64 'U' was chosen to indicate a 3-letter constraint. These are not in the SVE architecture manual as they are only for compatibility with GCC inline asm (see https://gcc.gnu.org/onlinedocs/gcc/Machine-Constraints.html#Machine-Constraints)

lib/Target/AArch64/AArch64ISelLowering.cpp

5747 ↗

(On Diff #218071)

Thanks for the suggestion @rovka - I have changed this to parsePredicateConstraint

[rovka]

I think all the outstanding comments have been addressed. LGTM.

llvm/trunk/docs/LangRef.rst

- ``N``: An immediate integer for use with the ``MOV`` assembly alias on a
   64-bit register. This is a superset of ``L``.
 - ``Q``: Memory address operand must be in a single register (no
   offsets). (However, LLVM currently does this for the ``m`` constraint as
   well.)
 - ``r``: A 32 or 64-bit integer register (W* or X*).
 - ``w``: A 32, 64, or 128-bit floating-point, SIMD or SVE vector register.
 - ``x``: Like w, but restricted to registers 0 to 15 inclusive.
 - ``y``: Like w, but restricted to SVE vector registers Z0 to Z7 inclusive.
+- ``Upl``: One of the low eight SVE predicate registers (P0 to P7)
+- ``Upa``: Any of the SVE predicate registers (P0 to P15)
 
 AMDGPU:
 
 - ``r``: A 32 or 64-bit integer register.
 - ``[0-9]v``: The 32-bit VGPR register, number 0-9.
 - ``[0-9]s``: The 32-bit SGPR register, number 0-9.
 
 

llvm/trunk/lib/IR/InlineAsm.cpp

if (*I == '{') {   // Physical register reference.
       multipleAlternativeIndex++;
       pCodes = &multipleAlternatives[multipleAlternativeIndex].Codes;
       ++I;
     } else if (*I == '^') {
       // Multi-letter constraint
       // FIXME: For now assuming these are 2-character constraints.
       pCodes->push_back(StringRef(I+1, 2));
       I += 3;
+    } else if (*I == '@') {
+      // Multi-letter constraint
+      ++I;
+      unsigned char C = static_cast<unsigned char>(*I);
+      assert(isdigit(C) && "Expected a digit!");
+      int N = C - '0';
+      assert(N > 0 && "Found a zero letter constraint!");
+      ++I;
+      pCodes->push_back(StringRef(I, N));
+      I += N;
     } else {
       // Single letter constraint.
       pCodes->push_back(StringRef(I, 1));
       ++I;
     }
   }
 
   return false;

llvm/trunk/lib/Target/AArch64/AArch64AsmPrinter.cpp

if (MO.isReg()) {
     if (AArch64::GPR32allRegClass.contains(Reg) ||
         AArch64::GPR64allRegClass.contains(Reg))
       return printAsmMRegister(MO, 'x', O);
 
     unsigned AltName = AArch64::NoRegAltName;
     const TargetRegisterClass *RegClass;
     if (AArch64::ZPRRegClass.contains(Reg)) {
       RegClass = &AArch64::ZPRRegClass;
+    } else if (AArch64::PPRRegClass.contains(Reg)) {
+      RegClass = &AArch64::PPRRegClass;
     } else {
       RegClass = &AArch64::FPR128RegClass;
       AltName = AArch64::vreg;
     }
 
     // If this is a b, h, s, d, or q register, print it as a v register.
     return printAsmRegInClass(MO, RegClass, AltName, O);
   }

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.cpp

const char *AArch64TargetLowering::LowerXConstraint(EVT ConstraintVT) const {
   if (ConstraintVT.isVector() &&
      (ConstraintVT.getSizeInBits() == 64 ||
       ConstraintVT.getSizeInBits() == 128))
     return "w";
 
   return "r";
 }
 
+enum PredicateConstraint {
+  Upl,
+  Upa,
+  Invalid
+};
+
+PredicateConstraint parsePredicateConstraint(StringRef Constraint) {
+  PredicateConstraint P = PredicateConstraint::Invalid;
+  if (Constraint == "Upa")
+    P = PredicateConstraint::Upa;
+  if (Constraint == "Upl")
+    P = PredicateConstraint::Upl;
+  return P;
+}
+
 /// getConstraintType - Given a constraint letter, return the type of
 /// constraint it is for this target.
 AArch64TargetLowering::ConstraintType
 AArch64TargetLowering::getConstraintType(StringRef Constraint) const {
   if (Constraint.size() == 1) {
     switch (Constraint[0]) {
     default:
       break;
if (Constraint.size() == 1) {
     case 'N':
     case 'Y':
     case 'Z':
       return C_Immediate;
     case 'z':
     case 'S': // A symbolic address
       return C_Other;
     }
-  }
+  } else if (parsePredicateConstraint(Constraint) !=
+             PredicateConstraint::Invalid)
+      return C_RegisterClass;
   return TargetLowering::getConstraintType(Constraint);
 }
 
 /// Examine constraint type and operand type and determine a weight value.
 /// This object must already have been set up with the operand type
 /// and the current alternative constraint selected.
 TargetLowering::ConstraintWeight
 AArch64TargetLowering::getSingleConstraintMatchWeight(
AArch64TargetLowering::getSingleConstraintMatchWeight(
   case 'w':
   case 'y':
     if (type->isFloatingPointTy() || type->isVectorTy())
       weight = CW_Register;
     break;
   case 'z':
     weight = CW_Constant;
     break;
+  case 'U':
+    if (parsePredicateConstraint(constraint) != PredicateConstraint::Invalid)
+      weight = CW_Register;
+    break;
   }
   return weight;
 }
 
 std::pair<unsigned, const TargetRegisterClass *>
 AArch64TargetLowering::getRegForInlineAsmConstraint(
     const TargetRegisterInfo *TRI, StringRef Constraint, MVT VT) const {
   if (Constraint.size() == 1) {
case 'x':
       break;
     case 'y':
       if (!Subtarget->hasFPARMv8())
         break;
       if (VT.isScalableVector())
         return std::make_pair(0U, &AArch64::ZPR_3bRegClass);
       break;
     }
+  } else {
+    PredicateConstraint PC = parsePredicateConstraint(Constraint);
+    if (PC != PredicateConstraint::Invalid) {
+      assert(VT.isScalableVector());
+      bool restricted = (PC == PredicateConstraint::Upl);
+      return restricted ? std::make_pair(0U, &AArch64::PPR_3bRegClass)
+                          : std::make_pair(0U, &AArch64::PPRRegClass);
+    }
   }
   if (StringRef("{cc}").equals_lower(Constraint))
     return std::make_pair(unsigned(AArch64::NZCV), &AArch64::CCRRegClass);
 
   // Use the default implementation in TargetLowering to convert the register
   // constraint into a member of a register class.
   std::pair<unsigned, const TargetRegisterClass *> Res;
   Res = TargetLowering::getRegForInlineAsmConstraint(TRI, Constraint, VT);

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.cpp

if (DestReg == AArch64::WSP || SrcReg == AArch64::WSP) {
         BuildMI(MBB, I, DL, get(AArch64::ORRWrr), DestReg)
             .addReg(AArch64::WZR)
             .addReg(SrcReg, getKillRegState(KillSrc));
       }
     }
     return;
   }
 
+  // Copy a Predicate register by ORRing with itself.
+  if (AArch64::PPRRegClass.contains(DestReg) &&
+      AArch64::PPRRegClass.contains(SrcReg)) {
+    assert(Subtarget.hasSVE() && "Unexpected SVE register.");
+    BuildMI(MBB, I, DL, get(AArch64::ORR_PPzPP), DestReg)
+      .addReg(SrcReg) // Pg
+      .addReg(SrcReg)
+      .addReg(SrcReg, getKillRegState(KillSrc));
+    return;
+  }
+
   // Copy a Z register by ORRing with itself.
   if (AArch64::ZPRRegClass.contains(DestReg) &&
       AArch64::ZPRRegClass.contains(SrcReg)) {
     assert(Subtarget.hasSVE() && "Unexpected SVE register.");
     BuildMI(MBB, I, DL, get(AArch64::ORR_ZZZ), DestReg)
       .addReg(SrcReg)
       .addReg(SrcReg, getKillRegState(KillSrc));
     return;

llvm/trunk/test/CodeGen/AArch64/aarch64-sve-asm.ll

 ; RUN: llc < %s -mtriple aarch64-none-linux-gnu -mattr=+sve -stop-after=finalize-isel | FileCheck %s --check-prefix=CHECK
 
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64-none-linux-gnu"
 
 ; Function Attrs: nounwind readnone
 ; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z1
 ; CHECK: [[ARG2:%[0-9]+]]:zpr = COPY $z0
 ; CHECK: [[ARG3:%[0-9]+]]:zpr = COPY [[ARG2]]
 ; CHECK: [[ARG4:%[0-9]+]]:zpr_3b = COPY [[ARG1]]
+; CHECK: INLINEASM {{.*}} [[ARG4]]
 define <vscale x 16 x i8> @test_svadd_i8(<vscale x 16 x i8> %Zn, <vscale x 16 x i8> %Zm) {
   %1 = tail call <vscale x 16 x i8> asm "add $0.b, $1.b, $2.b", "=w,w,y"(<vscale x 16 x i8> %Zn, <vscale x 16 x i8> %Zm)
   ret <vscale x 16 x i8> %1
 }
 
 ; Function Attrs: nounwind readnone
 ; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z1
 ; CHECK: [[ARG2:%[0-9]+]]:zpr = COPY $z0
 ; CHECK: [[ARG3:%[0-9]+]]:zpr = COPY [[ARG2]]
 ; CHECK: [[ARG4:%[0-9]+]]:zpr_4b = COPY [[ARG1]]
+; CHECK: INLINEASM {{.*}} [[ARG4]]
 define <vscale x 2 x i64> @test_svsub_i64(<vscale x 2 x i64> %Zn, <vscale x 2 x i64> %Zm) {
   %1 = tail call <vscale x 2 x i64> asm "sub $0.d, $1.d, $2.d", "=w,w,x"(<vscale x 2 x i64> %Zn, <vscale x 2 x i64> %Zm)
   ret <vscale x 2 x i64> %1
 }
 
 ; Function Attrs: nounwind readnone
 ; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z1
 ; CHECK: [[ARG2:%[0-9]+]]:zpr = COPY $z0
 ; CHECK: [[ARG3:%[0-9]+]]:zpr = COPY [[ARG2]]
 ; CHECK: [[ARG4:%[0-9]+]]:zpr_3b = COPY [[ARG1]]
+; CHECK: INLINEASM {{.*}} [[ARG4]]
 define <vscale x 8 x half> @test_svfmul_f16(<vscale x 8 x half> %Zn, <vscale x 8 x half> %Zm) {
   %1 = tail call <vscale x 8 x half> asm "fmul $0.h, $1.h, $2.h", "=w,w,y"(<vscale x 8 x half> %Zn, <vscale x 8 x half> %Zm)
   ret <vscale x 8 x half> %1
 }
 
 ; Function Attrs: nounwind readnone
 ; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z1
 ; CHECK: [[ARG2:%[0-9]+]]:zpr = COPY $z0
 ; CHECK: [[ARG3:%[0-9]+]]:zpr = COPY [[ARG2]]
 ; CHECK: [[ARG4:%[0-9]+]]:zpr_4b = COPY [[ARG1]]
+; CHECK: INLINEASM {{.*}} [[ARG4]]
 define <vscale x 4 x float> @test_svfmul_f(<vscale x 4 x float> %Zn, <vscale x 4 x float> %Zm) {
   %1 = tail call <vscale x 4 x float> asm "fmul $0.s, $1.s, $2.s", "=w,w,x"(<vscale x 4 x float> %Zn, <vscale x 4 x float> %Zm)
   ret <vscale x 4 x float> %1
 }
+
+; Function Attrs: nounwind readnone
+; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z1
+; CHECK: [[ARG2:%[0-9]+]]:zpr = COPY $z0
+; CHECK: [[ARG3:%[0-9]+]]:ppr = COPY $p0
+; CHECK: [[ARG4:%[0-9]+]]:ppr_3b = COPY [[ARG3]]
+; CHECK: INLINEASM {{.*}} [[ARG4]]
+define <vscale x 8 x half> @test_svfadd_f16(<vscale x 16 x i1> %Pg, <vscale x 8 x half> %Zn, <vscale x 8 x half> %Zm) {
+  %1 = tail call <vscale x 8 x half> asm "fadd $0.h, $1/m, $2.h, $3.h", "=w,@3Upl,w,w"(<vscale x 16 x i1> %Pg, <vscale x 8 x half> %Zn, <vscale x 8 x half> %Zm)
+  ret <vscale x 8 x half> %1
+}
+
+; Function Attrs: nounwind readnone
+; CHECK: [[ARG1:%[0-9]+]]:zpr = COPY $z0
+; CHECK: [[ARG2:%[0-9]+]]:ppr = COPY $p0
+; CHECK: [[ARG3:%[0-9]+]]:ppr = COPY [[ARG2]]
+; CHECK: [[ARG4:%[0-9]+]]:zpr = COPY [[ARG1]]
+; CHECK: INLINEASM {{.*}} [[ARG3]]
+define <vscale x 4 x i32> @test_incp(<vscale x 16 x i1> %Pg, <vscale x 4 x i32> %Zn) {
+  %1 = tail call <vscale x 4 x i32> asm "incp $0.s, $1", "=w,@3Upa,0"(<vscale x 16 x i1> %Pg, <vscale x 4 x i32> %Zn)
+  ret <vscale x 4 x i32> %1
+}