[AArch64] Add pipeline model for Exynos M3

[evandro]

Summary

Add the scheduling and cost model for Exynos M3.

[evandro]

Ping! 🔔

[MatzeB]

I can't say much about the scheduling model itself as I'm not familiar with Exynos.

LGTM if you introduce a subtarget feature for the isCheapAsAMove behavior (see below).

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

678–679 ↗

(On Diff #130931)

Looks like it is time to make a subtarget feature out of this.

llvm/test/CodeGen/AArch64/misched-fusion-aes.ll

1–9 ↗

(On Diff #130931)

More candidates here where at some point we better start testing -mattr=+fuse-aes / -mattr=-fuse-aes and have a seperate test to check target feature lists of CPUs...

(but as I noted in my other review that is work for another time/commit).

[MatzeB]

Actually can you rename the scheduling model to AArch64SchedExynosM3.td? (Feel free to do the same with the M1 file in a separate commit without review).

[evandro]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

678–679 ↗

(On Diff #130931)

I agree in principle, but there already is FeatureCustomCheapAsMoveHandling, so I'm afraid that it might be a bit more involved and that it perhaps deserves a separate discussion. Basically, these checks customize the custom handling of isAsCheapAsAMove because Exynos is different from the other AArch64 targets.

[evandro]

In D42387#990879, @MatzeB wrote:

Actually can you rename the scheduling model to AArch64SchedExynosM3.td? (Feel free to do the same with the M1 file in a separate commit without review).

The filename change for M1 was done in rL323686.

[MatzeB]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

678–679 ↗

(On Diff #130931)

On a general level: I really think we should push towards not using getProcFamily() outside AArch64Subtarget.cpp; It's a sign that things aren't modeled cleanly and it'll only get worse with new exception added. And while I wouldn't mind a case like this in isolation, in practice people copy the patterns they see elsewhere and I don't want this to be the copy&paste source for more getProcFamily() calls in the target.

Adding FeatureExynosCheapAsMoveHandling (or a better name if you have one) and using that here shouldn't be that hard. It's unfortunate that it would only kick in when CustomCheapAsMoveHandling is enabled too, but not a dealbreaker or reason not introduce the subtarget feature here.

[evandro]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

678–679 ↗

(On Diff #130931)

I had to point it out, but, if it sounds sensible to you, I do not object to it.

llvm/trunk/lib/Target/AArch64/AArch64.td

 // AArch64 Processors supported.
 //
 include "AArch64SchedA53.td"
 include "AArch64SchedA57.td"
 include "AArch64SchedCyclone.td"
 include "AArch64SchedFalkor.td"
 include "AArch64SchedKryo.td"
 include "AArch64SchedExynosM1.td"
+include "AArch64SchedExynosM3.td"
 include "AArch64SchedThunderX.td"
 include "AArch64SchedThunderX2T99.td"
 
 def ProcA35     : SubtargetFeature<"a35", "ARMProcFamily", "CortexA35",
                                    "Cortex-A35 ARM processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureFPARMv8,
def ProcExynosM2 : SubtargetFeature<"exynosm2", "ARMProcFamily", "ExynosM1",
                                      FeatureFPARMv8,
                                      FeatureFuseAES,
                                      FeatureNEON,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
                                      FeatureSlowMisaligned128Store,
                                      FeatureZCZeroing]>;
 
-def ProcExynosM3 : SubtargetFeature<"exynosm3", "ARMProcFamily", "ExynosM1",
+def ProcExynosM3 : SubtargetFeature<"exynosm3", "ARMProcFamily", "ExynosM3",
                                     "Samsung Exynos-M3 processors",
                                     [FeatureCRC,
                                      FeatureCrypto,
                                      FeatureCustomCheapAsMoveHandling,
                                      FeatureFPARMv8,
                                      FeatureFuseAES,
                                      FeatureFuseLiterals,
+                                     FeatureLSLFast,
                                      FeatureNEON,
                                      FeaturePerfMon,
                                      FeaturePostRAScheduler,
-                                     FeatureSlowMisaligned128Store,
-                                     FeatureSlowPaired128,
+                                     FeaturePredictableSelectIsExpensive,
                                      FeatureZCZeroing]>;
 
 def ProcKryo    : SubtargetFeature<"kryo", "ARMProcFamily", "Kryo",
                                    "Qualcomm Kryo processors", [
                                    FeatureCRC,
                                    FeatureCrypto,
                                    FeatureCustomCheapAsMoveHandling,
                                    FeatureFPARMv8,

 def : ProcessorModel<"cortex-a57", CortexA57Model, [ProcA57]>;
 // FIXME: Cortex-A72, Cortex-A73 and Cortex-A75 are currently modeled as a Cortex-A57.
 def : ProcessorModel<"cortex-a72", CortexA57Model, [ProcA72]>;
 def : ProcessorModel<"cortex-a73", CortexA57Model, [ProcA73]>;
 def : ProcessorModel<"cortex-a75", CortexA57Model, [ProcA75]>;
 def : ProcessorModel<"cyclone", CycloneModel, [ProcCyclone]>;
 def : ProcessorModel<"exynos-m1", ExynosM1Model, [ProcExynosM1]>;
 def : ProcessorModel<"exynos-m2", ExynosM1Model, [ProcExynosM2]>;
-def : ProcessorModel<"exynos-m3", ExynosM1Model, [ProcExynosM3]>;
+def : ProcessorModel<"exynos-m3", ExynosM3Model, [ProcExynosM3]>;
 def : ProcessorModel<"falkor", FalkorModel, [ProcFalkor]>;
 def : ProcessorModel<"saphira", FalkorModel, [ProcSaphira]>;
 def : ProcessorModel<"kryo", KryoModel, [ProcKryo]>;
 // Cavium ThunderX/ThunderX T8X  Processors
 def : ProcessorModel<"thunderx", ThunderXT8XModel,  [ProcThunderX]>;
 def : ProcessorModel<"thunderxt88", ThunderXT8XModel,  [ProcThunderXT88]>;
 def : ProcessorModel<"thunderxt81", ThunderXT8XModel,  [ProcThunderXT81]>;
 def : ProcessorModel<"thunderxt83", ThunderXT8XModel,  [ProcThunderXT83]>;

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.h

public:
   void insertOutlinerEpilogue(MachineBasicBlock &MBB, MachineFunction &MF,
                               const MachineOutlinerInfo &MInfo) const override;
   void insertOutlinerPrologue(MachineBasicBlock &MBB, MachineFunction &MF,
                               const MachineOutlinerInfo &MInfo) const override;
   MachineBasicBlock::iterator
   insertOutlinedCall(Module &M, MachineBasicBlock &MBB,
                      MachineBasicBlock::iterator &It, MachineFunction &MF,
                      const MachineOutlinerInfo &MInfo) const override;
+  /// Returns true if the instruction sets to an immediate value that can be
+  /// executed more efficiently.
+  bool isExynosResetFast(const MachineInstr &MI) const;
   /// Returns true if the instruction has a shift left that can be executed
   /// more efficiently.
   bool isExynosShiftLeftFast(const MachineInstr &MI) const;
   /// Returns true if the instruction has a shift by immediate that can be
   /// executed in one cycle less.
   bool isFalkorShiftExtFast(const MachineInstr &MI) const;
 
 private:

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.cpp

static bool canBeExpandedToORR(const MachineInstr &MI, unsigned BitSize) {
   return AArch64_AM::processLogicalImmediate(UImm, BitSize, Encoding);
 }
 
 // FIXME: this implementation should be micro-architecture dependent, so a
 // micro-architecture target hook should be introduced here in future.
 bool AArch64InstrInfo::isAsCheapAsAMove(const MachineInstr &MI) const {
   if (!Subtarget.hasCustomCheapAsMoveHandling())
     return MI.isAsCheapAsAMove();
-  if (Subtarget.getProcFamily() == AArch64Subtarget::ExynosM1 &&
-      isExynosShiftLeftFast(MI))
+
+  if (Subtarget.getProcFamily() == AArch64Subtarget::ExynosM1 ||
+      Subtarget.getProcFamily() == AArch64Subtarget::ExynosM3) {
+    if (isExynosResetFast(MI) || isExynosShiftLeftFast(MI))
       return true;
+    else
+      return MI.isAsCheapAsAMove();
+  }
 
   switch (MI.getOpcode()) {
   default:
     return false;
 
   // add/sub on register without shift
   case AArch64::ADDWri:
   case AArch64::ADDXri:
case TargetOpcode::COPY:
     return (Subtarget.hasZeroCycleZeroing() &&
             (MI.getOperand(1).getReg() == AArch64::WZR ||
              MI.getOperand(1).getReg() == AArch64::XZR));
   }
 
   llvm_unreachable("Unknown opcode to check as cheap as a move!");
 }
 
+bool AArch64InstrInfo::isExynosResetFast(const MachineInstr &MI) const {
+  switch (MI.getOpcode()) {
+  default:
+    return false;
+
+  case AArch64::ADR:
+  case AArch64::ADRP:
+
+  case AArch64::MOVNWi:
+  case AArch64::MOVNXi:
+  case AArch64::MOVZWi:
+  case AArch64::MOVZXi:
+    return true;
+
+  case AArch64::MOVID:
+  case AArch64::MOVIv2d_ns:
+  case AArch64::MOVIv8b_ns:
+  case AArch64::MOVIv16b_ns:
+    return (MI.getOperand(1).getImm() == 0);
+
+  case AArch64::MOVIv2i32:
+  case AArch64::MOVIv4i32:
+  case AArch64::MOVIv4i16:
+  case AArch64::MOVIv8i16:
+    return (MI.getOperand(1).getImm() == 0 &&
+            MI.getOperand(2).getImm() == 0);
+  }
+}
+
 bool AArch64InstrInfo::isExynosShiftLeftFast(const MachineInstr &MI) const {
   unsigned Imm, Shift;
   AArch64_AM::ShiftExtendType Ext;
 
   switch (MI.getOpcode()) {
   default:
     return false;
 

llvm/trunk/lib/Target/AArch64/AArch64SchedExynosM3.td

+//=- AArch64SchedExynosM3.td - Samsung Exynos M3 Sched Defs --*- tablegen -*-=//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the machine model for the Samsung Exynos M3 to support
+// instruction scheduling and other instruction cost heuristics.
+//
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// The Exynos-M3 is an advanced superscalar microprocessor with a 6-wide
+// in-order stage for decode and dispatch and a wider issue stage.
+// The execution units and loads and stores are out-of-order.
+
+def ExynosM3Model : SchedMachineModel {
+  let IssueWidth            =   6; // Up to 6 uops per cycle.
+  let MicroOpBufferSize     = 228; // ROB size.
+  let LoopMicroOpBufferSize =  40; // Based on the instruction queue size.
+  let LoadLatency           =   4; // Optimistic load cases.
+  let MispredictPenalty     =  16; // Minimum branch misprediction penalty.
+  let CompleteModel         =   1; // Use the default model otherwise.
+
+  list<Predicate> UnsupportedFeatures = [HasSVE];
+}
+
+//===----------------------------------------------------------------------===//
+// Define each kind of processor resource and number available on the Exynos-M3,
+// which has 12 pipelines, each with its own queue with out-of-order dispatch.
+
+def M3UnitA  : ProcResource<2>; // Simple integer
+def M3UnitC  : ProcResource<2>; // Simple and complex integer
+def M3UnitD  : ProcResource<2>; // Integer division (inside C, serialized)
+def M3UnitB  : ProcResource<2>; // Branch
+def M3UnitL  : ProcResource<2>; // Load
+def M3UnitS  : ProcResource<1>; // Store
+def M3PipeF0 : ProcResource<1>; // FP #0
+let Super = M3PipeF0 in {
+  def M3UnitFMAC0 : ProcResource<1>; // FP multiplication
+  def M3UnitFADD0 : ProcResource<1>; // Simple FP
+  def M3UnitFCVT0 : ProcResource<1>; // FP conversion
+  def M3UnitFSQR  : ProcResource<2>; // FP square root (serialized)
+  def M3UnitNALU0 : ProcResource<1>; // Simple vector
+  def M3UnitNMSC  : ProcResource<1>; // FP and vector miscellanea
+  def M3UnitNSHT0 : ProcResource<1>; // Vector shifting
+  def M3UnitNSHF0 : ProcResource<1>; // Vector shuffling
+}
+def M3PipeF1 : ProcResource<1>; // FP #1
+let Super = M3PipeF1 in {
+  def M3UnitFMAC1 : ProcResource<1>; // FP multiplication
+  def M3UnitFADD1 : ProcResource<1>; // Simple FP
+  def M3UnitFDIV0 : ProcResource<2>; // FP division (serialized)
+  def M3UnitFCVT1 : ProcResource<1>; // FP conversion
+  def M3UnitFST0  : ProcResource<1>; // FP store
+  def M3UnitNALU1 : ProcResource<1>; // Simple vector
+  def M3UnitNCRY0 : ProcResource<1>; // Cryptographic
+  def M3UnitNMUL  : ProcResource<1>; // Vector multiplication
+  def M3UnitNSHT1 : ProcResource<1>; // Vector shifting
+  def M3UnitNSHF1 : ProcResource<1>; // Vector shuffling
+}
+def M3PipeF2 : ProcResource<1>; // FP #2
+let Super = M3PipeF2 in {
+  def M3UnitFMAC2 : ProcResource<1>; // FP multiplication
+  def M3UnitFADD2 : ProcResource<1>; // Simple FP
+  def M3UnitFDIV1 : ProcResource<2>; // FP division (serialized)
+  def M3UnitFST1  : ProcResource<1>; // FP store
+  def M3UnitNALU2 : ProcResource<1>; // Simple vector
+  def M3UnitNCRY1 : ProcResource<1>; // Cryptographic
+  def M3UnitNSHT2 : ProcResource<1>; // Vector shifting
+  def M3UnitNSHF2 : ProcResource<1>; // Vector shuffling
+}
+
+let SchedModel = ExynosM3Model in {
+
+  def M3UnitALU  : ProcResGroup<[M3UnitA,
+                                 M3UnitC]>;
+  def M3UnitFMAC : ProcResGroup<[M3UnitFMAC0,
+                                 M3UnitFMAC1,
+                                 M3UnitFMAC2]>;
+  def M3UnitFADD : ProcResGroup<[M3UnitFADD0,
+                                 M3UnitFADD1,
+                                 M3UnitFADD2]>;
+  def M3UnitFDIV : ProcResGroup<[M3UnitFDIV0,
+                                 M3UnitFDIV1]>;
+  def M3UnitFCVT : ProcResGroup<[M3UnitFCVT0,
+                                 M3UnitFCVT1]>;
+  def M3UnitFST  : ProcResGroup<[M3UnitFST0,
+                                 M3UnitFST1]>;
+  def M3UnitNALU : ProcResGroup<[M3UnitNALU0,
+                                 M3UnitNALU1,
+                                 M3UnitNALU2]>;
+  def M3UnitNCRY : ProcResGroup<[M3UnitNCRY0,
+                                 M3UnitNCRY1]>;
+  def M3UnitNSHT : ProcResGroup<[M3UnitNSHT0,
+                                 M3UnitNSHT1,
+                                 M3UnitNSHT2]>;
+  def M3UnitNSHF : ProcResGroup<[M3UnitNSHF0,
+                                 M3UnitNSHF1,
+                                 M3UnitNSHF2]>;
+}
+
+let SchedModel = ExynosM3Model in {
+
+//===----------------------------------------------------------------------===//
+// Predicates.
+
+def M3BranchLinkFastPred : SchedPredicate<[{MI->getOpcode() == AArch64::BLR &&
+                                            MI->getOperand(0).getReg() != AArch64::LR}]>;
+def M3ResetFastPred      : SchedPredicate<[{TII->isExynosResetFast(*MI)}]>;
+def M3ShiftLeftFastPred  : SchedPredicate<[{TII->isExynosShiftLeftFast(*MI)}]>;
+
+//===----------------------------------------------------------------------===//
+// Coarse scheduling model.
+
+def M3WriteZ0 : SchedWriteRes<[]> { let Latency = 0;
+                                    let NumMicroOps = 1; }
+
+def M3WriteA1 : SchedWriteRes<[M3UnitALU]> { let Latency = 1; }
+def M3WriteAA : SchedWriteRes<[M3UnitALU]> { let Latency = 2;
+                                             let ResourceCycles = [2]; }
+def M3WriteAB : SchedWriteRes<[M3UnitALU,
+                               M3UnitC]>   { let Latency = 1;
+                                             let NumMicroOps = 2; }
+def M3WriteAC : SchedWriteRes<[M3UnitALU,
+                               M3UnitALU,
+                               M3UnitC]>   { let Latency = 2;
+                                             let NumMicroOps = 3; }
+def M3WriteAD : SchedWriteRes<[M3UnitALU,
+                               M3UnitC]>   { let Latency = 2;
+                                             let NumMicroOps = 2; }
+def M3WriteC1 : SchedWriteRes<[M3UnitC]>   { let Latency = 1; }
+def M3WriteC2 : SchedWriteRes<[M3UnitC]>   { let Latency = 2; }
+def M3WriteAX : SchedWriteVariant<[SchedVar<M3ShiftLeftFastPred, [M3WriteA1]>,
+                                   SchedVar<NoSchedPred,         [M3WriteAA]>]>;
+
+def M3WriteB1 : SchedWriteRes<[M3UnitB]> { let Latency = 1; }
+def M3WriteBX : SchedWriteVariant<[SchedVar<M3BranchLinkFastPred, [M3WriteAB]>,
+                                   SchedVar<NoSchedPred,          [M3WriteAC]>]>;
+
+def M3WriteL4 : SchedWriteRes<[M3UnitL]> { let Latency = 4; }
+def M3WriteL5 : SchedWriteRes<[M3UnitL]> { let Latency = 5; }
+def M3WriteLA : SchedWriteRes<[M3UnitL,
+                               M3UnitL]> { let Latency = 5;
+                                           let NumMicroOps = 1; }
+def M3WriteLB : SchedWriteRes<[M3UnitA,
+                               M3UnitL]> { let Latency = 5;
+                                           let NumMicroOps = 2; }
+def M3WriteLC : SchedWriteRes<[M3UnitA,
+                               M3UnitL,
+                               M3UnitL]> { let Latency = 5;
+                                           let NumMicroOps = 2; }
+def M3WriteLD : SchedWriteRes<[M3UnitA,
+                               M3UnitL]> { let Latency = 4;
+                                           let NumMicroOps = 2; }
+def M3WriteLH : SchedWriteRes<[]>        { let Latency = 5;
+                                           let NumMicroOps = 0; }
+
+def M3WriteLX : SchedWriteVariant<[SchedVar<M3ShiftLeftFastPred, [M3WriteL5]>,
+                                   SchedVar<NoSchedPred,         [M3WriteLB]>]>;
+
+def M3WriteS1 : SchedWriteRes<[M3UnitS]>   { let Latency = 1; }
+def M3WriteSA : SchedWriteRes<[M3UnitA,
+                               M3UnitS,
+                               M3UnitFST]> { let Latency = 2;
+                                             let NumMicroOps = 2; }
+def M3WriteSB : SchedWriteRes<[M3UnitA,
+                               M3UnitS]>   { let Latency = 1;
+                                             let NumMicroOps = 2; }
+def M3WriteSC : SchedWriteRes<[M3UnitA,
+                               M3UnitS]>   { let Latency = 2;
+                                             let NumMicroOps = 2; }
+
+def M3WriteSX : SchedWriteVariant<[SchedVar<M3ShiftLeftFastPred, [M3WriteS1]>,
+                                   SchedVar<NoSchedPred,         [M3WriteSB]>]>;
+def M3WriteSY : SchedWriteVariant<[SchedVar<M3ShiftLeftFastPred, [M3WriteS1]>,
+                                   SchedVar<NoSchedPred,         [M3WriteSC]>]>;
+
+def M3ReadAdrBase : SchedReadVariant<[SchedVar<ScaledIdxPred, [ReadDefault]>,
+                                      SchedVar<NoSchedPred,   [ReadDefault]>]>;
+
+// Branch instructions.
+def : SchedAlias<WriteBr, M3WriteZ0>;
+def : WriteRes<WriteBrReg, [M3UnitC]> { let Latency = 1; }
+
+// Arithmetic and logical integer instructions.
+def : WriteRes<WriteI,     [M3UnitALU]> { let Latency = 1; }
+def : WriteRes<WriteISReg, [M3UnitALU]> { let Latency = 1; }
+def : WriteRes<WriteIEReg, [M3UnitALU]> { let Latency = 1; }
+def : WriteRes<WriteIS,    [M3UnitALU]> { let Latency = 1; }
+
+// Move instructions.
+def : WriteRes<WriteImm, [M3UnitALU]> { let Latency = 1; }
+
+// Divide and multiply instructions.
+def : WriteRes<WriteID32, [M3UnitC,
+                           M3UnitD]>  { let Latency = 12;
+                                        let ResourceCycles = [1, 12]; }
+def : WriteRes<WriteID64, [M3UnitC,
+                           M3UnitD]>  { let Latency = 12;
+                                        let ResourceCycles = [1, 12]; }
+def : WriteRes<WriteIM32, [M3UnitC]>  { let Latency = 3; }
+def : WriteRes<WriteIM64, [M3UnitC]>  { let Latency = 4;
+                                        let ResourceCycles = [2]; }
+
+// Miscellaneous instructions.
+def : WriteRes<WriteExtr, [M3UnitALU,
+                           M3UnitALU]> { let Latency = 1;
+                                         let NumMicroOps = 2; }
+
+// Addressing modes.
+def : WriteRes<WriteAdr, []> { let Latency = 1;
+                               let NumMicroOps = 0; }
+def : SchedAlias<ReadAdrBase, M3ReadAdrBase>;
+
+// Load instructions.
+def : SchedAlias<WriteLD, M3WriteL4>;
+def : WriteRes<WriteLDHi, []> { let Latency = 4;
+                                let NumMicroOps = 0; }
+def : SchedAlias<WriteLDIdx, M3WriteLX>;
+
+// Store instructions.
+def : SchedAlias<WriteST,    M3WriteS1>;
+def : SchedAlias<WriteSTP,   M3WriteS1>;
+def : SchedAlias<WriteSTX,   M3WriteS1>;
+def : SchedAlias<WriteSTIdx, M3WriteSX>;
+
+// FP data instructions.
+def : WriteRes<WriteF,    [M3UnitFADD]>  { let Latency = 2; }
+def : WriteRes<WriteFCmp, [M3UnitNMSC]>  { let Latency = 2; }
+def : WriteRes<WriteFDiv, [M3UnitFDIV]>  { let Latency = 12;
+                                           let ResourceCycles = [12]; }
+def : WriteRes<WriteFMul, [M3UnitFMAC]>  { let Latency = 4; }
+
+// FP miscellaneous instructions.
+// TODO: Conversion between register files is much different.
+def : WriteRes<WriteFCvt,  [M3UnitFCVT]> { let Latency = 3; }
+def : WriteRes<WriteFImm,  [M3UnitNALU]> { let Latency = 1; }
+def : WriteRes<WriteFCopy, [M3UnitNALU]> { let Latency = 1; }
+
+// FP load instructions.
+def : SchedAlias<WriteVLD, M3WriteL5>;
+
+// FP store instructions.
+def : WriteRes<WriteVST, [M3UnitS,
+                          M3UnitFST]> { let Latency = 1;
+                                        let NumMicroOps = 1; }
+
+// ASIMD FP instructions.
+def : WriteRes<WriteV, [M3UnitNALU]> { let Latency = 3; }
+
+// Other miscellaneous instructions.
+def : WriteRes<WriteAtomic,  []> { let Unsupported = 1; }
+def : WriteRes<WriteBarrier, []> { let Latency = 1; }
+def : WriteRes<WriteHint,    []> { let Latency = 1; }
+def : WriteRes<WriteSys,     []> { let Latency = 1; }
+
+//===----------------------------------------------------------------------===//
+// Generic fast forwarding.
+
+// TODO: Add FP register forwarding rules.
+
+def : ReadAdvance<ReadI,       0>;
+def : ReadAdvance<ReadISReg,   0>;
+def : ReadAdvance<ReadIEReg,   0>;
+def : ReadAdvance<ReadIM,      0>;
+// TODO: The forwarding for 32 bits actually saves 2 cycles.
+def : ReadAdvance<ReadIMA,     3, [WriteIM32, WriteIM64]>;
+def : ReadAdvance<ReadID,      0>;
+def : ReadAdvance<ReadExtrHi,  0>;
+def : ReadAdvance<ReadAdrBase, 0>;
+def : ReadAdvance<ReadVLD,     0>;
+
+//===----------------------------------------------------------------------===//
+// Finer scheduling model.
+
+def M3WriteNEONA   : SchedWriteRes<[M3UnitNSHF,
+                                    M3UnitFADD]>  { let Latency = 3;
+                                                    let NumMicroOps = 2; }
+def M3WriteNEONB   : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitFST]>   { let Latency = 10;
+                                                    let NumMicroOps = 2; }
+def M3WriteNEOND   : SchedWriteRes<[M3UnitNSHF,
+                                    M3UnitFST]>   { let Latency = 6;
+                                                    let NumMicroOps = 2; }
+def M3WriteNEONH   : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitS]>     { let Latency = 5;
+                                                    let NumMicroOps = 2; }
+def M3WriteNEONV   : SchedWriteRes<[M3UnitFDIV,
+                                    M3UnitFDIV]>  { let Latency = 7;
+                                                    let NumMicroOps = 1;
+                                                    let ResourceCycles = [8]; }
+def M3WriteNEONW   : SchedWriteRes<[M3UnitFDIV,
+                                    M3UnitFDIV]>  { let Latency = 12;
+                                                    let NumMicroOps = 1;
+                                                    let ResourceCycles = [13]; }
+def M3WriteNEONX   : SchedWriteRes<[M3UnitFSQR,
+                                    M3UnitFSQR]>  { let Latency = 18;
+                                                    let NumMicroOps = 1;
+                                                    let ResourceCycles = [19]; }
+def M3WriteNEONY   : SchedWriteRes<[M3UnitFSQR,
+                                    M3UnitFSQR]>  { let Latency = 25;
+                                                    let NumMicroOps = 1;
+                                                    let ResourceCycles = [26]; }
+def M3WriteNEONZ   : SchedWriteRes<[M3UnitNMSC,
+                                    M3UnitNMSC]>  { let Latency = 4;
+                                                    let NumMicroOps = 2; }
+def M3WriteFADD2   : SchedWriteRes<[M3UnitFADD]>  { let Latency = 2; }
+def M3WriteFCVT2   : SchedWriteRes<[M3UnitFCVT]>  { let Latency = 2; }
+def M3WriteFCVT3   : SchedWriteRes<[M3UnitFCVT]>  { let Latency = 3; }
+def M3WriteFCVT3A  : SchedWriteRes<[M3UnitFCVT0]> { let Latency = 3; }
+def M3WriteFCVT4A  : SchedWriteRes<[M3UnitFCVT0]> { let Latency = 4; }
+def M3WriteFCVT4   : SchedWriteRes<[M3UnitFCVT]>  { let Latency = 4; }
+def M3WriteFDIV10  : SchedWriteRes<[M3UnitFDIV]>  { let Latency = 7;
+                                                    let ResourceCycles = [8]; }
+def M3WriteFDIV12  : SchedWriteRes<[M3UnitFDIV]>  { let Latency = 12;
+                                                    let ResourceCycles = [13]; }
+def M3WriteFMAC3   : SchedWriteRes<[M3UnitFMAC]>  { let Latency = 3; }
+def M3WriteFMAC4   : SchedWriteRes<[M3UnitFMAC]>  { let Latency = 4; }
+def M3WriteFMAC5   : SchedWriteRes<[M3UnitFMAC]>  { let Latency = 5; }
+def M3WriteFSQR17  : SchedWriteRes<[M3UnitFSQR]>  { let Latency = 18;
+                                                    let ResourceCycles = [19]; }
+def M3WriteFSQR25  : SchedWriteRes<[M3UnitFSQR]>  { let Latency = 25;
+                                                    let ResourceCycles = [26]; }
+def M3WriteNALU1   : SchedWriteRes<[M3UnitNALU]>  { let Latency = 1; }
+def M3WriteNCRY1A  : SchedWriteRes<[M3UnitNCRY0]> { let Latency = 1; }
+def M3WriteNCRY3A  : SchedWriteRes<[M3UnitNCRY0]> { let Latency = 3; }
+def M3WriteNCRY5A  : SchedWriteRes<[M3UnitNCRY]>  { let Latency = 5; }
+def M3WriteNMSC1   : SchedWriteRes<[M3UnitNMSC]>  { let Latency = 1; }
+def M3WriteNMSC2   : SchedWriteRes<[M3UnitNMSC]>  { let Latency = 2; }
+def M3WriteNMSC3   : SchedWriteRes<[M3UnitNMSC]>  { let Latency = 3; }
+def M3WriteNMUL3   : SchedWriteRes<[M3UnitNMUL]>  { let Latency = 3; }
+def M3WriteNSHF1   : SchedWriteRes<[M3UnitNSHF]>  { let Latency = 1; }
+def M3WriteNSHF3   : SchedWriteRes<[M3UnitNSHF]>  { let Latency = 3; }
+def M3WriteNSHT1   : SchedWriteRes<[M3UnitNSHT]>  { let Latency = 1; }
+def M3WriteNSHT2   : SchedWriteRes<[M3UnitNSHT]>  { let Latency = 2; }
+def M3WriteNSHT3   : SchedWriteRes<[M3UnitNSHT]>  { let Latency = 3; }
+def M3WriteVLDA    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL]>     { let Latency = 5;
+                                                    let NumMicroOps = 2; }
+def M3WriteVLDB    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL]>     { let Latency = 6;
+                                                    let NumMicroOps = 3; }
+def M3WriteVLDC    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL]>     { let Latency = 6;
+                                                    let NumMicroOps = 4; }
+def M3WriteVLDD    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU]>  { let Latency = 7;
+                                                    let NumMicroOps = 2;
+                                                    let ResourceCycles = [2]; }
+def M3WriteVLDE    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU]>  { let Latency = 6;
+                                                    let NumMicroOps = 2;
+                                                    let ResourceCycles = [2]; }
+def M3WriteVLDF    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL]>     { let Latency = 10;
+                                                    let NumMicroOps = 2;
+                                                    let ResourceCycles = [5]; }
+def M3WriteVLDG    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU]>  { let Latency = 7;
+                                                    let NumMicroOps = 3;
+                                                    let ResourceCycles = [2]; }
+def M3WriteVLDH    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU]>  { let Latency = 6;
+                                                    let NumMicroOps = 3;
+                                                    let ResourceCycles = [2]; }
+def M3WriteVLDI    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL]>     { let Latency = 12;
+                                                    let NumMicroOps = 3;
+                                                    let ResourceCycles = [6]; }
+def M3WriteVLDJ    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU,
+                                    M3UnitNALU]>  { let Latency = 7;
+                                                    let NumMicroOps = 4;
+                                                    let ResourceCycles = [2]; }
+def M3WriteVLDK    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU,
+                                    M3UnitNALU,
+                                    M3UnitNALU]>  { let Latency = 9;
+                                                    let NumMicroOps = 5;
+                                                    let ResourceCycles = [4]; }
+def M3WriteVLDL    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU,
+                                    M3UnitL,
+                                    M3UnitNALU]>  { let Latency = 6;
+                                                    let NumMicroOps = 5;
+                                                    let ResourceCycles = [3]; }
+def M3WriteVLDM    : SchedWriteRes<[M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU,
+                                    M3UnitL,
+                                    M3UnitNALU,
+                                    M3UnitNALU]>  { let Latency = 7;
+                                                    let NumMicroOps = 6;
+                                                    let ResourceCycles = [3]; }
+def M3WriteVLDN    : SchedWriteRes<[M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL,
+                                    M3UnitL]>     { let Latency = 14;
+                                                    let NumMicroOps = 4;
+                                                    let ResourceCycles = [7]; }
+def M3WriteVSTA    : WriteSequence<[WriteVST], 2>;
+def M3WriteVSTB    : WriteSequence<[WriteVST], 3>;
+def M3WriteVSTC    : WriteSequence<[WriteVST], 4>;
+def M3WriteVSTD    : SchedWriteRes<[M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 7;
+                                                    let NumMicroOps = 2;
+                                                    let ResourceCycles = [7]; }
+def M3WriteVSTE    : SchedWriteRes<[M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 8;
+                                                    let NumMicroOps = 3;
+                                                    let ResourceCycles = [8]; }
+def M3WriteVSTF    : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitFST,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 15;
+                                                    let NumMicroOps = 5;
+                                                    let ResourceCycles = [15]; }
+def M3WriteVSTG    : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitFST,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 16;
+                                                    let NumMicroOps = 6;
+                                                    let ResourceCycles = [16]; }
+def M3WriteVSTH    : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitFST,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 14;
+                                                    let NumMicroOps = 4;
+                                                    let ResourceCycles = [14]; }
+def M3WriteVSTI    : SchedWriteRes<[M3UnitNALU,
+                                    M3UnitFST,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST,
+                                    M3UnitS,
+                                    M3UnitFST]>   { let Latency = 17;
+                                                    let NumMicroOps = 7;
+                                                    let ResourceCycles = [17]; }
+
+// Special cases.
+def M3WriteAES     : SchedWriteRes<[M3UnitNCRY]>  { let Latency = 1; }
+def M3ReadAES      : SchedReadAdvance<1, [M3WriteAES]>;
+def M3ReadFMAC     : SchedReadAdvance<1, [M3WriteFMAC4,
+                                          M3WriteFMAC5]>;
+def M3WriteMOVI    : SchedWriteVariant<[SchedVar<M3ResetFastPred, [M3WriteZ0]>,
+                                        SchedVar<NoSchedPred,     [M3WriteNALU1]>]>;
+def M3ReadNMUL     : SchedReadAdvance<1, [M3WriteNMUL3]>;
+
+// Branch instructions
+def : InstRW<[M3WriteB1], (instrs Bcc)>;
+def : InstRW<[M3WriteA1], (instrs BL)>;
+def : InstRW<[M3WriteBX], (instrs BLR)>;
+def : InstRW<[M3WriteC1], (instregex "^CBN?Z[WX]")>;
+def : InstRW<[M3WriteAD], (instregex "^TBN?Z[WX]")>;
+
+// Arithmetic and logical integer instructions.
+def : InstRW<[M3WriteA1], (instrs COPY)>;
+def : InstRW<[M3WriteAX], (instregex "(ADD|AND|BIC|EON|EOR|ORN|ORR|SUB)S?[WX]r[sx](64)?$")>;
+
+// Move instructions.
+def : InstRW<[M3WriteZ0], (instrs ADR, ADRP)>;
+def : InstRW<[M3WriteZ0], (instregex "^MOV[NZ][WX]i")>;
+
+// Divide and multiply instructions.
+
+// Miscellaneous instructions.
+
+// Load instructions.
+def : InstRW<[M3WriteLD,
+              WriteLDHi,
+              WriteAdr],    (instregex "^LDP(SW|W|X)(post|pre)")>;
+def : InstRW<[M3WriteLX,
+              ReadAdrBase], (instregex "^PRFMro[WX]")>;
+
+// Store instructions.
+
+// FP data instructions.
+def : InstRW<[M3WriteNSHF1],  (instregex "^FABS[DS]r")>;
+def : InstRW<[M3WriteFADD2],  (instregex "^F(ADD|SUB)[DS]rr")>;
+def : InstRW<[M3WriteFDIV10], (instrs FDIVSrr)>;
+def : InstRW<[M3WriteFDIV12], (instrs FDIVDrr)>;
+def : InstRW<[M3WriteNMSC1],  (instregex "^F(MAX|MIN).+rr")>;
+def : InstRW<[M3WriteFMAC3],  (instregex "^FN?MUL[DS]rr")>;
+def : InstRW<[M3WriteFMAC4,
+              M3ReadFMAC],    (instregex "^FN?M(ADD|SUB)[DS]rrr")>;
+def : InstRW<[M3WriteNALU1],  (instregex "^FNEG[DS]r")>;
+def : InstRW<[M3WriteFCVT3A], (instregex "^FRINT.+r")>;
+def : InstRW<[M3WriteNEONH],  (instregex "^FCSEL[DS]rrr")>;
+def : InstRW<[M3WriteFSQR17], (instrs FSQRTSr)>;
+def : InstRW<[M3WriteFSQR25], (instrs FSQRTDr)>;
+
+// FP miscellaneous instructions.
+def : InstRW<[M3WriteFCVT3],  (instregex "^FCVT[DHS][DHS]r")>;
+def : InstRW<[M3WriteFCVT4A], (instregex "^[SU]CVTF[SU][XW][DHS]ri")>;
+def : InstRW<[M3WriteFCVT3A], (instregex "^FCVT[AMNPZ][SU]U[XW][DHS]r")>;
+def : InstRW<[M3WriteFCVT3A], (instregex "^FCVTZ[SU][dhs]")>;
+def : InstRW<[M3WriteNALU1],  (instregex "^FMOV[DS][ir]")>;
+def : InstRW<[M3WriteFCVT4],  (instregex "^[FU](RECP|RSQRT)Ev1")>;
+def : InstRW<[M3WriteNMSC1],  (instregex "^FRECPXv1")>;
+def : InstRW<[M3WriteFMAC4,
+              M3ReadFMAC],    (instregex "^F(RECP|RSQRT)S(16|32|64)")>;
+def : InstRW<[M3WriteNALU1],  (instregex "^FMOV[WX][DS](High)?r")>;
+def : InstRW<[M3WriteNALU1],  (instregex "^FMOV[DS][WX](High)?r")>;
+
+// FP load instructions.
+def : InstRW<[WriteVLD],    (instregex "^LDR[DSQ]l")>;
+def : InstRW<[WriteVLD],    (instregex "^LDUR[BDHSQ]i")>;
+def : InstRW<[WriteVLD,
+              WriteAdr],    (instregex "^LDR[BDHSQ](post|pre)")>;
+def : InstRW<[WriteVLD],    (instregex "^LDR[BDHSQ]ui")>;
+def : InstRW<[M3WriteLX,
+              ReadAdrBase], (instregex "^LDR[BDHS]ro[WX]")>;
+def : InstRW<[M3WriteLB,
+              ReadAdrBase], (instregex "^LDRQro[WX]")>;
+def : InstRW<[WriteVLD,
+              M3WriteLH],   (instregex "^LDN?P[DS]i")>;
+def : InstRW<[M3WriteLA,
+              M3WriteLH],   (instregex "^LDN?PQi")>;
+def : InstRW<[M3WriteLB,
+              M3WriteLH,
+              WriteAdr],    (instregex "^LDP[DS](post|pre)")>;
+def : InstRW<[M3WriteLC,
+              M3WriteLH,
+              WriteAdr],    (instregex "^LDPQ(post|pre)")>;
+
+// FP store instructions.
+def : InstRW<[WriteVST],    (instregex "^STUR[BDHSQ]i")>;
+def : InstRW<[WriteVST,
+              WriteAdr],    (instregex "^STR[BDHSQ](post|pre)")>;
+def : InstRW<[WriteVST],    (instregex "^STR[BDHSQ]ui")>;
+def : InstRW<[M3WriteSY,
+              ReadAdrBase], (instregex "^STR[BDHS]ro[WX]")>;
+def : InstRW<[M3WriteSA,
+              ReadAdrBase], (instregex "^STRQro[WX]")>;
+def : InstRW<[WriteVST],    (instregex "^STN?P[DSQ]i")>;
+def : InstRW<[WriteVST,
+              WriteAdr],    (instregex "^STP[DS](post|pre)")>;
+def : InstRW<[M3WriteSA,
+              WriteAdr],    (instregex "^STPQ(post|pre)")>;
+
+// ASIMD instructions.
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]ABAL?v")>;
+def : InstRW<[M3WriteNMSC1], (instregex "^[SU]ABDL?v")>;
+def : InstRW<[M3WriteNMSC1], (instregex "^(SQ)?(ABS|NEG)v")>;
+def : InstRW<[M3WriteNALU1], (instregex "^(ADD|NEG|SUB)v")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]?ADDL?Pv")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]H(ADD|SUB)v")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU](ADD|SUB)[LW]V?v")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^R?(ADD|SUB)HN2?v")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]Q(ADD|SUB)v")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^(SU|US)QADDv")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]RHADDv")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]?ADDL?Vv")>;
+def : InstRW<[M3WriteNMSC1], (instregex "^CM(EQ|GE|GT|HI|HS|LE|LT)v")>;
+def : InstRW<[M3WriteNALU1], (instregex "^CMTSTv")>;
+def : InstRW<[M3WriteNALU1], (instregex "^(AND|BIC|EOR|MVNI|NOT|ORN|ORR)v")>;
+def : InstRW<[M3WriteNMSC1], (instregex "^[SU](MIN|MAX)v")>;
+def : InstRW<[M3WriteNMSC2], (instregex "^[SU](MIN|MAX)Pv")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU](MIN|MAX)Vv")>;
+def : InstRW<[M3WriteNMUL3], (instregex "^(MUL|SQR?DMULH)v")>;
+def : InstRW<[M3WriteNMUL3,
+              M3ReadNMUL],   (instregex "^ML[AS]v")>;
+def : InstRW<[M3WriteNMUL3], (instregex "^[SU]ML[AS]Lv")>;
+def : InstRW<[M3WriteNMUL3], (instregex "^SQDML[AS]L")>;
+def : InstRW<[M3WriteNMUL3], (instregex "^(S|U|SQD)MULLv")>;
+def : InstRW<[M3WriteNMSC3], (instregex "^[SU]ADALPv")>;
+def : InstRW<[M3WriteNSHT3], (instregex "^[SU]R?SRAv")>;
+def : InstRW<[M3WriteNSHT1], (instregex "^SHL[dv]")>;
+def : InstRW<[M3WriteNSHT1], (instregex "^[SU]SH[LR][dv]")>;
+def : InstRW<[M3WriteNSHT1], (instregex "^S[RS]I[dv]")>;
+def : InstRW<[M3WriteNSHT2], (instregex "^[SU]?SHLLv")>;
+def : InstRW<[M3WriteNSHT3], (instregex "^(([SU]Q)?R)?SHRU?N[bhsv]")>;
+def : InstRW<[M3WriteNSHT3], (instregex "^[SU]RSH[LR][dv]")>;
+def : InstRW<[M3WriteNSHT3], (instregex "^[SU]QR?SHLU?[bdhsv]")>;
+
+// ASIMD FP instructions.
+def : InstRW<[M3WriteNSHF1],  (instregex "^FABSv")>;
+def : InstRW<[M3WriteFADD2],  (instregex "^F(ABD|ADD|SUB)v")>;
+def : InstRW<[M3WriteNEONA],  (instregex "^FADDP")>;
+def : InstRW<[M3WriteNMSC1],  (instregex "^F(AC|CM)(EQ|GE|GT|LE|LT)v[^1]")>;
+def : InstRW<[M3WriteFCVT3],  (instregex "^FCVT(L|N|XN)v")>;
+def : InstRW<[M3WriteFCVT2],  (instregex "^FCVT[AMNPZ][SU]v")>;
+def : InstRW<[M3WriteFCVT3],  (instregex "^[SU]CVTFv")>;
+def : InstRW<[M3WriteFDIV10], (instrs FDIVv2f32)>;
+def : InstRW<[M3WriteNEONV],  (instrs FDIVv4f32)>;
+def : InstRW<[M3WriteNEONW],  (instrs FDIVv2f64)>;
+def : InstRW<[M3WriteNMSC1],  (instregex "^F(MAX|MIN)(NM)?v")>;
+def : InstRW<[M3WriteNMSC2],  (instregex "^F(MAX|MIN)(NM)?Pv")>;
+def : InstRW<[M3WriteNEONZ],  (instregex "^F(MAX|MIN)(NM)?Vv")>;
+def : InstRW<[M3WriteFMAC3],  (instregex "^FMULX?v.[fi]")>;
+def : InstRW<[M3WriteFMAC4,
+              M3ReadFMAC],    (instregex "^FML[AS]v.f")>;
+def : InstRW<[M3WriteFMAC5,
+              M3ReadFMAC],    (instregex "^FML[AS]v.i")>;
+def : InstRW<[M3WriteNALU1],  (instregex "^FNEGv")>;
+def : InstRW<[M3WriteFCVT3A], (instregex "^FRINT[AIMNPXZ]v")>;
+def : InstRW<[M3WriteFSQR17], (instrs FSQRTv2f32)>;
+def : InstRW<[M3WriteNEONX],  (instrs FSQRTv4f32)>;
+def : InstRW<[M3WriteNEONY],  (instrs FSQRTv2f64)>;
+
+// ASIMD miscellaneous instructions.
+def : InstRW<[M3WriteNALU1], (instregex "^RBITv")>;
+def : InstRW<[M3WriteNALU1], (instregex "^(BIF|BIT|BSL)v")>;
+def : InstRW<[M3WriteNEONB], (instregex "^DUPv.+gpr")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^DUPv.+lane")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^EXTv")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^[SU]?Q?XTU?Nv")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^CPY")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^INSv.+lane")>;
+def : InstRW<[M3WriteMOVI],  (instregex "^MOVI")>;
+def : InstRW<[M3WriteNALU1], (instregex "^FMOVv")>;
+def : InstRW<[M3WriteFCVT4], (instregex "^[FU](RECP|RSQRT)Ev[248]")>;
+def : InstRW<[M3WriteFMAC4,
+              M3ReadFMAC],   (instregex "^F(RECP|RSQRT)Sv")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^REV(16|32|64)v")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^TB[LX]v")>;
+def : InstRW<[M3WriteNEOND], (instregex "^[SU]MOVv")>;
+def : InstRW<[M3WriteNSHF3], (instregex "^INSv.+gpr")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^(TRN|UZP|ZIP)[12]v")>;
+
+// ASIMD load instructions.
+def : InstRW<[M3WriteL5],   (instregex "LD1Onev(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteL5,
+              WriteAdr],    (instregex "LD1Onev(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteL5],   (instregex "LD1Onev(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteL5,
+              WriteAdr],    (instregex "LD1Onev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDA], (instregex "LD1Twov(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDA,
+              WriteAdr],    (instregex "LD1Twov(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDA], (instregex "LD1Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDA,
+              WriteAdr],    (instregex "LD1Twov(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDB], (instregex "LD1Threev(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDB,
+              WriteAdr],    (instregex "LD1Threev(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDB], (instregex "LD1Threev(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDB,
+              WriteAdr],    (instregex "LD1Threev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDC], (instregex "LD1Fourv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDC,
+              WriteAdr],    (instregex "LD1Fourv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDC], (instregex "LD1Fourv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDC,
+              WriteAdr],    (instregex "LD1Fourv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDD], (instregex "LD1i(8|16|32)$")>;
+def : InstRW<[M3WriteVLDD,
+              WriteAdr],    (instregex "LD1i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVLDE], (instregex "LD1i(64)$")>;
+def : InstRW<[M3WriteVLDE,
+              WriteAdr],    (instregex "LD1i(64)_POST")>;
+
+def : InstRW<[M3WriteL5],   (instregex "LD1Rv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteL5,
+              WriteAdr],    (instregex "LD1Rv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteL5],   (instregex "LD1Rv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteL5,
+              WriteAdr],    (instregex "LD1Rv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDF], (instregex "LD2Twov(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVLDF,
+              WriteAdr],    (instregex "LD2Twov(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVLDF], (instregex "LD2Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDF,
+              WriteAdr],    (instregex "LD2Twov(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDG], (instregex "LD2i(8|16|32)$")>;
+def : InstRW<[M3WriteVLDG,
+              WriteAdr],    (instregex "LD2i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVLDH], (instregex "LD2i(64)$")>;
+def : InstRW<[M3WriteVLDH,
+              WriteAdr],    (instregex "LD2i(64)_POST")>;
+
+def : InstRW<[M3WriteVLDA], (instregex "LD2Rv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDA,
+              WriteAdr],    (instregex "LD2Rv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDA], (instregex "LD2Rv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDA,
+              WriteAdr],    (instregex "LD2Rv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDI], (instregex "LD3Threev(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVLDI,
+              WriteAdr],    (instregex "LD3Threev(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVLDI], (instregex "LD3Threev(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDI,
+              WriteAdr],    (instregex "LD3Threev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDJ], (instregex "LD3i(8|16|32)$")>;
+def : InstRW<[M3WriteVLDJ,
+              WriteAdr],    (instregex "LD3i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVLDL], (instregex "LD3i(64)$")>;
+def : InstRW<[M3WriteVLDL,
+              WriteAdr],    (instregex "LD3i(64)_POST")>;
+
+def : InstRW<[M3WriteVLDB], (instregex "LD3Rv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDB,
+              WriteAdr],    (instregex "LD3Rv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDB], (instregex "LD3Rv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDB,
+              WriteAdr],    (instregex "LD3Rv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDN], (instregex "LD4Fourv(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVLDN,
+              WriteAdr],    (instregex "LD4Fourv(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVLDN], (instregex "LD4Fourv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDN,
+              WriteAdr],    (instregex "LD4Fourv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVLDK], (instregex "LD4i(8|16|32)$")>;
+def : InstRW<[M3WriteVLDK,
+              WriteAdr],    (instregex "LD4i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVLDM], (instregex "LD4i(64)$")>;
+def : InstRW<[M3WriteVLDM,
+              WriteAdr],    (instregex "LD4i(64)_POST")>;
+
+def : InstRW<[M3WriteVLDC], (instregex "LD4Rv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVLDC,
+              WriteAdr],    (instregex "LD4Rv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVLDC], (instregex "LD4Rv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVLDC,
+              WriteAdr],    (instregex "LD4Rv(16b|8h|4s|2d)_POST")>;
+
+// ASIMD store instructions.
+def : InstRW<[WriteVST],    (instregex "ST1Onev(8b|4h|2s|1d)$")>;
+def : InstRW<[WriteVST,
+              WriteAdr],    (instregex "ST1Onev(8b|4h|2s|1d)_POST")>;
+def : InstRW<[WriteVST],    (instregex "ST1Onev(16b|8h|4s|2d)$")>;
+def : InstRW<[WriteVST,
+              WriteAdr],    (instregex "ST1Onev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTA], (instregex "ST1Twov(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVSTA,
+              WriteAdr],    (instregex "ST1Twov(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVSTA], (instregex "ST1Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTA,
+              WriteAdr],    (instregex "ST1Twov(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTB], (instregex "ST1Threev(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVSTB,
+              WriteAdr],    (instregex "ST1Threev(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVSTB], (instregex "ST1Threev(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTB,
+              WriteAdr],    (instregex "ST1Threev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTC], (instregex "ST1Fourv(8b|4h|2s|1d)$")>;
+def : InstRW<[M3WriteVSTC,
+              WriteAdr],    (instregex "ST1Fourv(8b|4h|2s|1d)_POST")>;
+def : InstRW<[M3WriteVSTC], (instregex "ST1Fourv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTC,
+              WriteAdr],    (instregex "ST1Fourv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTD], (instregex "ST1i(8|16|32|64)$")>;
+def : InstRW<[M3WriteVSTD,
+              WriteAdr],    (instregex "ST1i(8|16|32|64)_POST")>;
+
+def : InstRW<[M3WriteVSTD], (instregex "ST2Twov(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVSTD,
+              WriteAdr],    (instregex "ST2Twov(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVSTE], (instregex "ST2Twov(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTE,
+              WriteAdr],    (instregex "ST2Twov(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTD], (instregex "ST2i(8|16|32)$")>;
+def : InstRW<[M3WriteVSTD,
+              WriteAdr],    (instregex "ST2i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVSTD], (instregex "ST2i(64)$")>;
+def : InstRW<[M3WriteVSTD,
+              WriteAdr],    (instregex "ST2i(64)_POST")>;
+
+def : InstRW<[M3WriteVSTF], (instregex "ST3Threev(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVSTF,
+              WriteAdr],    (instregex "ST3Threev(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVSTG], (instregex "ST3Threev(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTG,
+              WriteAdr],    (instregex "ST3Threev(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTH], (instregex "ST3i(8|16|32)$")>;
+def : InstRW<[M3WriteVSTH,
+              WriteAdr],    (instregex "ST3i(8|16|32)_POST")>;
+def : InstRW<[M3WriteVSTF], (instregex "ST3i(64)$")>;
+def : InstRW<[M3WriteVSTF,
+              WriteAdr],    (instregex "ST3i(64)_POST")>;
+
+def : InstRW<[M3WriteVSTF], (instregex "ST4Fourv(8b|4h|2s)$")>;
+def : InstRW<[M3WriteVSTF,
+              WriteAdr],    (instregex "ST4Fourv(8b|4h|2s)_POST")>;
+def : InstRW<[M3WriteVSTI], (instregex "ST4Fourv(16b|8h|4s|2d)$")>;
+def : InstRW<[M3WriteVSTI,
+              WriteAdr],    (instregex "ST4Fourv(16b|8h|4s|2d)_POST")>;
+
+def : InstRW<[M3WriteVSTF], (instregex "ST4i(8|16|32|64)$")>;
+def : InstRW<[M3WriteVSTF,
+              WriteAdr],    (instregex "ST4i(8|16|32|64)_POST")>;
+
+// Cryptography instructions.
+def : InstRW<[M3WriteAES],    (instregex "^AES[DE]")>;
+def : InstRW<[M3WriteAES,
+              M3ReadAES],     (instregex "^AESI?MC")>;
+
+def : InstRW<[M3WriteNCRY3A], (instregex "^PMULL?v")>;
+
+def : InstRW<[M3WriteNCRY1A], (instregex "^SHA1([CHMP]|SU[01])")>;
+def : InstRW<[M3WriteNCRY1A], (instregex "^SHA256SU0")>;
+def : InstRW<[M3WriteNCRY5A], (instregex "^SHA256(H2?|SU1)")>;
+
+// CRC instructions.
+def : InstRW<[M3WriteC2], (instregex "^CRC32")>;
+
+} // SchedModel = ExynosM3Model

llvm/trunk/lib/Target/AArch64/AArch64Subtarget.h

enum ARMProcFamilyEnum : uint8_t {
     CortexA53,
     CortexA55,
     CortexA57,
     CortexA72,
     CortexA73,
     CortexA75,
     Cyclone,
     ExynosM1,
+    ExynosM3,
     Falkor,
     Kryo,
     Saphira,
     ThunderX2T99,
     ThunderX,
     ThunderXT81,
     ThunderXT83,
     ThunderXT88

llvm/trunk/lib/Target/AArch64/AArch64Subtarget.cpp

case CortexA57:
     PrefFunctionAlignment = 4;
     break;
   case ExynosM1:
     MaxInterleaveFactor = 4;
     MaxJumpTableSize = 8;
     PrefFunctionAlignment = 4;
     PrefLoopAlignment = 3;
     break;
+  case ExynosM3:
+    MaxInterleaveFactor = 4;
+    MaxJumpTableSize = 20;
+    PrefFunctionAlignment = 5;
+    PrefLoopAlignment = 4;
+    break;
   case Falkor:
     MaxInterleaveFactor = 4;
     // FIXME: remove this to enable 64-bit SLP if performance looks good.
     MinVectorRegisterBitWidth = 128;
     CacheLineSize = 128;
     PrefetchDistance = 820;
     MinPrefetchStride = 2048;
     MaxPrefetchIterationsAhead = 8;

llvm/trunk/test/CodeGen/AArch64/max-jump-table.ll

 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40                        -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK0  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -max-jump-table-size=4 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK4  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -max-jump-table-size=8 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK8  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -mcpu=exynos-m1        -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECKM1 < %t
+; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -mcpu=exynos-m3        -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECKM3 < %t
 
 declare void @ext(i32)
 
 define i32 @jt1(i32 %a, i32 %b) {
 entry:
   switch i32 %a, label %return [
     i32 1, label %bb1
     i32 2, label %bb2

 ; CHECK4-SAME: %jump-table.3:
 ; CHECK4-NOT: %jump-table.4:
 ; CHECK8-NEXT: %jump-table.0:
 ; CHECK8-SAME: %jump-table.1:
 ; CHECK8-NOT: %jump-table.2:
 ; CHECKM1-NEXT: %jump-table.0:
 ; CHECKM1-SAME: %jump-table.1
 ; CHECKM1-NOT: %jump-table.2:
-; CHEC-NEXT: Function Live Ins:
+; CHECKM3-NEXT: %jump-table.0:
+; CHECKM3-NOT: %jump-table.1:
 
 bb1: tail call void @ext(i32 0) br label %return
 bb2: tail call void @ext(i32 2) br label %return
 bb3: tail call void @ext(i32 4) br label %return
 bb4: tail call void @ext(i32 6) br label %return
 bb5: tail call void @ext(i32 8) br label %return
 bb6: tail call void @ext(i32 10) br label %return
 bb7: tail call void @ext(i32 12) br label %return
switch i32 %x, label %return [
     i32 4, label %bb4
 
     i32 14, label %bb5
     i32 15, label %bb6
   ]
 ; CHECK-LABEL: function jt2:
 ; CHECK-NEXT: Jump Tables:
 ; CHECK0-NEXT: %jump-table.0:  %bb.1 %bb.2 %bb.3 %bb.4 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.5 %bb.6{{$}}
+; CHECK0-NOT: %jump-table.1
 ; CHECK4-NEXT: %jump-table.0:  %bb.1 %bb.2 %bb.3 %bb.4{{$}}
+; CHECK4-NOT: %jump-table.1
 ; CHECK8-NEXT: %jump-table.0:  %bb.1 %bb.2 %bb.3 %bb.4{{$}}
+; CHECK8-NOT: %jump-table.1
 ; CHECKM1-NEXT: %jump-table.0:  %bb.1 %bb.2 %bb.3 %bb.4{{$}}
-; CHEC-NEXT: Function Live Ins:
+; CHECKM1-NOT: %jump-table.1
+; CHECKM3-NEXT: %jump-table.0:  %bb.1 %bb.2 %bb.3 %bb.4 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.5 %bb.6{{$}}
+; CHECKM3-NOT: %jump-table.1
 
 bb1: tail call void @ext(i32 1) br label %return
 bb2: tail call void @ext(i32 2) br label %return
 bb3: tail call void @ext(i32 3) br label %return
 bb4: tail call void @ext(i32 4) br label %return
 bb5: tail call void @ext(i32 5) br label %return
 bb6: tail call void @ext(i32 6) br label %return
 return: ret void
 }

llvm/trunk/test/CodeGen/AArch64/misched-fusion-aes.ll

 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-aes,+crypto | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=generic -mattr=+crypto | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a53 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a72 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a73 | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m1  | FileCheck %s
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m2  | FileCheck %s
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m3  | FileCheck %s
 
 declare <16 x i8> @llvm.aarch64.crypto.aese(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesmc(<16 x i8> %d)
 declare <16 x i8> @llvm.aarch64.crypto.aesd(<16 x i8> %d, <16 x i8> %k)
 declare <16 x i8> @llvm.aarch64.crypto.aesimc(<16 x i8> %d)
 
 define void @aesea(<16 x i8>* %a0, <16 x i8>* %b0, <16 x i8>* %c0, <16 x i8> %d, <16 x i8> %e) {
   %d0 = load <16 x i8>, <16 x i8>* %a0

llvm/trunk/test/CodeGen/AArch64/misched-fusion-lit.ll

 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=-fuse-literals | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKDONT
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mattr=+fuse-literals | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 ; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=cortex-a57      | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
+; RUN: llc %s -o - -mtriple=aarch64-unknown -mcpu=exynos-m3       | FileCheck %s --check-prefix=CHECK --check-prefix=CHECKFUSE
 
 @g = common local_unnamed_addr global i8* null, align 8
 
 define i8* @litp(i32 %a, i32 %b) {
 entry:
   %add = add nsw i32 %b, %a
   %idx.ext = sext i32 %add to i64
   %add.ptr = getelementptr i8, i8* bitcast (i8* (i32, i32)* @litp to i8*), i64 %idx.ext

llvm/trunk/test/CodeGen/AArch64/preferred-function-alignment.ll

 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=generic < %s | FileCheck --check-prefixes=ALIGN2,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cortex-a35 < %s | FileCheck --check-prefixes=ALIGN2,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cyclone < %s | FileCheck --check-prefixes=ALIGN2,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=falkor < %s | FileCheck --check-prefixes=ALIGN2,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=kryo < %s | FileCheck --check-prefixes=ALIGN2,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cortex-a53 < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=thunderx < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=thunderxt81 < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=thunderxt83 < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=thunderxt88 < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=thunderx2t99 < %s | FileCheck --check-prefixes=ALIGN3,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cortex-a57 < %s | FileCheck --check-prefixes=ALIGN4,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cortex-a72 < %s | FileCheck --check-prefixes=ALIGN4,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=cortex-a73 < %s | FileCheck --check-prefixes=ALIGN4,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=exynos-m1 < %s | FileCheck ---check-prefixes=ALIGN4,CHECK %s
 ; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=exynos-m2 < %s | FileCheck ---check-prefixes=ALIGN4,CHECK %s
-; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=exynos-m3 < %s | FileCheck ---check-prefixes=ALIGN4,CHECK %s
+; RUN: llc -mtriple=aarch64-unknown-linux -mcpu=exynos-m3 < %s | FileCheck ---check-prefixes=ALIGN5,CHECK %s
 
 define void @test() {
   ret void
 }
 
 ; CHECK-LABEL: test
 ; ALIGN2: .p2align 2
 ; ALIGN3: .p2align 3
 ; ALIGN4: .p2align 4
+; ALIGN5: .p2align 5
 
 define void @test_optsize() optsize {
   ret void
 }
 
 ; CHECK-LABEL: test_optsize
 ; CHECK-NEXT: .p2align 2