[llvm][AArch64] Handle arrays of struct properly (from IR)

[DavidSpickett]

Summary

This only applies to FastIsel. GlobalIsel seems to sidestep
the issue.

This fixes https://bugs.llvm.org/show_bug.cgi?id=46996

One of the things we do in llvm is decide if a type needs
consecutive registers. Previously, we just checked if it
was an array or not.
(plus an SVE specific check that is not changing here)

This causes some confusion when you arbitrary IR like:

%T1 = type { double, i1 };
define [ 1 x %T1 ] @foo() {
entry:
  ret [ 1 x %T1 ] zeroinitializer
}

We see it is an array so we call CC_AArch64_Custom_Block
which bails out when it sees the i1, a type we don't want
to put into a block.

This leaves the location of the double in some kind of
intermediate state and leads to odd codegen. Which then crashes
the backend because it doesn't know how to implement
what it's been asked for.

You get this:

renamable $d0 = FMOVD0
$w0 = COPY killed renamable $d0

Rather than this:

$d0 = FMOVD0
$w0 = COPY $wzr

The backend knows how to copy 64 bit to 64 bit registers,
but not 64 to 32. It can certainly be taught how but the real
issue seems to be us even trying to assign a register block
in the first place.

This change makes the logic of
AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters
a bit more in depth. If we find an array, also check that all the
nested aggregates in that array have a single member type.

Then CC_AArch64_Custom_Block's assumption of a type that looks
like [ N x type ] will be valid and we get the expected codegen.

New tests have been added to exercise these situations. Note that
some of the output is not ABI compliant. The aim of this change is
to simply handle these situations and not to make our processing
of arbitrary IR ABI compliant.

[DavidSpickett]

Though globalisel doesn't crash with the reported snippet, it does crash on the added tests when we spill to memory. I still need to look into that but I don't think it should block this. (unless the bug reporter turns out to be using globalisel)

On the subject of not being ABI compliant when going direct from IR, here's an example of that: https://godbolt.org/z/oMcPd87nc , this doesn't make the situation any worse.

We started considering more types for register blocks after https://reviews.llvm.org/D61259 but there is no fault in that commit's usage of it. It simply uncovered something that already existed.

[DavidSpickett]

Momchil this is the patch to fix what we were talking about. Walking the whole type might be a bit much but it should exit early 99% of the time.

[chill]

I think this change make sense. I wouldn't worry about the AAPCS64 compliance as long as the (implicit, ill-defined) "LLVM IR PCS" is flexible enough, so as to allow implementing AAPCS64.
Perhaps it's useful to have more tests, on both sides of the calls and returns:

target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "aarch64-eabi" 

%T = type {double, double, double, double, double, [4 x double]}

@x = dso_local global %T zeroinitializer, align 8

; Check how values are returned
define %T @f0() {
entry:
  ret %T zeroinitializer
}

; Check where the caller expects the values
define void @g() {
  %1 = call %T @f0()
  store %T %1, %T* @x
  ret void
}

; Check where the callee expects the values
define void @u(%T %a) {
  store %T %a, %T* @x
  ret void
}

; Check how values are passed
define void @h() {
  %1 = load %T, %T * @x
  call void @u(%T %1)
  ret void
}

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

17215

No else after return.

17215

Typo.

17224

Could be like for (Type *T : Ty->subtypes()) ...

[DavidSpickett]

• Fix clang-tidy warnings
• Add caller/callee tests. For passing as a block, in memory and in normal registers.

[efriedma]

Seems fine to me. An alternative implementation could use ComputeValueVTs, instead of implementing the recursion yourself, but this seems okay as-is.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

17223

Unused variable subtypes_end

[DavidSpickett]

Use ComputeValueVTs instead of recursing manually.

Note that is_splat does return false for a container
with 0 items but this is fine. Just means there's
nothing concrete for us to handle anyway.

[efriedma]

LGTM

[efriedma]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

17224

Actually, looking at this again, I'm not sure I like using createDataLayout() like this... it's kind of an expensive/unusual operation. Can we pass down the datalayout from the callers?

[DavidSpickett]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

17224

By expensive do you mean in that it creates a copy of the DataLayout object each time? (I thought maybe it reparses the layout string but it doesn't)

I will see about passing it down. (we have IsVarArg there already and only used by Arm)

[DavidSpickett]

Pass the data layout to functionArgumentNeedsConsecutiveRegisters
instead of getting a new copy for every call.

Found out along the way that in fact Arm has a isHomogeneousAggregate
function that is uses. It's following the letter of the ABI, which would
be nice to do here too but I'd rather stick to this for now.

[DavidSpickett]

Remove unused Optional include

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

17224

Turned out it was simple to do anyway.

[efriedma]

LGTM

llvm/include/llvm/CodeGen/TargetLowering.h

 //===- llvm/CodeGen/TargetLowering.h - Target Lowering Info -----*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
virtual EVT getTypeForExtReturn(LLVMContext &Context, EVT VT,
     return VT.bitsLT(MinVT) ? MinVT : VT;
   }
 
   /// For some targets, an LLVM struct type must be broken down into multiple
   /// simple types, but the calling convention specifies that the entire struct
   /// must be passed in a block of consecutive registers.
   virtual bool
   functionArgumentNeedsConsecutiveRegisters(Type *Ty, CallingConv::ID CallConv,
-                                            bool isVarArg) const {
+                                            bool isVarArg,
+                                            const DataLayout &DL) const {
     return false;
   }
 
   /// For most targets, an LLVM type must be broken down into multiple
   /// smaller types. Usually the halves are ordered according to the endianness
   /// but for some platform that would break. So this method will default to
   /// matching the endianness but can be overridden.
   virtual bool

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

 //===-- lib/CodeGen/GlobalISel/CallLowering.cpp - Call lowering -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
SplitArgs.emplace_back(OrigArg.Regs[0], SplitVTs[0].getTypeForEVT(Ctx),
                            OrigArg.OrigValue);
     return;
   }
 
   // Create one ArgInfo for each virtual register in the original ArgInfo.
   assert(OrigArg.Regs.size() == SplitVTs.size() && "Regs / types mismatch");
 
   bool NeedsRegBlock = TLI->functionArgumentNeedsConsecutiveRegisters(
-      OrigArg.Ty, CallConv, false);
+      OrigArg.Ty, CallConv, false, DL);
   for (unsigned i = 0, e = SplitVTs.size(); i < e; ++i) {
     Type *SplitTy = SplitVTs[i].getTypeForEVT(Ctx);
     SplitArgs.emplace_back(OrigArg.Regs[i], SplitTy, OrigArg.Flags[0],
                            OrigArg.IsFixed);
     if (NeedsRegBlock)
       SplitArgs.back().Flags[0].setInConsecutiveRegs();
   }
 

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

 //===- FastISel.cpp - Implementation of the FastISel class ----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
bool FastISel::lowerCallTo(CallLoweringInfo &CLI) {
 
   // Handle all of the outgoing arguments.
   CLI.clearOuts();
   for (auto &Arg : CLI.getArgs()) {
     Type *FinalType = Arg.Ty;
     if (Arg.IsByVal)
       FinalType = cast<PointerType>(Arg.Ty)->getElementType();
     bool NeedsRegBlock = TLI.functionArgumentNeedsConsecutiveRegisters(
-        FinalType, CLI.CallConv, CLI.IsVarArg);
+        FinalType, CLI.CallConv, CLI.IsVarArg, DL);
 
     ISD::ArgFlagsTy Flags;
     if (Arg.IsZExt)
       Flags.setZExt();
     if (Arg.IsSExt)
       Flags.setSExt();
     if (Arg.IsInReg)
       Flags.setInReg();

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

 //===- SelectionDAGBuilder.cpp - Selection-DAG building -------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
if (!FuncInfo.CanLowerReturn) {
     unsigned NumValues = ValueVTs.size();
     if (NumValues) {
       SDValue RetOp = getValue(I.getOperand(0));
 
       const Function *F = I.getParent()->getParent();
 
       bool NeedsRegBlock = TLI.functionArgumentNeedsConsecutiveRegisters(
           I.getOperand(0)->getType(), F->getCallingConv(),
-          /*IsVarArg*/ false);
+          /*IsVarArg*/ false, DL);
 
       ISD::NodeType ExtendKind = ISD::ANY_EXTEND;
       if (F->getAttributes().hasAttribute(AttributeList::ReturnIndex,
                                           Attribute::SExt))
         ExtendKind = ISD::SIGN_EXTEND;
       else if (F->getAttributes().hasAttribute(AttributeList::ReturnIndex,
                                                Attribute::ZExt))
         ExtendKind = ISD::ZERO_EXTEND;
if (!CanLowerReturn) {
     CLI.NumFixedArgs += 1;
     CLI.RetTy = Type::getVoidTy(CLI.RetTy->getContext());
 
     // sret demotion isn't compatible with tail-calls, since the sret argument
     // points into the callers stack frame.
     CLI.IsTailCall = false;
   } else {
     bool NeedsRegBlock = functionArgumentNeedsConsecutiveRegisters(
-        CLI.RetTy, CLI.CallConv, CLI.IsVarArg);
+        CLI.RetTy, CLI.CallConv, CLI.IsVarArg, DL);
     for (unsigned I = 0, E = RetTys.size(); I != E; ++I) {
       ISD::ArgFlagsTy Flags;
       if (NeedsRegBlock) {
         Flags.setInConsecutiveRegs();
         if (I == RetTys.size() - 1)
           Flags.setInConsecutiveRegsLast();
       }
       EVT VT = RetTys[I];
TargetLowering::LowerCallTo(TargetLowering::CallLoweringInfo &CLI) const {
   for (unsigned i = 0, e = Args.size(); i != e; ++i) {
     SmallVector<EVT, 4> ValueVTs;
     ComputeValueVTs(*this, DL, Args[i].Ty, ValueVTs);
     // FIXME: Split arguments if CLI.IsPostTypeLegalization
     Type *FinalType = Args[i].Ty;
     if (Args[i].IsByVal)
       FinalType = cast<PointerType>(Args[i].Ty)->getElementType();
     bool NeedsRegBlock = functionArgumentNeedsConsecutiveRegisters(
-        FinalType, CLI.CallConv, CLI.IsVarArg);
+        FinalType, CLI.CallConv, CLI.IsVarArg, DL);
     for (unsigned Value = 0, NumValues = ValueVTs.size(); Value != NumValues;
          ++Value) {
       EVT VT = ValueVTs[Value];
       Type *ArgTy = VT.getTypeForEVT(CLI.RetTy->getContext());
       SDValue Op = SDValue(Args[i].Node.getNode(),
                            Args[i].Node.getResNo() + Value);
       ISD::ArgFlagsTy Flags;
 
for (const Argument &Arg : F.args()) {
     SmallVector<EVT, 4> ValueVTs;
     ComputeValueVTs(*TLI, DAG.getDataLayout(), Arg.getType(), ValueVTs);
     bool isArgValueUsed = !Arg.use_empty();
     unsigned PartBase = 0;
     Type *FinalType = Arg.getType();
     if (Arg.hasAttribute(Attribute::ByVal))
       FinalType = Arg.getParamByValType();
     bool NeedsRegBlock = TLI->functionArgumentNeedsConsecutiveRegisters(
-        FinalType, F.getCallingConv(), F.isVarArg());
+        FinalType, F.getCallingConv(), F.isVarArg(), DL);
     for (unsigned Value = 0, NumValues = ValueVTs.size();
          Value != NumValues; ++Value) {
       EVT VT = ValueVTs[Value];
       Type *ArgTy = VT.getTypeForEVT(*DAG.getContext());
       ISD::ArgFlagsTy Flags;
 
 
       if (Arg.getType()->isPointerTy()) {

llvm/lib/Target/AArch64/AArch64ISelLowering.h

 //==-- AArch64ISelLowering.h - AArch64 DAG Lowering Interface ----*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
public:
   /// Returns the number of interleaved accesses that will be generated when
   /// lowering accesses of the given type.
   unsigned getNumInterleavedAccesses(VectorType *VecTy,
                                      const DataLayout &DL) const;
 
   MachineMemOperand::Flags getTargetMMOFlags(
     const Instruction &I) const override;
 
-  bool functionArgumentNeedsConsecutiveRegisters(Type *Ty,
-                                                 CallingConv::ID CallConv,
-                                                 bool isVarArg) const override;
+  bool functionArgumentNeedsConsecutiveRegisters(
+      Type *Ty, CallingConv::ID CallConv, bool isVarArg,
+      const DataLayout &DL) const override;
+
   /// Used for exception handling on Win64.
   bool needsFixedCatchObjects() const override;
 
   bool fallBackToDAGISel(const Instruction &Inst) const override;
 
   /// SVE code generation for fixed length vectors does not custom lower
   /// BUILD_VECTOR. This makes BUILD_VECTOR legalisation a source of stores to
   /// merge. However, merging them creates a BUILD_VECTOR that is just as

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 //===-- AArch64ISelLowering.cpp - AArch64 DAG Lowering Implementation  ----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the AArch64TargetLowering class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64ISelLowering.h"
 #include "AArch64CallingConvention.h"
 #include "AArch64ExpandImm.h"
 #include "AArch64MachineFunctionInfo.h"
 #include "AArch64PerfectShuffle.h"
 #include "AArch64RegisterInfo.h"
 #include "AArch64Subtarget.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "Utils/AArch64BaseInfo.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Analysis/ObjCARCUtil.h"
 #include "llvm/Analysis/VectorUtils.h"
+#include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
Value *AArch64TargetLowering::emitStoreConditional(IRBuilderBase &Builder,
 
   return Builder.CreateCall(Stxr,
                             {Builder.CreateZExtOrBitCast(
                                  Val, Stxr->getFunctionType()->getParamType(0)),
                              Addr});
 }
 
 bool AArch64TargetLowering::functionArgumentNeedsConsecutiveRegisters(
-    Type *Ty, CallingConv::ID CallConv, bool isVarArg) const {
-  if (Ty->isArrayTy())
-    return true;
-
+    Type *Ty, CallingConv::ID CallConv, bool isVarArg,
+    const DataLayout &DL) const {
+  if (!Ty->isArrayTy()) {
     const TypeSize &TySize = Ty->getPrimitiveSizeInBits();
-  if (TySize.isScalable() && TySize.getKnownMinSize() > 128)
-    return true;
+    return TySize.isScalable() && TySize.getKnownMinSize() > 128;
+  }
 
-  return false;
+  // All non aggregate members of the type must have the same type
+  SmallVector<EVT> ValueVTs;
+  ComputeValueVTs(*this, DL, Ty, ValueVTs);
+  return is_splat(ValueVTs);
 }
 
 bool AArch64TargetLowering::shouldNormalizeToSelectSequence(LLVMContext &,
                                                             EVT) const {
   return false;
 }
 
 static Value *UseTlsOffset(IRBuilderBase &IRB, unsigned Offset) {

llvm/lib/Target/ARM/ARMISelLowering.h

 //===- ARMISelLowering.h - ARM DAG Lowering Interface -----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
bool shouldFormOverflowOp(unsigned Opcode, EVT VT,
                               bool MathUsed) const override {
       // Using overflow ops for overflow checks only should beneficial on ARM.
       return TargetLowering::shouldFormOverflowOp(Opcode, VT, true);
     }
 
     /// Returns true if an argument of type Ty needs to be passed in a
     /// contiguous block of registers in calling convention CallConv.
     bool functionArgumentNeedsConsecutiveRegisters(
-        Type *Ty, CallingConv::ID CallConv, bool isVarArg) const override;
+        Type *Ty, CallingConv::ID CallConv, bool isVarArg,
+        const DataLayout &DL) const override;
 
     /// If a physical register, this returns the register that receives the
     /// exception address on entry to an EH pad.
     Register
     getExceptionPointerRegister(const Constant *PersonalityFn) const override;
 
     /// If a physical register, this returns the register that receives the
     /// exception typeid on entry to a landing pad.

llvm/lib/Target/ARM/ARMISelLowering.cpp

 //===- ARMISelLowering.cpp - ARM DAG Lowering Implementation --------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
Align ARMTargetLowering::getABIAlignmentForCallingConv(
   // stack and waste space for no real benefit.
   return std::min(ABITypeAlign, DL.getStackAlignment());
 }
 
 /// Return true if a type is an AAPCS-VFP homogeneous aggregate or one of
 /// [N x i32] or [N x i64]. This allows front-ends to skip emitting padding when
 /// passing according to AAPCS rules.
 bool ARMTargetLowering::functionArgumentNeedsConsecutiveRegisters(
-    Type *Ty, CallingConv::ID CallConv, bool isVarArg) const {
+    Type *Ty, CallingConv::ID CallConv, bool isVarArg,
+    const DataLayout &DL) const {
   if (getEffectiveCallingConv(CallConv, isVarArg) !=
       CallingConv::ARM_AAPCS_VFP)
     return false;
 
   HABaseType Base = HA_UNKNOWN;
   uint64_t Members = 0;
   bool IsHA = isHomogeneousAggregate(Ty, Base, Members);
   LLVM_DEBUG(dbgs() << "isHA: " << IsHA << " "; Ty->dump());

llvm/lib/Target/PowerPC/PPCISelLowering.h

 //===-- PPCISelLowering.h - PPC32 DAG Lowering Interface --------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
public:
     /// createFastISel - This method returns a target-specific FastISel object,
     /// or null if the target does not support "fast" instruction selection.
     FastISel *createFastISel(FunctionLoweringInfo &FuncInfo,
                              const TargetLibraryInfo *LibInfo) const override;
 
     /// Returns true if an argument of type Ty needs to be passed in a
     /// contiguous block of registers in calling convention CallConv.
     bool functionArgumentNeedsConsecutiveRegisters(
-      Type *Ty, CallingConv::ID CallConv, bool isVarArg) const override {
+        Type *Ty, CallingConv::ID CallConv, bool isVarArg,
+        const DataLayout &DL) const override {
       // We support any array type as "consecutive" block in the parameter
       // save area.  The element type defines the alignment requirement and
       // whether the argument should go in GPRs, FPRs, or VRs if available.
       //
       // Note that clang uses this capability both to implement the ELFv2
       // homogeneous float/vector aggregate ABI, and to avoid having to use
       // "byval" when passing aggregates that might fully fit in registers.
       return Ty->isArrayTy();

llvm/test/CodeGen/AArch64/argument-blocks-array-of-struct.ll

+; RUN: llc -mtriple=aarch64-none-linux-gnu -o - %s | FileCheck %s
+
+;; Check that the llvm aarch64 backend can handle arrays of
+;; structs and vice versa when passed from IR.
+;; (this layering is something clang would normally simplify)
+;;
+;; Some of these examples are not ABI compliant and they're not
+;; meant to be. For instance according to the ABI an aggregate
+;; with more than 4 members must go in memory. This restriction
+;; is applied earlier in the compilation process so here we do
+;; see 8 member types in registers.
+;;
+;; When we have more than 8 members we simply run out of registers
+;; and that's what produces the 8 limit here.
+
+;; Plain arrays
+
+define [ 0 x double ] @array_0() {
+; CHECK-LABEL: array_0:
+; CHECK:  ret
+  ret [ 0 x double ] zeroinitializer
+}
+
+define [ 1 x double ] @array_1() {
+; CHECK-LABEL: array_1:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 1 x double ] zeroinitializer
+}
+
+define [ 8 x double ] @array_8() {
+; CHECK-LABEL: array_8:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  movi d4, #0000000000000000
+; CHECK-NEXT:  movi d5, #0000000000000000
+; CHECK-NEXT:  movi d6, #0000000000000000
+; CHECK-NEXT:  movi d7, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 8 x double ] zeroinitializer
+}
+
+;; > 8 items goes on the stack
+
+define [ 9 x double ] @array_9() {
+; CHECK-LABEL: array_9:
+; CHECK:      movi v0.2d, #0000000000000000
+; CHECK-NEXT: str xzr, [x8, #64]
+; CHECK-NEXT: stp q0, q0, [x8, #32]
+; CHECK-NEXT: stp q0, q0, [x8]
+; CHECK-NEXT:  ret
+  ret [ 9 x double ] zeroinitializer
+}
+
+;; Won't use any registers, just checking for assumptions.
+%T_STRUCT_0M = type { }
+
+define %T_STRUCT_0M @struct_zero_fields() {
+; CHECK-LABEL: struct_zero_fields:
+; CHECK:  ret
+  ret %T_STRUCT_0M zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_0M ] @array_of_struct_zero_fields() {
+; CHECK-LABEL: array_of_struct_zero_fields:
+; CHECK:  ret
+  ret [ 1 x %T_STRUCT_0M ] zeroinitializer
+}
+
+define [ 2 x %T_STRUCT_0M ] @array_of_struct_zero_fields_in_struct() {
+; CHECK-LABEL: array_of_struct_zero_fields_in_struct:
+; CHECK:  ret
+  ret [ 2 x %T_STRUCT_0M ] zeroinitializer
+}
+
+%T_STRUCT_1M = type { i32 }
+
+define %T_STRUCT_1M @struct_one_field() {
+; CHECK-LABEL: struct_one_field:
+; CHECK:       w0, wzr
+; CHECK-NEXT:  ret
+  ret %T_STRUCT_1M zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_1M ] @array_of_struct_one_field() {
+; CHECK-LABEL: array_of_struct_one_field:
+; CHECK:       w0, wzr
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_STRUCT_1M ] zeroinitializer
+}
+
+;; This one will be a reg block
+define [ 2 x %T_STRUCT_1M ] @array_of_struct_one_field_2() {
+; CHECK-LABEL: array_of_struct_one_field_2:
+; CHECK:       w0, wzr
+; CHECK:       w1, wzr
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_STRUCT_1M ] zeroinitializer
+}
+
+;; Different types for each field, will not be put in a reg block
+%T_STRUCT_DIFFM = type { double, i32 }
+
+define %T_STRUCT_DIFFM @struct_different_field_types() {
+; CHECK-LABEL: struct_different_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  ret
+  ret %T_STRUCT_DIFFM zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_DIFFM ] @array_of_struct_different_field_types() {
+; CHECK-LABEL: array_of_struct_different_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_STRUCT_DIFFM ] zeroinitializer
+}
+
+define [ 2 x %T_STRUCT_DIFFM ] @array_of_struct_different_field_types_2() {
+; CHECK-LABEL: array_of_struct_different_field_types_2:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  w1, wzr
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_STRUCT_DIFFM ] zeroinitializer
+}
+
+;; Each field is the same type, can be put in a reg block
+%T_STRUCT_SAMEM = type { double, double }
+
+;; Here isn't a block as such, we just allocate two consecutive registers
+define %T_STRUCT_SAMEM @struct_same_field_types() {
+; CHECK-LABEL: struct_same_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  ret
+  ret %T_STRUCT_SAMEM zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_SAMEM ] @array_of_struct_same_field_types() {
+; CHECK-LABEL: array_of_struct_same_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_STRUCT_SAMEM ] zeroinitializer
+}
+
+define [ 2 x %T_STRUCT_SAMEM ] @array_of_struct_same_field_types_2() {
+; CHECK-LABEL: array_of_struct_same_field_types_2:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_STRUCT_SAMEM ] zeroinitializer
+}
+
+;; Same field type but integer this time. Put into x registers instead.
+%T_STRUCT_SAMEM_INT = type { i64, i64 }
+
+define %T_STRUCT_SAMEM_INT @struct_same_field_types_int() {
+; CHECK-LABEL: struct_same_field_types_int:
+; CHECK:       x0, xzr
+; CHECK-NEXT:  x1, xzr
+; CHECK-NEXT:  ret
+  ret %T_STRUCT_SAMEM_INT zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_SAMEM_INT ] @array_of_struct_same_field_types_int() {
+; CHECK-LABEL: array_of_struct_same_field_types_int:
+; CHECK:       x0, xzr
+; CHECK-NEXT:  x1, xzr
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_STRUCT_SAMEM_INT ] zeroinitializer
+}
+
+define [ 2 x %T_STRUCT_SAMEM_INT ] @array_of_struct_same_field_types_int_2() {
+; CHECK-LABEL: array_of_struct_same_field_types_int_2:
+; CHECK:       x0, xzr
+; CHECK-NEXT:  x1, xzr
+; CHECK-NEXT:  x2, xzr
+; CHECK-NEXT:  x3, xzr
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_STRUCT_SAMEM_INT ] zeroinitializer
+}
+
+;; An aggregate of more than 8 items must go in memory.
+;; 4x2 struct fields = 8 items so it goes in a block.
+
+define [ 4 x %T_STRUCT_SAMEM ] @array_of_struct_8_fields() {
+; CHECK-LABEL: array_of_struct_8_fields:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  movi d4, #0000000000000000
+; CHECK-NEXT:  movi d5, #0000000000000000
+; CHECK-NEXT:  movi d6, #0000000000000000
+; CHECK-NEXT:  movi d7, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 4 x %T_STRUCT_SAMEM ] zeroinitializer
+}
+
+;; 5x2 fields = 10 so it is returned in memory.
+
+define [ 5 x %T_STRUCT_SAMEM ] @array_of_struct_in_memory() {
+; CHECK-LABEL: array_of_struct_in_memory:
+; CHECK:       movi    v0.2d, #0000000000000000
+; CHECK-NEXT:  stp     q0, q0, [x8, #48]
+; CHECK-NEXT:  stp     q0, q0, [x8, #16]
+; CHECK-NEXT:  str     q0, [x8]
+; CHECK-NEXT:  ret
+  ret [ 5 x %T_STRUCT_SAMEM ] zeroinitializer
+}
+
+;; A struct whose field is an array.
+%T_STRUCT_ARRAYM = type { [ 2 x double ]};
+
+define %T_STRUCT_ARRAYM @struct_array_field() {
+; CHECK-LABEL: struct_array_field:
+; CHECK:       movi    d0, #0000000000000000
+; CHECK-NEXT:  movi    d1, #0000000000000000
+; CHECK-NEXT:  ret
+  ret %T_STRUCT_ARRAYM zeroinitializer
+}
+
+define [ 1 x %T_STRUCT_ARRAYM ] @array_of_struct_array_field() {
+; CHECK-LABEL: array_of_struct_array_field:
+; CHECK:       movi    d0, #0000000000000000
+; CHECK-NEXT:  movi    d1, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_STRUCT_ARRAYM ] zeroinitializer
+}
+
+define [ 2 x %T_STRUCT_ARRAYM ] @array_of_struct_array_field_2() {
+; CHECK-LABEL: array_of_struct_array_field_2:
+; CHECK:       movi    d0, #0000000000000000
+; CHECK-NEXT:  movi    d1, #0000000000000000
+; CHECK-NEXT:  movi    d2, #0000000000000000
+; CHECK-NEXT:  movi    d3, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_STRUCT_ARRAYM ] zeroinitializer
+}
+
+;; All non-aggregate fields must have the same type, all through the
+;; overall aggreagate. This is false here because of the i32.
+%T_NESTED_STRUCT_DIFFM = type {
+  [ 1 x { { double, double } } ],
+  [ 1 x { { double, i32 } } ]
+};
+
+define %T_NESTED_STRUCT_DIFFM @struct_nested_different_field_types() {
+; CHECK-LABEL: struct_nested_different_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK:       movi d1, #0000000000000000
+; CHECK:       movi d2, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  ret
+  ret %T_NESTED_STRUCT_DIFFM zeroinitializer
+}
+
+define [ 1 x %T_NESTED_STRUCT_DIFFM ] @array_of_struct_nested_different_field_types() {
+; CHECK-LABEL: array_of_struct_nested_different_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK:       movi d1, #0000000000000000
+; CHECK:       movi d2, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_NESTED_STRUCT_DIFFM ] zeroinitializer
+}
+
+define [ 2 x %T_NESTED_STRUCT_DIFFM ] @array_of_struct_nested_different_field_types_2() {
+; CHECK-LABEL: array_of_struct_nested_different_field_types_2:
+; CHECK:       movi d0, #0000000000000000
+; CHECK:       movi d1, #0000000000000000
+; CHECK:       movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  movi d4, #0000000000000000
+; CHECK-NEXT:  movi d5, #0000000000000000
+; CHECK-NEXT:  w0, wzr
+; CHECK-NEXT:  w1, wzr
+; CHECK-NEXT:  ret
+  ret [ 2 x %T_NESTED_STRUCT_DIFFM ] zeroinitializer
+}
+
+;; All fields here are the same type, more nesting to stress the recursive walk.
+%T_NESTED_STRUCT_SAMEM = type {
+  { { double} },
+  { [ 2 x { double, double } ] }
+};
+
+define %T_NESTED_STRUCT_SAMEM @struct_nested_same_field_types() {
+; CHECK-LABEL: struct_nested_same_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  movi d4, #0000000000000000
+; CHECK-NEXT:  ret
+  ret %T_NESTED_STRUCT_SAMEM zeroinitializer
+}
+
+define [ 1 x %T_NESTED_STRUCT_SAMEM ] @array_of_struct_nested_same_field_types() {
+; CHECK-LABEL: array_of_struct_nested_same_field_types:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  movi d2, #0000000000000000
+; CHECK-NEXT:  movi d3, #0000000000000000
+; CHECK-NEXT:  movi d4, #0000000000000000
+; CHECK-NEXT:  ret
+  ret [ 1 x %T_NESTED_STRUCT_SAMEM ] zeroinitializer
+}
+
+;; 2 x (1 + (2 x 2)) = 10 so this is returned in memory
+define [ 2 x %T_NESTED_STRUCT_SAMEM ] @array_of_struct_nested_same_field_types_2() {
+; CHECK-LABEL: array_of_struct_nested_same_field_types_2:
+; CHECK:      movi    v0.2d, #0000000000000000
+; CHECK-NEXT: stp     q0, q0, [x8, #48]
+; CHECK-NEXT: stp     q0, q0, [x8, #16]
+; CHECK-NEXT: str     q0, [x8]
+; CHECK-NEXT: ret
+  ret [ 2 x %T_NESTED_STRUCT_SAMEM ] zeroinitializer
+}
+
+;; Check combinations of call, return and argument passing
+
+%T_IN_BLOCK = type [ 2 x { double, { double, double } } ]
+
+define %T_IN_BLOCK @return_in_block() {
+; CHECK-LABEL: return_in_block:
+; CHECK:      movi d0, #0000000000000000
+; CHECK-NEXT: movi d1, #0000000000000000
+; CHECK-NEXT: movi d2, #0000000000000000
+; CHECK-NEXT: movi d3, #0000000000000000
+; CHECK-NEXT: movi d4, #0000000000000000
+; CHECK-NEXT: movi d5, #0000000000000000
+; CHECK-NEXT: ret
+  ret %T_IN_BLOCK zeroinitializer
+}
+
+@in_block_store = dso_local global %T_IN_BLOCK zeroinitializer, align 8
+
+define void @caller_in_block() {
+; CHECK-LABEL: caller_in_block:
+; CHECK: bl   return_in_block
+; CHECK-NEXT: adrp x8, in_block_store
+; CHECK-NEXT: add x8, x8, :lo12:in_block_store
+; CHECK-NEXT: stp d0, d1, [x8]
+; CHECK-NEXT: stp d2, d3, [x8, #16]
+; CHECK-NEXT: stp d4, d5, [x8, #32]
+; CHECK-NEXT: ldr x30, [sp], #16
+; CHECK-NEXT: ret
+  %1 = call %T_IN_BLOCK @return_in_block()
+  store %T_IN_BLOCK %1, %T_IN_BLOCK* @in_block_store
+  ret void
+}
+
+define void @callee_in_block(%T_IN_BLOCK %a) {
+; CHECK-LABEL: callee_in_block:
+; CHECK:      adrp x8, in_block_store
+; CHECK-NEXT: add x8, x8, :lo12:in_block_store
+; CHECK-NEXT: stp d4, d5, [x8, #32]
+; CHECK-NEXT: stp d2, d3, [x8, #16]
+; CHECK-NEXT: stp d0, d1, [x8]
+; CHECK-NEXT: ret
+  store %T_IN_BLOCK %a, %T_IN_BLOCK* @in_block_store
+  ret void
+}
+
+define void @argument_in_block() {
+; CHECK-LABEL: argument_in_block:
+; CHECK:      adrp x8, in_block_store
+; CHECK-NEXT: add x8, x8, :lo12:in_block_store
+; CHECK-NEXT: ldp d4, d5, [x8, #32]
+; CHECK-NEXT: ldp d2, d3, [x8, #16]
+; CHECK-NEXT: ldp d0, d1, [x8]
+; CHECK-NEXT: bl callee_in_block
+  %1 = load %T_IN_BLOCK, %T_IN_BLOCK* @in_block_store
+  call void @callee_in_block(%T_IN_BLOCK %1)
+  ret void
+}
+
+%T_IN_MEMORY = type [ 3 x { double, { double, double } } ]
+
+define %T_IN_MEMORY @return_in_memory() {
+; CHECK-LABEL: return_in_memory:
+; CHECK:       movi v0.2d, #0000000000000000
+; CHECK-NEXT:  str xzr, [x8, #64]
+; CHECK-NEXT:  stp q0, q0, [x8, #32]
+; CHECK-NEXT:  stp q0, q0, [x8]
+; CHECK-NEXT:  ret
+  ret %T_IN_MEMORY zeroinitializer
+}
+
+@in_memory_store = dso_local global %T_IN_MEMORY zeroinitializer, align 8
+
+define void @caller_in_memory() {
+; CHECK-LABEL: caller_in_memory:
+; CHECK:      add     x8, sp, #8
+; CHECK-NEXT: bl      return_in_memory
+; CHECK-NEXT: ldr     d0, [sp, #72]
+; CHECK-NEXT: ldur    q1, [sp, #24]
+; CHECK-NEXT: ldur    q2, [sp, #8]
+; CHECK-NEXT: ldur    q3, [sp, #56]
+; CHECK-NEXT: ldur    q4, [sp, #40]
+; CHECK-NEXT: ldr     x30, [sp, #80]
+; CHECK-NEXT: adrp    x8, in_memory_store
+; CHECK-NEXT: add     x8, x8, :lo12:in_memory_store
+; CHECK-NEXT: stp     q2, q1, [x8]
+; CHECK-NEXT: stp     q4, q3, [x8, #32]
+; CHECK-NEXT: str     d0, [x8, #64]
+; CHECK-NEXT: add     sp, sp, #96
+; CHECK-NEXT: ret
+  %1 = call %T_IN_MEMORY @return_in_memory()
+  store %T_IN_MEMORY %1, %T_IN_MEMORY* @in_memory_store
+  ret void
+}
+
+define void @callee_in_memory(%T_IN_MEMORY %a) {
+; CHECK-LABEL: callee_in_memory:
+; CHECK:      ldp     q0, q1, [sp, #32]
+; CHECK-NEXT: ldr     d2, [sp, #64]
+; CHECK-NEXT: ldp     q3, q4, [sp]
+; CHECK-NEXT: adrp    x8, in_memory_store
+; CHECK-NEXT: add     x8, x8, :lo12:in_memory_store
+; CHECK-NEXT: str     d2, [x8, #64]
+; CHECK-NEXT: stp     q0, q1, [x8, #32]
+; CHECK-NEXT: stp     q3, q4, [x8]
+; CHECK-NEXT: ret
+  store %T_IN_MEMORY %a, %T_IN_MEMORY* @in_memory_store
+  ret void
+}
+
+define void @argument_in_memory() {
+; CHECK-LABEL: argument_in_memory:
+; CHECK:      adrp    x8, in_memory_store
+; CHECK-NEXT: add     x8, x8, :lo12:in_memory_store
+; CHECK-NEXT: ldp     q0, q1, [x8]
+; CHECK-NEXT: ldp     q2, q3, [x8, #32]
+; CHECK-NEXT: ldr     d4, [x8, #64]
+; CHECK-NEXT: stp     q0, q1, [sp]
+; CHECK-NEXT: stp     q2, q3, [sp, #32]
+; CHECK-NEXT: str     d4, [sp, #64]
+; CHECK-NEXT: bl      callee_in_memory
+  %1 = load %T_IN_MEMORY, %T_IN_MEMORY* @in_memory_store
+  call void @callee_in_memory(%T_IN_MEMORY %1)
+  ret void
+}
+
+%T_NO_BLOCK = type [ 2 x { double, { i32 } } ]
+
+define %T_NO_BLOCK @return_no_block() {
+; CHECK-LABEL: return_no_block:
+; CHECK:       movi d0, #0000000000000000
+; CHECK-NEXT:  movi d1, #0000000000000000
+; CHECK-NEXT:  mov w0, wzr
+; CHECK-NEXT:  mov w1, wzr
+; CHECK-NEXT:  ret
+  ret %T_NO_BLOCK zeroinitializer
+}
+
+@no_block_store = dso_local global %T_NO_BLOCK zeroinitializer, align 8
+
+define void @caller_no_block() {
+; CHECK-LABEL: caller_no_block:
+; CHECK:       bl return_no_block
+; CHECK-NEXT:  adrp x8, no_block_store
+; CHECK-NEXT:  add x8, x8, :lo12:no_block_store
+; CHECK-NEXT:  str d0, [x8]
+; CHECK-NEXT:  str w0, [x8, #8]
+; CHECK-NEXT:  str d1, [x8, #16]
+; CHECK-NEXT:  str w1, [x8, #24]
+; CHECK-NEXT:  ldr x30, [sp], #16
+; CHECK-NEXT:  ret
+  %1 = call %T_NO_BLOCK @return_no_block()
+  store %T_NO_BLOCK %1, %T_NO_BLOCK* @no_block_store
+  ret void
+}
+
+define void @callee_no_block(%T_NO_BLOCK %a) {
+; CHECK-LABEL: callee_no_block:
+; CHECK:       adrp x8, no_block_store
+; CHECK-NEXT:  add x8, x8, :lo12:no_block_store
+; CHECK-NEXT:  str w1, [x8, #24]
+; CHECK-NEXT:  str d1, [x8, #16]
+; CHECK-NEXT:  str w0, [x8, #8]
+; CHECK-NEXT:  str d0, [x8]
+; CHECK-NEXT:  ret
+  store %T_NO_BLOCK %a, %T_NO_BLOCK* @no_block_store
+  ret void
+}
+
+define void @argument_no_block() {
+; CHECK-LABEL: argument_no_block:
+; CHECK:       adrp x8, no_block_store
+; CHECK-NEXT:  add x8, x8, :lo12:no_block_store
+; CHECK-NEXT:  ldr w1, [x8, #24]
+; CHECK-NEXT:  ldr d1, [x8, #16]
+; CHECK-NEXT:  ldr w0, [x8, #8]
+; CHECK-NEXT:  ldr d0, [x8]
+; CHECK-NEXT:  bl callee_no_block
+; CHECK-NEXT:  ldr x30, [sp], #16
+; CHECK-NEXT:  ret
+  %1 = load %T_NO_BLOCK, %T_NO_BLOCK* @no_block_store
+  call void @callee_no_block(%T_NO_BLOCK %1)
+  ret void
+}