[AArch64][CodeGen] Fixing stack alignment of HFA arguments on AArch64 PCS

[pratlucas]

Summary

Properly complying with AArch64 PCS on the handling of over-aligned HFA
arguments when those are placed on the stack. AAPCS64 specifies that the
stacked argument address should be rounded up to the Natural Alignment
of the argument before the argument is copied to memory.

Over alignment information extracted from language attributes on clang
was not properly propagated to the backend for those arguments, as it
does not map to the backend's base type alignments. As the standard also
specifies that, when placed in registers, a single FP register should be
allocated for each individual HFA element, type coercion is no suitable
for capturing the alignment constraints.

This patch fixes the alignment of these arguments by capturing their
stack alignment requirements in an IR argument attribute, making sure
this information is available for the calling convention lowering stage.

[pratlucas]

Clang-format.

[ostannard]

I've not looked at the code in detail yet, but I think this still gets the ABI wrong for this example:

typedef struct {
  __attribute__ ((__aligned__(32))) double v[4];
} TYPE1;

double func(double d0, double d1, double d2, double d3,
            double d4, double d5, double d6, double d7,
            float stk0, TYPE1 stk1) {
  return stk1.v[0];
}

The ABI says (https://github.com/ARM-software/abi-aa/blob/master/aapcs64/aapcs64.rst, rule B.5):

If the argument is an alignment adjusted type its value is passed as a copy of the actual value. The copy will have an alignment defined as follows.
* For a Fundamental Data Type, the alignment is the natural alignment of that type, after any promotions.
* For a Composite Type, the alignment of the copy will have 8-byte alignment if its natural alignment is <= 8 and 16-byte alignment if its natural alignment is >= 16.
The alignment of the copy is used for applying marshaling rules.

This means that stk1 should be passed as a copy with alignment 16 bytes, putting it at sp+16. GCC does this, clang without this patch passes it at sp+8, and clang with this patch passes it at sp+32.

[pratlucas]

I believe rule B.2 from the AAPCS64 should take precedence over rule B.5 for HFA arguments:

B.2	If the argument type is an HFA or an HVA, then the argument is used unmodified.

[pratlucas]

Ping.

[rnk]

In D75903#1914715, @ostannard wrote:

This means that stk1 should be passed as a copy with alignment 16 bytes, putting it at sp+16. GCC does this, clang without this patch passes it at sp+8, and clang with this patch passes it at sp+32.

MSVC puts it at sp+8: https://gcc.godbolt.org/z/aAku9j

They allegedly follow this same document. Either way, we need to remain compatible.

Given that neither GCC nor MSVC do things they way you are proposing, are you sure this is correct? What compiler does this change make us more compatible with?

I don't believe there is an issue from the user's standpoint, since Clang copies the entire argument into an appropriately aligned alloca. If it is address-taken and stored to, there will not be any faults.

[pratlucas]

From the AAPCS64's Parameter Passing Rules section (https://github.com/ARM-software/abi-aa/blob/master/aapcs64/aapcs64.rst#642parameter-passing-rules), I believe the proposed handling is correct. The HFA related rules described in this section are:

Stage B – Pre-padding and extension of arguments
[...]
B.2 	If the argument type is an HFA or an HVA, then the argument is used unmodified.
[...]

Stage C – Assignment of arguments to registers and stack
[...]
C.2 	If the argument is an HFA or an HVA and there are sufficient unallocated SIMD and Floating-point registers (NSRN + number of members <= 8), then the argument is allocated to SIMD and Floating-point Registers (with one register per member of the HFA or HVA). The NSRN is incremented by the number of registers used. The argument has now been allocated.
C.3 	If the argument is an HFA or an HVA then the NSRN is set to 8 and the size of the argument is rounded up to the nearest multiple of 8 bytes.
C.4 	If the argument is an HFA, an HVA, a Quad-precision Floating-point or Short Vector Type then the NSAA is rounded up to the larger of 8 or the Natural Alignment of the argument’s type.
[...]

As per rule C.4, the argument should be allocated on the stack address rounded to the larger of 8 and its Natural Alignment, which is 32 according to what is specified by the Composite Types rules in sectoin 5.6 of that same document (https://github.com/ARM-software/abi-aa/blob/master/aapcs64/aapcs64.rst#composite-types):

5.6   Composite Types
[...]
- The natural alignment of a composite type is the maximum of each of the member alignments of the 'top-level' members of the composite type i.e. before any alignment adjustment of the entire composite is applied

In regards to the compatibility with other compilers, I'm not sure that following what seems to be an uncompliant behavior would be the best way to proceed. @rnk and @ostannard, what would be your take on this?

[rnk]

In D75903#1963382, @pratlucas wrote:

In regards to the compatibility with other compilers, I'm not sure that following what seems to be an uncompliant behavior would be the best way to proceed. @rnk and @ostannard, what would be your take on this?

I don't have any familiarity with the prevailing practices for ARM ABI compatibility, so I couldn't say. It might be worth checking in with stakeholders from the other compilers, i.e. file bugs against both compilers and ask for an opinion.

llvm/docs/LangRef.rst

1220

This seems like you are introducing a new meaning to alignstack, which according to the comments, only affects function SP alignment, not parameter alignment.

I'm assuming the reason you can't use the regular align attribute is that it is overloaded to mean two things: the alignment of the pointer when applied to a pointer, and the alignment of the argument memory when that pointer argument is marked byval. If you want to resolve this ambiguity, it seems like something that should be discussed on llvm-dev with a wider audience.

[chill]

llvm/docs/LangRef.rst

1220

Sorry, I couldn't quite get it, do you suggest we should be using the align attribute instead of alignstack, if there are no
(major) objections on the llvm-dev list?

It certainly makes sense to me to use align as it already pertains to individual argument alignment (even though it's for pointers only now).

[rnk]

llvm/docs/LangRef.rst

1220

Mostly I think I meant that this will be a big change in the meaning of either the align or the alignstack attributes, and that should be hashed out on llvm-dev.

Right now align is kind of a hybrid between an optimization annotation attribute, like dereferenceable or nonnull, and an ABI attribute, like byval or inreg. When align is used with byval, it affects argument memory layout. When byval is not present, it is just an optimizer hint. IMO, ideally, we should separate those two roles.

I should be able to control the alignment of the memory used to pass a pointer argument, at the same time that I annotate which low bits of the pointer are known to be zero.

clang/include/clang/CodeGen/CGFunctionInfo.h

private:
   bool PaddingInReg : 1;
   bool InAllocaSRet : 1;    // isInAlloca()
   bool InAllocaIndirect : 1;// isInAlloca()
   bool IndirectByVal : 1;   // isIndirect()
   bool IndirectRealign : 1; // isIndirect()
   bool SRetAfterThis : 1;   // isIndirect()
   bool InReg : 1;           // isDirect() || isExtend() || isIndirect()
   bool CanBeFlattened: 1;   // isDirect()
+  bool HasStackAlign: 1;    // isDirect()
   bool SignExt : 1;         // isExtend()
 
   bool canHavePaddingType() const {
     return isDirect() || isExtend() || isIndirect() || isExpand();
   }
   void setPaddingType(llvm::Type *T) {
     assert(canHavePaddingType());
     PaddingType = T;
   }
 
   void setUnpaddedCoerceToType(llvm::Type *T) {
     assert(isCoerceAndExpand());
     UnpaddedCoerceAndExpandType = T;
   }
 
 public:
   ABIArgInfo(Kind K = Direct)
       : TypeData(nullptr), PaddingType(nullptr), DirectOffset(0), TheKind(K),
         PaddingInReg(false), InAllocaSRet(false), InAllocaIndirect(false),
         IndirectByVal(false), IndirectRealign(false), SRetAfterThis(false),
         InReg(false), CanBeFlattened(false), SignExt(false) {}
 
   static ABIArgInfo getDirect(llvm::Type *T = nullptr, unsigned Offset = 0,
                               llvm::Type *Padding = nullptr,
-                              bool CanBeFlattened = true) {
+                              bool CanBeFlattened = true,
+                              bool HasStackAlign = false) {
     auto AI = ABIArgInfo(Direct);
     AI.setCoerceToType(T);
     AI.setPaddingType(Padding);
     AI.setDirectOffset(Offset);
     AI.setCanBeFlattened(CanBeFlattened);
+    AI.setStackAlign(HasStackAlign);
     return AI;
   }
   static ABIArgInfo getDirectInReg(llvm::Type *T = nullptr) {
     auto AI = getDirect(T);
     AI.setInReg(true);
     return AI;
   }
 
bool getCanBeFlattened() const {
     return CanBeFlattened;
   }
 
   void setCanBeFlattened(bool Flatten) {
     assert(isDirect() && "Invalid kind!");
     CanBeFlattened = Flatten;
   }
 
+  bool hasStackAlign() const {
+    return HasStackAlign;
+  }
+
+  void setStackAlign(bool HasAlign) {
+    assert(isDirect() && "Invalid kind!");
+    HasStackAlign = HasAlign;
+  }
+
   void dump() const;
 };
 
 /// A class for recording the number of arguments that a function
 /// signature requires.
 class RequiredArgs {
   /// The number of required arguments, or ~0 if the signature does
   /// not permit optional arguments.

clang/lib/CodeGen/CGCall.cpp

case ABIArgInfo::Extend:
       else
         Attrs.addAttribute(llvm::Attribute::ZExt);
       LLVM_FALLTHROUGH;
     case ABIArgInfo::Direct:
       if (ArgNo == 0 && FI.isChainCall())
         Attrs.addAttribute(llvm::Attribute::Nest);
       else if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
+      if (AI.hasStackAlign())
+        Attrs.addStackAlignmentAttr(
+            getContext().getTypeAlignInChars(ParamType).getQuantity());
       break;
 
     case ABIArgInfo::Indirect: {
       if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
 
       if (AI.getIndirectByVal())
         Attrs.addByValAttr(getTypes().ConvertTypeForMem(ParamType));

clang/lib/CodeGen/TargetInfo.cpp

if (IsEmpty && Size == 0)
       return ABIArgInfo::getIgnore();
     return ABIArgInfo::getDirect(llvm::Type::getInt8Ty(getVMContext()));
   }
 
   // Homogeneous Floating-point Aggregates (HFAs) need to be expanded.
   const Type *Base = nullptr;
   uint64_t Members = 0;
   if (isHomogeneousAggregate(Ty, Base, Members)) {
+    bool NeedsStackAlignment = getContext().getTypeAlignInChars(Ty) !=
+                               getContext().getTypeAlignInChars(Base);
     return ABIArgInfo::getDirect(
-        llvm::ArrayType::get(CGT.ConvertType(QualType(Base, 0)), Members));
+        llvm::ArrayType::get(CGT.ConvertType(QualType(Base, 0)), Members),
+        /*Offset=*/0, /*Padding=*/nullptr, /*CanBeFlattened=*/true,
+        NeedsStackAlignment);
   }
 
   // Aggregates <= 16 bytes are passed directly in registers or on the stack.
   if (Size <= 128) {
     // On RenderScript, coerce Aggregates <= 16 bytes to an integer array of
     // same size and alignment.
     if (getTarget().isRenderScriptTarget()) {
       return coerceToIntArray(Ty, getContext(), getVMContext());

clang/test/CodeGen/aarch64-args-hfa.c

+// RUN: %clang_cc1 -triple aarch64-none-eabi -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple arm64-apple-ios7.0 -target-abi darwinpcs -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple aarch64-linux-gnu -emit-llvm -o - -x c %s | FileCheck %s
+
+// CHECK: %struct.hfa_align = type { [2 x double] }
+typedef struct {
+  __attribute__((__aligned__(16))) double v[2];
+} hfa_align;
+
+// CHECK: define double @test_hfa_align_arg([2 x double] alignstack(16) %h.coerce)
+double test_hfa_align_arg(hfa_align h) {
+  return h.v[0];
+}
+
+// CHECK: define double @test_hfa_align_call()
+// CHECK: %call = call double @test_hfa_align_arg([2 x double] alignstack(16) %1)
+double test_hfa_align_call() {
+  hfa_align h1 = {1.0, 2.0};
+  return test_hfa_align_arg(h1);
+}
+

clang/test/CodeGen/arm64-arguments.c

 struct s35
 {
    float v[4]; //Testing HFA.
 } __attribute__((aligned(16)));
 typedef struct s35 s35_with_align;
 
 typedef __attribute__((neon_vector_type(4))) float float32x4_t;
 float32x4_t f35(int i, s35_with_align s1, s35_with_align s2) {
-// CHECK: define <4 x float> @f35(i32 %i, [4 x float] %s1.coerce, [4 x float] %s2.coerce)
+// CHECK: define <4 x float> @f35(i32 %i, [4 x float] alignstack(16) %s1.coerce, [4 x float] alignstack(16) %s2.coerce)
 // CHECK: %s1 = alloca %struct.s35, align 16
 // CHECK: %s2 = alloca %struct.s35, align 16
 // CHECK: %[[a:.*]] = bitcast %struct.s35* %s1 to <4 x float>*
 // CHECK: load <4 x float>, <4 x float>* %[[a]], align 16
 // CHECK: %[[b:.*]] = bitcast %struct.s35* %s2 to <4 x float>*
 // CHECK: load <4 x float>, <4 x float>* %[[b]], align 16
   float32x4_t v = vaddq_f32(*(float32x4_t *)&s1,
                             *(float32x4_t *)&s2);

llvm/docs/LangRef.rst

 
 ``immarg``
     This indicates the parameter is required to be an immediate
     value. This must be a trivial immediate integer or floating-point
     constant. Undef or constant expressions are not valid. This is
     only valid on intrinsic declarations and cannot be applied to a
     call site or arbitrary function.
 
+``alignstack(<n>)``
+    This indicates the alignment that should be considered by the backend when
+    assigning this parameter to a stack slot during calling convention
+    lowering. The enforcement of the specified alignment is target-dependent,
+    as target-specific calling convention rules may override this value. This
+    attribute serves the purpose of carrying language specific alignment
+    information that is not mapped to base types in the backend (for example,
+    over-alignment specification through language attributes).
+
 .. _gc:
 
 Garbage Collector Strategy Names
 --------------------------------
 
 Each function may specify a garbage collector strategy name, which is simply a
 string:
 

llvm/include/llvm/CodeGen/TargetCallingConv.h

private:
     unsigned IsSwiftSelf : 1;  ///< Swift self parameter
     unsigned IsSwiftError : 1; ///< Swift error parameter
     unsigned IsCFGuardTarget : 1; ///< Control Flow Guard target
     unsigned IsHva : 1;        ///< HVA field for
     unsigned IsHvaStart : 1;   ///< HVA structure start
     unsigned IsSecArgPass : 1; ///< Second argument
     unsigned ByValAlign : 4;   ///< Log 2 of byval alignment
     unsigned OrigAlign : 5;    ///< Log 2 of original alignment
+    unsigned StackAlign : 3;    ///< Log 2 of stack slot alignment
     unsigned IsInConsecutiveRegsLast : 1;
     unsigned IsInConsecutiveRegs : 1;
     unsigned IsCopyElisionCandidate : 1; ///< Argument copy elision candidate
     unsigned IsPointer : 1;
 
     unsigned ByValSize; ///< Byval struct size
 
     unsigned PointerAddrSpace; ///< Address space of pointer argument
 
   public:
     ArgFlagsTy()
         : IsZExt(0), IsSExt(0), IsInReg(0), IsSRet(0), IsByVal(0), IsNest(0),
           IsReturned(0), IsSplit(0), IsInAlloca(0), IsSplitEnd(0),
           IsSwiftSelf(0), IsSwiftError(0), IsCFGuardTarget(0), IsHva(0),
           IsHvaStart(0), IsSecArgPass(0), ByValAlign(0), OrigAlign(0),
-          IsInConsecutiveRegsLast(0), IsInConsecutiveRegs(0),
+          StackAlign(0), IsInConsecutiveRegsLast(0), IsInConsecutiveRegs(0),
           IsCopyElisionCandidate(0), IsPointer(0), ByValSize(0),
           PointerAddrSpace(0) {
       static_assert(sizeof(*this) == 3 * sizeof(unsigned), "flags are too big");
     }
 
     bool isZExt() const { return IsZExt; }
     void setZExt() { IsZExt = 1; }
 
unsigned getOrigAlign() const {
       MaybeAlign A = decodeMaybeAlign(OrigAlign);
       return A ? A->value() : 0;
     }
     void setOrigAlign(Align A) {
       OrigAlign = encode(A);
       assert(getOrigAlign() == A.value() && "bitfield overflow");
     }
 
+    unsigned getStackAlign() const {
+      MaybeAlign A = decodeMaybeAlign(StackAlign);
+      return A ? A->value() : 0;
+    }
+    void setStackAlign(Align A) {
+      StackAlign = encode(A);
+      assert(getStackAlign() == A.value() && "bitfield overflow");
+    }
+
     unsigned getByValSize() const { return ByValSize; }
     void setByValSize(unsigned S) { ByValSize = S; }
 
     unsigned getPointerAddrSpace() const { return PointerAddrSpace; }
     void setPointerAddrSpace(unsigned AS) { PointerAddrSpace = AS; }
 };
 
   /// InputArg - This struct carries flags and type information about a

llvm/include/llvm/CodeGen/TargetLowering.h

public:
     bool IsSRet : 1;
     bool IsNest : 1;
     bool IsByVal : 1;
     bool IsInAlloca : 1;
     bool IsReturned : 1;
     bool IsSwiftSelf : 1;
     bool IsSwiftError : 1;
     bool IsCFGuardTarget : 1;
+    MaybeAlign StackAlignment;
     uint16_t Alignment = 0;
     Type *ByValType = nullptr;
 
     ArgListEntry()
         : IsSExt(false), IsZExt(false), IsInReg(false), IsSRet(false),
           IsNest(false), IsByVal(false), IsInAlloca(false), IsReturned(false),
-          IsSwiftSelf(false), IsSwiftError(false), IsCFGuardTarget(false) {}
+          IsSwiftSelf(false), IsSwiftError(false), IsCFGuardTarget(false),
+          StackAlignment() {}
 
     void setAttributes(const CallBase *Call, unsigned ArgIdx);
 
     void setAttributes(ImmutableCallSite *CS, unsigned ArgIdx) {
       return setAttributes(cast<CallBase>(CS->getInstruction()), ArgIdx);
     }
   };
   using ArgListTy = std::vector<ArgListEntry>;

llvm/include/llvm/IR/Argument.h

public:
   /// If this is a byval or inalloca argument, return its alignment.
   /// FIXME: Remove this function once transition to Align is over.
   /// Use getParamAlign() instead.
   unsigned getParamAlignment() const;
 
   /// If this is a byval or inalloca argument, return its alignment.
   MaybeAlign getParamAlign() const;
 
+  /// Return argument's stack alignment if specified
+  MaybeAlign getParamStackAlign() const;
+
   /// If this is a byval argument, return its type.
   Type *getParamByValType() const;
 
   /// Return true if this argument has the nest attribute.
   bool hasNestAttr() const;
 
   /// Return true if this argument has the noalias attribute.
   bool hasNoAliasAttr() const;

llvm/include/llvm/IR/Attributes.h

public:
   }
 
   /// Return the alignment of the return value.
   MaybeAlign getRetAlignment() const;
 
   /// Return the alignment for the specified function parameter.
   MaybeAlign getParamAlignment(unsigned ArgNo) const;
 
+  /// Return the stack alignment for the specified function parameter.
+  MaybeAlign getParamStackAlignment(unsigned ArgNo) const;
+
   /// Return the byval type for the specified function parameter.
   Type *getParamByValType(unsigned ArgNo) const;
 
   /// Get the stack alignment.
   MaybeAlign getStackAlignment(unsigned Index) const;
 
   /// Get the number of dereferenceable bytes (or zero if unknown).
   uint64_t getDereferenceableBytes(unsigned Index) const;

llvm/include/llvm/IR/Attributes.td

 def SafeStack : EnumAttr<"safestack">;
 
 /// Shadow Call Stack protection.
 def ShadowCallStack : EnumAttr<"shadowcallstack">;
 
 /// Sign extended before/after call.
 def SExt : EnumAttr<"signext">;
 
-/// Alignment of stack for function (3 bits)  stored as log2 of alignment with
-/// +1 bias 0 means unaligned (different from alignstack=(1)).
+/// Alignment of stack for function or argument (3 bits) stored as log2 of
+/// alignment with +1 bias 0 means unaligned (different from alignstack=(1)).
 def StackAlignment : EnumAttr<"alignstack">;
 
 /// Function can be speculated.
 def Speculatable : EnumAttr<"speculatable">;
 
 /// Stack protection.
 def StackProtect : EnumAttr<"ssp">;
 

llvm/include/llvm/IR/Function.h

if (const auto MA = getParamAlign(ArgNo))
       return MA->value();
     return 0;
   }
 
   MaybeAlign getParamAlign(unsigned ArgNo) const {
     return AttributeSets.getParamAlignment(ArgNo);
   }
 
+  MaybeAlign getParamStackAlign(unsigned ArgNo) const {
+    return AttributeSets.getParamStackAlignment(ArgNo);
+  }
+
   /// Extract the byval type for a parameter.
   Type *getParamByValType(unsigned ArgNo) const {
     Type *Ty = AttributeSets.getParamByValType(ArgNo);
     return Ty ? Ty : (arg_begin() + ArgNo)->getType()->getPointerElementType();
   }
 
   /// Extract the number of dereferenceable bytes for a call or
   /// parameter (0=unknown).

llvm/include/llvm/IR/InstrTypes.h

unsigned getParamAlignment(unsigned ArgNo) const {
     return 0;
   }
 
   /// Extract the alignment for a call or parameter (0=unknown).
   MaybeAlign getParamAlign(unsigned ArgNo) const {
     return Attrs.getParamAlignment(ArgNo);
   }
 
+  /// Extract the stack alignment for a call or parameter (0=unknown).
+  MaybeAlign getParamStackAlign(unsigned ArgNo) const {
+    return Attrs.getParamStackAlignment(ArgNo);
+  }
+
   /// Extract the byval type for a call or parameter.
   Type *getParamByValType(unsigned ArgNo) const {
     Type *Ty = Attrs.getParamByValType(ArgNo);
     return Ty ? Ty : getArgOperand(ArgNo)->getType()->getPointerElementType();
   }
 
   /// Extract the number of dereferenceable bytes for a call or
   /// parameter (0=unknown).

llvm/lib/AsmParser/LLParser.cpp

while (true) {
     }
     case lltok::kw_align: {
       MaybeAlign Alignment;
       if (ParseOptionalAlignment(Alignment))
         return true;
       B.addAlignmentAttr(Alignment);
       continue;
     }
+    case lltok::kw_alignstack: {
+      unsigned Alignment;
+      if (ParseOptionalStackAlignment(Alignment))
+        return true;
+      B.addStackAlignmentAttr(Alignment);
+      continue;
+    }
     case lltok::kw_byval: {
       Type *Ty;
       if (ParseByValWithOptionalType(Ty))
         return true;
       B.addByValAttr(Ty);
       continue;
     }
     case lltok::kw_dereferenceable: {
while (true) {
     case lltok::kw_signext:         B.addAttribute(Attribute::SExt); break;
     case lltok::kw_sret:            B.addAttribute(Attribute::StructRet); break;
     case lltok::kw_swifterror:      B.addAttribute(Attribute::SwiftError); break;
     case lltok::kw_swiftself:       B.addAttribute(Attribute::SwiftSelf); break;
     case lltok::kw_writeonly:       B.addAttribute(Attribute::WriteOnly); break;
     case lltok::kw_zeroext:         B.addAttribute(Attribute::ZExt); break;
     case lltok::kw_immarg:          B.addAttribute(Attribute::ImmArg); break;
 
-    case lltok::kw_alignstack:
     case lltok::kw_alwaysinline:
     case lltok::kw_argmemonly:
     case lltok::kw_builtin:
     case lltok::kw_inlinehint:
     case lltok::kw_jumptable:
     case lltok::kw_minsize:
     case lltok::kw_naked:
     case lltok::kw_nobuiltin:

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

for (unsigned Value = 0, NumValues = ValueVTs.size(); Value != NumValues;
           FrameAlign = getByValTypeAlignment(ElementTy, DL);
         Flags.setByValAlign(Align(FrameAlign));
       }
       if (Args[i].IsNest)
         Flags.setNest();
       if (NeedsRegBlock)
         Flags.setInConsecutiveRegs();
       Flags.setOrigAlign(OriginalAlignment);
+      if (Args[i].StackAlignment)
+        Flags.setStackAlign(*Args[i].StackAlignment);
 
       MVT PartVT = getRegisterTypeForCallingConv(CLI.RetTy->getContext(),
                                                  CLI.CallConv, VT);
       unsigned NumParts = getNumRegistersForCallingConv(CLI.RetTy->getContext(),
                                                         CLI.CallConv, VT);
       SmallVector<SDValue, 4> Parts(NumParts);
       ISD::NodeType ExtendKind = ISD::ANY_EXTEND;
 
for (unsigned Value = 0, NumValues = ValueVTs.size();
         Flags.setNest();
       if (NeedsRegBlock)
         Flags.setInConsecutiveRegs();
       Flags.setOrigAlign(OriginalAlignment);
       if (ArgCopyElisionCandidates.count(&Arg))
         Flags.setCopyElisionCandidate();
       if (Arg.hasAttribute(Attribute::Returned))
         Flags.setReturned();
+      if (Arg.getParamStackAlign())
+        Flags.setStackAlign(*Arg.getParamStackAlign());
 
       MVT RegisterVT = TLI->getRegisterTypeForCallingConv(
           *CurDAG->getContext(), F.getCallingConv(), VT);
       unsigned NumRegs = TLI->getNumRegistersForCallingConv(
           *CurDAG->getContext(), F.getCallingConv(), VT);
       for (unsigned i = 0; i != NumRegs; ++i) {
         // For scalable vectors, use the minimum size; individual targets
         // are responsible for handling scalable vector arguments and

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

void TargetLoweringBase::ArgListEntry::setAttributes(const CallBase *Call,
   IsSRet = Call->paramHasAttr(ArgIdx, Attribute::StructRet);
   IsNest = Call->paramHasAttr(ArgIdx, Attribute::Nest);
   IsByVal = Call->paramHasAttr(ArgIdx, Attribute::ByVal);
   IsInAlloca = Call->paramHasAttr(ArgIdx, Attribute::InAlloca);
   IsReturned = Call->paramHasAttr(ArgIdx, Attribute::Returned);
   IsSwiftSelf = Call->paramHasAttr(ArgIdx, Attribute::SwiftSelf);
   IsSwiftError = Call->paramHasAttr(ArgIdx, Attribute::SwiftError);
   Alignment = Call->getParamAlignment(ArgIdx);
+  StackAlignment = Call->getParamStackAlign(ArgIdx);
   ByValType = nullptr;
   if (Call->paramHasAttr(ArgIdx, Attribute::ByVal))
     ByValType = Call->getParamByValType(ArgIdx);
 }
 
 /// Generate a libcall taking the given operands as arguments and returning a
 /// result of type RetVT.
 std::pair<SDValue, SDValue>

llvm/lib/IR/Attributes.cpp

 MaybeAlign AttributeList::getRetAlignment() const {
   return getAttributes(ReturnIndex).getAlignment();
 }
 
 MaybeAlign AttributeList::getParamAlignment(unsigned ArgNo) const {
   return getAttributes(ArgNo + FirstArgIndex).getAlignment();
 }
 
+MaybeAlign AttributeList::getParamStackAlignment(unsigned ArgNo) const {
+  return getAttributes(ArgNo + FirstArgIndex).getStackAlignment();
+}
+
 Type *AttributeList::getParamByValType(unsigned Index) const {
   return getAttributes(Index+FirstArgIndex).getByValType();
 }
 
 MaybeAlign AttributeList::getStackAlignment(unsigned Index) const {
   return getAttributes(Index).getStackAlignment();
 }
 

llvm/lib/IR/Function.cpp

unsigned Argument::getParamAlignment() const {
   return getParent()->getParamAlignment(getArgNo());
 }
 
 MaybeAlign Argument::getParamAlign() const {
   assert(getType()->isPointerTy() && "Only pointers have alignments");
   return getParent()->getParamAlign(getArgNo());
 }
 
+MaybeAlign Argument::getParamStackAlign() const {
+  return getParent()->getParamStackAlign(getArgNo());
+}
+
 Type *Argument::getParamByValType() const {
   assert(getType()->isPointerTy() && "Only pointers have byval types");
   return getParent()->getParamByValType(getArgNo());
 }
 
 uint64_t Argument::getDereferenceableBytes() const {
   assert(getType()->isPointerTy() &&
          "Only pointers have dereferenceable bytes");

llvm/lib/IR/Verifier.cpp

static bool isFuncOnlyAttr(Attribute::AttrKind Kind) {
   case Attribute::StackProtectReq:
   case Attribute::StackProtectStrong:
   case Attribute::SafeStack:
   case Attribute::ShadowCallStack:
   case Attribute::NoRedZone:
   case Attribute::NoImplicitFloat:
   case Attribute::Naked:
   case Attribute::InlineHint:
-  case Attribute::StackAlignment:
   case Attribute::UWTable:
   case Attribute::NonLazyBind:
   case Attribute::ReturnsTwice:
   case Attribute::SanitizeAddress:
   case Attribute::SanitizeHWAddress:
   case Attribute::SanitizeMemTag:
   case Attribute::SanitizeThread:
   case Attribute::SanitizeMemory:
static bool isFuncOnlyAttr(Attribute::AttrKind Kind) {
   }
   return false;
 }
 
 /// Return true if this is a function attribute that can also appear on
 /// arguments.
 static bool isFuncOrArgAttr(Attribute::AttrKind Kind) {
   return Kind == Attribute::ReadOnly || Kind == Attribute::WriteOnly ||
-         Kind == Attribute::ReadNone || Kind == Attribute::NoFree;
+         Kind == Attribute::ReadNone || Kind == Attribute::NoFree ||
+         Kind == Attribute::StackAlignment;
 }
 
 void Verifier::verifyAttributeTypes(AttributeSet Attrs, bool IsFunction,
                                     const Value *V) {
   for (Attribute A : Attrs) {
     if (A.isStringAttribute())
       continue;
 

llvm/lib/Target/AArch64/AArch64CallingConvention.cpp

 
 static bool finishStackBlock(SmallVectorImpl<CCValAssign> &PendingMembers,
                              MVT LocVT, ISD::ArgFlagsTy &ArgFlags,
                              CCState &State, unsigned SlotAlign) {
   unsigned Size = LocVT.getSizeInBits() / 8;
   const Align StackAlign =
       State.getMachineFunction().getDataLayout().getStackAlignment();
   const Align OrigAlign(ArgFlags.getOrigAlign());
-  const Align Align = std::min(OrigAlign, StackAlign);
+  const Align Alignment = std::min(OrigAlign, StackAlign);
+
+  if (ArgFlags.getStackAlign()) {
+    const Align ArgStackAlign(ArgFlags.getStackAlign());
+    State.AllocateStack(0, ArgStackAlign.value());
+  }
 
   for (auto &It : PendingMembers) {
     It.convertToMem(State.AllocateStack(
-        Size, std::max((unsigned)Align.value(), SlotAlign)));
+        Size, std::max((unsigned)Alignment.value(), SlotAlign)));
     State.addLoc(It);
     SlotAlign = 1;
   }
 
   // All pending members have now been allocated
   PendingMembers.clear();
   return true;
 }

llvm/test/Bitcode/compatibility.ll

 declare i8* @f.param.returned(i8* returned)
 ; CHECK: declare i8* @f.param.returned(i8* returned)
 declare void @f.param.nonnull(i8* nonnull)
 ; CHECK: declare void @f.param.nonnull(i8* nonnull)
 declare void @f.param.dereferenceable(i8* dereferenceable(4))
 ; CHECK: declare void @f.param.dereferenceable(i8* dereferenceable(4))
 declare void @f.param.dereferenceable_or_null(i8* dereferenceable_or_null(4))
 ; CHECK: declare void @f.param.dereferenceable_or_null(i8* dereferenceable_or_null(4))
+declare void @f.param.stack_align([2 x double] alignstack(16))
+; CHECK: declare void @f.param.stack_align([2 x double] alignstack(16))
 
 ; Functions -- unnamed_addr and local_unnamed_addr
 declare void @f.unnamed_addr() unnamed_addr
 ; CHECK: declare void @f.unnamed_addr() unnamed_addr
 declare void @f.local_unnamed_addr() local_unnamed_addr
 ; CHECK: declare void @f.local_unnamed_addr() local_unnamed_addr
 
 ; Functions -- fn Attrs (Function attributes)

llvm/test/CodeGen/AArch64/arm64-abi-hfa-args.ll

+; RUN: llc < %s -mtriple=arm64-none-eabi | FileCheck %s
+
+; Over-aligned HFA argument placed on register - one element per register
+define double @test_hfa_align_arg_reg([2 x double] alignstack(16) %h.coerce) local_unnamed_addr #0 {
+entry:
+; CHECK-LABEL: test_hfa_align_arg_reg:
+; CHECK-DAG: ret
+
+  %h.coerce.fca.0.extract = extractvalue [2 x double] %h.coerce, 0
+  ret double %h.coerce.fca.0.extract
+}
+
+; Call with over-aligned HFA argument placed on register - one element per register
+define double @test_hfa_align_call_reg() local_unnamed_addr #0 {
+entry:
+; CHECK-LABEL: test_hfa_align_call_reg:
+; CHECK-DAG: fmov d0, #1.00000000
+; CHECK-DAG: fmov d1, #2.00000000
+; CHECK-DAG: bl test_hfa_align_arg_reg
+; CHECK-DAG: ldr  x30, [sp], #16          // 8-byte Folded Reload
+; CHECK-DAG: ret
+
+  %call = call double @test_hfa_align_arg_reg([2 x double] alignstack(16) [double 1.000000e+00, double 2.000000e+00])
+  ret double %call
+}
+
+; Over-aligned HFA argument placed on stack - stack round up to alignment
+define double @test_hfa_align_arg_stack(double %d0, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, float %f, [2 x double] alignstack(16) %h.coerce) local_unnamed_addr #0 {
+entry:
+; CHECK-LABEL: test_hfa_align_arg_stack:
+; CHECK-DAG: ldr  d0, [sp, #16]
+; CHECK-DAG: ret
+
+  %h.coerce.fca.0.extract = extractvalue [2 x double] %h.coerce, 0
+  ret double %h.coerce.fca.0.extract
+}
+
+; Call with over-aligned HFA argument placed on stack - stack round up to alignment
+define double @test_hfa_align_call_stack() local_unnamed_addr #0 {
+entry:
+; CHECK-LABEL: test_hfa_align_call_stack:
+; CHECK-DAG: mov  x8, #4611686018427387904
+; CHECK-DAG: mov  x9, #4607182418800017408
+; CHECK-DAG: stp  x8, x30, [sp, #24]      // 8-byte Folded Spill
+; CHECK-DAG: str  x9, [sp, #16]
+; CHECK-DAG: bl test_hfa_align_arg
+; CHECK-DAG: ldr  x30, [sp, #32]          // 8-byte Folded Reload
+; CHECK-DAG: add  sp, sp, #48             // =48
+; CHECK-DAG: ret
+
+  %call = call double @test_hfa_align_arg_stack(double undef, double undef, double undef, double undef, double undef, double undef, double undef, double undef, float undef, [2 x double] alignstack(16) [double 1.000000e+00, double 2.000000e+00])
+  ret double %call
+}
+