[amdgpu] WIP variadics

[JonChesterfield]

Summary

Won't ship as is, testing in progress. Posting to collaborate with @jhuber6 on libc testing. snprintf passing at present. Commit message will change from this.

RFC is at https://discourse.llvm.org/t/rfc-desugar-variadics-codegen-for-new-targets-optimisation-for-existing/72854

I'm looking to implement this as an ABI lowering for amdgpu (possibly nvptx) so those backends don't have to consider variadic functions, and as an optimisation for x64/aarch64 etc.

[JonChesterfield]

The lowering pass is broadly right, missing a few edge cases.

libc/config/gpu/entrypoints.txt

86

^these try to build, but fail. I haven't managed to debug the libc cmake to work out what I'm missing

libc/test/src/__support/CMakeLists.txt

62–70

this one builds and passes

[JonChesterfield]

llvm/lib/CodeGen/ExpandVAIntrinsics.cpp

159 ↗

(On Diff #551373)

Cut this from the WIP patch as the draft is noisy. Ran some tests through x64 with a patched clang locally, expanding functions that pass that precondition and leaving others unchanged.

[JonChesterfield]

@arsenm suggested va_start/va_end should be addrspace aware, similar idea to https://reviews.llvm.org/D156666. Should be less invasive for the va intrinsics as they're not used so widely.

[arsenm]

llvm/lib/CodeGen/ExpandVAIntrinsics.cpp

38 ↗

(On Diff #551373)

Don't need

44–47 ↗

(On Diff #551373)

Don't understand the point of this

151 ↗

(On Diff #551373)

isDefinitionExact?

If this is just how the ABI is going to lower, I don't see why you need to worry about any such restrictions

181 ↗

(On Diff #551373)

Probably should defend against sret and the other ABI attributes

209 ↗

(On Diff #551373)

How is StructType::create not using Twine

215–217 ↗

(On Diff #551373)

What's wrong with just Builder.CreateAlloca?

217 ↗

(On Diff #551373)

what's wrong with the abi type alignment? why does the stack alignment matter?

226 ↗

(On Diff #551373)

alignTo shouldn't be difficult

229–230 ↗

(On Diff #551373)

just CreateAddrSpaceCast should work?

253 ↗

(On Diff #551373)

there are more CallBase types than call and invoke now. Can't you just mutate the call operand in place?

258 ↗

(On Diff #551373)

I'd assume all metadata would be preservable if this is just how the ABI works

272–276 ↗

(On Diff #551373)

no return after else,

llvm/test/CodeGen/Generic/expand-variadic-intrinsics.ll

3

-passes.

Also "generic" codegen tests are a fiction, use a real triple

49

Use named values

77

Needs some indirect variadic call tests

[arsenm]

llvm/test/CodeGen/Generic/expand-variadic-intrinsics.ll

77

Also some metadata and signext/zeroext preservation tests

77

Also a case where the user isn't the call operand

[JonChesterfield]

• Rename ExpandVAIntrinsics to DesugarVariadics

[arsenm]

libc/config/gpu/entrypoints.txt

84–85

Split of the libc stuff into a separate patch, the lowering pass should be a standalone change

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

22

Can you expand on the ABI requirements of this in the comment? If we make this be the way the ABI works for AMDGPU, we don't need any conditions. Should there be some control for the targets with established ABIs to apply this to internal functions?

38

Don't need cstdio

102–103

Can you use nontrivial types with this? Might be better to just start with memcpy

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

621

desugar is a weird name for a lowering pass

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

188

This is rather late, I'd think we'd be better off with an earlier run (e.g. as part of the initial module passes, along with sanitizers)

llvm/test/CodeGen/AMDGPU/unsupported-calls.ll

57–58

Deleted the wrong error?

[JonChesterfield]

llvm/lib/CodeGen/ExpandVAIntrinsics.cpp

215–217 ↗

(On Diff #551373)

I want to mark the callee parameter with alignment metadata so the pointer alignment can constant fold (todo, said folding isn't currently working very well).

The struct contains whatever the caller was passing which we don't know much about. If we call CreateAlloca, it gets whatever alignment suffices for those fields.

If those two don't line up nicely, you get bad times. I'm using the stack alignment from the target string as a heuristic for a reasonable alignment to use for things on the stack. For amdgpu that's 4, which means passing a double gets an dynamic realignment which doesn't optimise out, but otherwise it seems reasonable.

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

208–209

Should we go for a packed struct forced to align 4? There's no upside to stack values with > 4 align

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

297

Test the comdat? Weird that copyAttributesFrom seems to not cover it

339

Add a test with a def and decl in a global initializer, and with a constantexpr cast in the initializer. I think this should also handle blockaddress

[JonChesterfield]

Name candidates

• expand
• lower
• desugar
• transform

Lowering probably makes the most sense for the abi level apply to all functions, I like desugar to cover rewriting a subset of the graph

libc/config/gpu/entrypoints.txt

84–85

There's quite a lot of wip in this patch. Your feedback is welcome, but you could also wait for a cleaned up version with your name on the reviewer list if you prefer. The libc plumbing is keeping track of which parts are currently passing for collaboration with jhuber6

llvm/test/CodeGen/AMDGPU/unsupported-calls.ll

57–58

this is curious - the test is passing as written, maybe a quirk of running it with not llc

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

74–77

Can you use ptrmask?

142

This would be better as a pass parameter. Don't see much point to specifying individual functions by name

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

296

This might be missing copying the linkage

[arsenm]

llvm/lib/CodeGen/DesugarVariadics.cpp

145

I think you need to guard against calls to variadic intrinsics

clang/lib/CodeGen/Targets/AMDGPU.cpp

for (auto &Arg : FI.arguments()) {
     } else {
       Arg.info = classifyArgumentType(Arg.type, NumRegsLeft);
     }
   }
 }
 
 Address AMDGPUABIInfo::EmitVAArg(CodeGenFunction &CGF, Address VAListAddr,
                                  QualType Ty) const {
-  llvm_unreachable("AMDGPU does not support varargs");
+  return EmitVAArgInstr(CGF, VAListAddr, Ty, ABIArgInfo::getDirect());
 }
 
 ABIArgInfo AMDGPUABIInfo::classifyReturnType(QualType RetTy) const {
   if (isAggregateTypeForABI(RetTy)) {
     // Records with non-trivial destructors/copy-constructors should not be
     // returned by value.
     if (!getRecordArgABI(RetTy, getCXXABI())) {
       // Ignore empty structs/unions.

libc/config/gpu/entrypoints.txt

set(TARGET_LIBC_ENTRYPOINTS
     libc.src.stdlib.aligned_alloc
     libc.src.stdlib.realloc
     libc.src.stdlib.free
 
     # errno.h entrypoints
     libc.src.errno.errno
 
     # stdio.h entrypoints
+    libc.src.stdio.snprintf
+    libc.src.stdio.vsnprintf
     libc.src.stdio.puts
     libc.src.stdio.fopen
     libc.src.stdio.fclose
     libc.src.stdio.fread
     libc.src.stdio.fputs
     libc.src.stdio.stdin
     libc.src.stdio.stdout
     libc.src.stdio.stderr
 

libc/test/src/__support/CMakeLists.txt

SRCS
     integer_to_string_test.cpp
   DEPENDS
     libc.src.__support.integer_to_string
     libc.src.__support.CPP.string_view
     libc.src.__support.uint
     libc.src.__support.uint128
 )
 
-# The GPU does not support varargs currently.
-if(NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
-  add_libc_test(
-    arg_list_test
-    SUITE
-      libc-support-tests
-    SRCS
-      arg_list_test.cpp
-    DEPENDS
-      libc.src.__support.arg_list
-  )
-endif()
 
 add_libc_test(
   uint_test
   SUITE
     libc-support-tests
   SRCS
     uint_test.cpp
   DEPENDS

libc/test/src/stdio/CMakeLists.txt

add_fp_unittest(
   SRCS
     sprintf_test.cpp
   DEPENDS
     libc.src.stdio.sprintf
     libc.src.__support.FPUtil.fp_bits
     libc.src.__support.FPUtil.platform_defs
 )
 
-add_libc_unittest(
+add_libc_test(
   snprintf_test
   SUITE
     libc_stdio_unittests
   SRCS
     snprintf_test.cpp
   DEPENDS
     libc.src.stdio.snprintf
 )
add_fp_unittest(
   SUITE
     libc_stdio_unittests
   SRCS
     vsprintf_test.cpp
   DEPENDS
     libc.src.stdio.vsprintf
 )
 
-add_libc_unittest(
+add_libc_test(
   vsnprintf_test
   SUITE
     libc_stdio_unittests
   SRCS
     vsnprintf_test.cpp
   DEPENDS
     libc.src.stdio.vsnprintf
 )

llvm/include/llvm/CodeGen/DesugarVariadics.h

+#ifndef LLVM_CODEGEN_DESUGARVARIADICS_H
+#define LLVM_CODEGEN_DESUGARVARIADICS_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+class Module;
+
+class DesugarVariadicsPass : public PassInfoMixin<DesugarVariadicsPass> {
+public:
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
+};
+
+} // end namespace llvm
+
+#endif // LLVM_CODEGEN_DESUGAR_VARIADICS_H

llvm/include/llvm/CodeGen/MachinePassRegistry.def

 #endif
 MODULE_ANALYSIS("pass-instrumentation", PassInstrumentationAnalysis, (PIC))
 #undef MODULE_ANALYSIS
 
 #ifndef MODULE_PASS
 #define MODULE_PASS(NAME, PASS_NAME, CONSTRUCTOR)
 #endif
 MODULE_PASS("pre-isel-intrinsic-lowering", PreISelIntrinsicLoweringPass, ())
+MODULE_PASS("desugar-variadics", DesugarVariadicsPass, ())
 #undef MODULE_PASS
 
 #ifndef FUNCTION_ANALYSIS
 #define FUNCTION_ANALYSIS(NAME, PASS_NAME, CONSTRUCTOR)
 #endif
 FUNCTION_ANALYSIS("pass-instrumentation", PassInstrumentationAnalysis, (PIC))
 FUNCTION_ANALYSIS("targetir", TargetIRAnalysis, (std::move(TM.getTargetIRAnalysis())))
 #undef FUNCTION_ANALYSIS

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
 
   /// This pass converts conditional moves to conditional jumps when profitable.
   FunctionPass *createSelectOptimizePass();
 
   FunctionPass *createCallBrPass();
 
   /// Lowers KCFI operand bundles for indirect calls.
   FunctionPass *createKCFIPass();
+
+  // Lower variadic functions and calls. Unconditionally if passed true.
+  ModulePass *createDesugarVariadicsPass(bool ApplicableToAllFunctions);
+
 } // End llvm namespace
 
 #endif

llvm/include/llvm/InitializePasses.h

 void initializeEarlyTailDuplicatePass(PassRegistry&);
 void initializeEdgeBundlesPass(PassRegistry&);
 void initializeEHContGuardCatchretPass(PassRegistry &);
 void initializeExpandLargeFpConvertLegacyPassPass(PassRegistry&);
 void initializeExpandLargeDivRemLegacyPassPass(PassRegistry&);
 void initializeExpandMemCmpPassPass(PassRegistry&);
 void initializeExpandPostRAPass(PassRegistry&);
 void initializeExpandReductionsPass(PassRegistry&);
+void initializeDesugarVariadicsPass(PassRegistry &);
 void initializeExpandVectorPredicationPass(PassRegistry &);
 void initializeMakeGuardsExplicitLegacyPassPass(PassRegistry&);
 void initializeExternalAAWrapperPassPass(PassRegistry&);
 void initializeFEntryInserterPass(PassRegistry&);
 void initializeFinalizeISelPass(PassRegistry&);
 void initializeFinalizeMachineBundlesPass(PassRegistry&);
 void initializeFixIrreduciblePass(PassRegistry &);
 void initializeFixupStatepointCallerSavedPass(PassRegistry&);

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   CodeGen.cpp
   CodeGenCommonISel.cpp
   CodeGenPassBuilder.cpp
   CodeGenPrepare.cpp
   CommandFlags.cpp
   ComplexDeinterleavingPass.cpp
   CriticalAntiDepBreaker.cpp
   DeadMachineInstructionElim.cpp
+  DesugarVariadics.cpp
   DetectDeadLanes.cpp
   DFAPacketizer.cpp
   DwarfEHPrepare.cpp
   EarlyIfConversion.cpp
   EdgeBundles.cpp
   EHContGuardCatchret.cpp
   ExecutionDomainFix.cpp
   ExpandLargeDivRem.cpp

llvm/lib/CodeGen/CodeGen.cpp

void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeCallBrPreparePass(Registry);
   initializeCFGuardLongjmpPass(Registry);
   initializeCFIFixupPass(Registry);
   initializeCFIInstrInserterPass(Registry);
   initializeCheckDebugMachineModulePass(Registry);
   initializeCodeGenPreparePass(Registry);
   initializeDeadMachineInstructionElimPass(Registry);
   initializeDebugifyMachineModulePass(Registry);
+  initializeDesugarVariadicsPass(Registry);
   initializeDetectDeadLanesPass(Registry);
   initializeDwarfEHPrepareLegacyPassPass(Registry);
   initializeEarlyIfConverterPass(Registry);
   initializeEarlyIfPredicatorPass(Registry);
   initializeEarlyMachineLICMPass(Registry);
   initializeEarlyTailDuplicatePass(Registry);
   initializeExpandLargeDivRemLegacyPassPass(Registry);
   initializeExpandLargeFpConvertLegacyPassPass(Registry);

llvm/lib/CodeGen/DesugarVariadics.cpp

+//===-- DesugarVariadicsPass.cpp --------------------------------*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Can desugar variadic functions, their calls, va_arg and the intrinsics.
+//
+// The lowering replaces the variadic argument (...) with a Int8Ty*, moves
+// function arguments into a alloca struct and passes that address instead.
+// The struct alignment is the natural stack alignment, or greater if arguments
+// have greater ABI alignment.
+//
+// Order of operations is chosen to keep the IR semantically well formed.
+// 1/ Expand intrinsics that don't involve function parameters
+// 2/ Declare new functions with the ... arg replaced with void*
+// 3/ Replace call instructions to the variadics with calls to the declarations
+// 4/ Splice the body of original functions into the new ones
+// 5/ Delete the remaining parts of the original functions
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/DesugarVariadics.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+
+#include <cstdio>
+
+#define DEBUG_TYPE "desugar-variadics"
+
+using namespace llvm;
+
+static cl::opt<bool>
+    ApplyToAllOverride(DEBUG_TYPE "-all", cl::init(false),
+                       cl::desc("Lower all variadic functions and calls"),
+                       cl::Hidden);
+
+namespace {
+
+class DesugarVariadics : public ModulePass {
+public:
+  static char ID;
+  bool ApplicableToAllDefault;
+  DesugarVariadics(bool A = false)
+      : ModulePass(ID), ApplicableToAllDefault(A) {}
+
+  static void ExpandVAArg(VAArgInst *Inst, const DataLayout &DL) {
+    auto &Ctx = Inst->getContext();
+
+    Type *IntPtrTy = DL.getIntPtrType(Inst->getContext());
+    IRBuilder<> Builder(Inst);
+
+    Value *vaListPointer = Inst->getPointerOperand();
+    Value *vaListValue = Builder.CreateLoad(Type::getInt8PtrTy(Ctx),
+                                            vaListPointer, "arglist_current");
+
+    Align DataAlign = DL.getABITypeAlign(Inst->getType());
+    uint64_t DataAlignMinusOne = DataAlign.value() - 1;
+
+    Value *Incr = Builder.CreateConstInBoundsGEP1_32(
+        Type::getInt8Ty(Ctx), vaListValue, DataAlignMinusOne);
+
+    Value *Mask = ConstantInt::get(IntPtrTy, ~(DataAlignMinusOne));
+    Value *vaListAligned = Builder.CreateIntToPtr(
+        Builder.CreateAnd(Builder.CreatePtrToInt(Incr, IntPtrTy), Mask),
+        Incr->getType());
+
+    auto *Result = Builder.CreateAlignedLoad(Inst->getType(), vaListAligned,
+                                             DataAlign, "va_arg");
+    Result->takeName(Inst);
+
+    Value *Indexes[] = {ConstantInt::get(IntPtrTy, 1)};
+    auto *Next = Builder.CreateInBoundsGEP(Inst->getType(), vaListAligned,
+                                           Indexes, "arglist_next");
+    Builder.CreateStore(Next, vaListPointer);
+
+    Inst->replaceAllUsesWith(Result);
+    Inst->eraseFromParent();
+  }
+
+  static void ExpandVAStart(VAStartInst *Inst, Argument *StructPtr) {
+    IRBuilder<> Builder(Inst);
+    Builder.CreateStore(StructPtr, Inst->getArgList());
+    Inst->eraseFromParent();
+  }
+
+  static void ExpandVACopy(VACopyInst *Inst) {
+    IRBuilder<> Builder(Inst);
+    Value *dst = Inst->getDest();
+    Value *src = Inst->getSrc();
+    Value *ld = Builder.CreateLoad(src->getType(), src, "vacopy");
+    Builder.CreateStore(ld, dst);
+    Inst->eraseFromParent();
+  }
+
+  static void ExpandVAEnd(VAEndInst *Inst) { Inst->eraseFromParent(); }
+
+  static bool runOnFunction(Function &F) {
+    Module &M = *F.getParent();
+    const DataLayout &DL = M.getDataLayout();
+    bool Changed = false;
+    for (BasicBlock &BB : F) {
+      for (Instruction &I : llvm::make_early_inc_range(BB)) {
+        if (VAArgInst *II = dyn_cast<VAArgInst>(&I)) {
+          Changed = true;
+          ExpandVAArg(II, DL);
+          continue;
+        }
+        if (VAEndInst *II = dyn_cast<VAEndInst>(&I)) {
+          Changed = true;
+          ExpandVAEnd(II);
+          continue;
+        }
+        if (VACopyInst *II = dyn_cast<VACopyInst>(&I)) {
+          Changed = true;
+          ExpandVACopy(II);
+          continue;
+        }
+      }
+    }
+
+    if (F.isVarArg()) {
+      ExpandVariadicFunction(M, F);
+      Changed = true;
+    }
+
+    return Changed;
+  }
+
+  bool runOnModule(Module &M) override {
+    bool Apply = ApplicableToAllDefault | ApplyToAllOverride;
+    bool Changed = false;
+    for (Function &F : llvm::make_early_inc_range(M))
+      if (Apply || canTransformFunctionInIsolation(F))
+        Changed |= runOnFunction(F);
+    return Changed;
+  }
+
+  bool canTransformFunctionInIsolation(Function &F) {
+    if (!F.isVarArg() || F.isDeclaration() || !F.hasLocalLinkage() ||
+        F.hasAddressTaken() || F.hasFnAttribute(Attribute::Naked)) {
+      return false;
+    }
+
+    // TODO: function is plumbing for extending this lowering pass for
+    // optimisation on targets which use different variadic calling
+    // conventions. Escape analysis on va_list values.
+    return false;
+  }
+
+  static void ExpandCall(Module &M, CallBase *CB, Function *NF) {
+    const DataLayout &DL = M.getDataLayout();
+
+    FunctionType *FuncType = CB->getFunctionType();
+    auto &Ctx = CB->getContext();
+    unsigned NumArgs = FuncType->getNumParams();
+
+    SmallVector<Value *> Args;
+    Args.assign(CB->arg_begin(), CB->arg_begin() + NumArgs);
+
+    SmallVector<std::pair<Value *, uint64_t>> Varargs;
+    SmallVector<Type *> LocalVarTypes;
+
+    Align MaxFieldAlign(1);
+    uint64_t CurrentOffset = 0;
+    for (unsigned I = FuncType->getNumParams(), E = CB->arg_size(); I < E;
+         I++) {
+      Value *ArgVal = CB->getArgOperand(I);
+
+      bool isByVal = CB->paramHasAttr(I, Attribute::ByVal);
+      if (isByVal)
+        report_fatal_error("Unimplemented byval");
+
+      Type *ArgType = ArgVal->getType();
+      Align DataAlign = DL.getABITypeAlign(ArgType);
+      MaxFieldAlign = std::max(MaxFieldAlign, DataAlign);
+      uint64_t DataAlignV = DataAlign.value();
+
+      if (uint64_t Rem = CurrentOffset % DataAlignV) {
+        uint64_t Padding = DataAlignV - Rem;
+        Type *ATy = ArrayType::get(Type::getInt8Ty(Ctx), Padding);
+        LocalVarTypes.push_back(ATy);
+        CurrentOffset += Padding;
+      }
+
+      Varargs.push_back({ArgVal, LocalVarTypes.size()});
+      LocalVarTypes.push_back(ArgType);
+
+      CurrentOffset += DL.getTypeAllocSize(ArgType).getFixedValue();
+    }
+
+    if (Varargs.empty()) {
+      // todo, pass nullptr instead?
+      LocalVarTypes.push_back(Type::getInt32Ty(Ctx));
+    }
+
+    StructType *VarargsTy = StructType::create(
+        Ctx, LocalVarTypes, (Twine(NF->getName()) + ".vararg").str());
+
+    Function *CBF = CB->getParent()->getParent();
+    BasicBlock &BB = CBF->getEntryBlock();
+    IRBuilder<> Builder(&*BB.getFirstInsertionPt());
+
+    auto alloced = Builder.Insert(
+        new AllocaInst(VarargsTy, DL.getAllocaAddrSpace(), nullptr,
+                       std::max(MaxFieldAlign, assumedStructAlignment(DL))),
+        "vararg_buffer");
+
+    // TODO: Lifetime annotate it
+    Builder.SetInsertPoint(CB);
+    for (size_t i = 0; i < Varargs.size(); i++) {
+      // todo: byval here?
+      auto r = Builder.CreateStructGEP(VarargsTy, alloced, Varargs[i].second);
+      Builder.CreateStore(Varargs[i].first,
+                          r); // alignment info could be better
+    }
+
+    Args.push_back(Builder.CreatePointerBitCastOrAddrSpaceCast(
+        alloced, Type::getInt8PtrTy(Ctx)));
+
+    // Attributes excluding any on the vararg arguments
+    AttributeList PAL = CB->getAttributes();
+    if (!PAL.isEmpty()) {
+      SmallVector<AttributeSet, 8> ArgAttrs;
+      for (unsigned ArgNo = 0; ArgNo < NumArgs; ArgNo++)
+        ArgAttrs.push_back(PAL.getParamAttrs(ArgNo));
+      PAL = AttributeList::get(Ctx, PAL.getFnAttrs(), PAL.getRetAttrs(),
+                               ArgAttrs);
+    }
+
+    SmallVector<OperandBundleDef, 1> OpBundles;
+    CB->getOperandBundlesAsDefs(OpBundles);
+
+    CallBase *NewCB = nullptr;
+    if (InvokeInst *II = dyn_cast<InvokeInst>(CB)) {
+      NewCB = InvokeInst::Create(NF, II->getNormalDest(), II->getUnwindDest(),
+                                 Args, OpBundles, "", CB);
+    } else {
+      NewCB = CallInst::Create(NF, Args, OpBundles, "", CB);
+      cast<CallInst>(NewCB)->setTailCallKind(
+          cast<CallInst>(CB)->getTailCallKind());
+    }
+
+    NewCB->setAttributes(PAL);
+    NewCB->takeName(CB);
+    NewCB->setCallingConv(CB->getCallingConv());
+    NewCB->copyMetadata(*CB, {LLVMContext::MD_prof, LLVMContext::MD_dbg});
+
+    if (!CB->use_empty())
+      CB->replaceAllUsesWith(NewCB);
+    CB->eraseFromParent();
+  }
+
+  static Align assumedStructAlignment(const DataLayout &DL) {
+    // TODO: Change the DataLayout API so there's an easier way to test
+    // whether the stack alignment is known. Nvptx doesn't always have S
+    // in the data layout string, this sidesteps an assertion there.
+    Align ExcessiveAlignment = Align(UINT64_C(1) << 63u);
+    bool knownNaturalStackAlignment =
+        DL.exceedsNaturalStackAlignment(ExcessiveAlignment);
+    if (knownNaturalStackAlignment) {
+      return DL.getStackAlignment();
+    } else {
+      return {};
+    }
+  }
+
+  static void ExpandVariadicFunction(Module &M, Function &F) {
+    auto &Ctx = M.getContext();
+    const DataLayout &DL = M.getDataLayout();
+
+    IRBuilder<> Builder(Ctx);
+
+    FunctionType *FTy = F.getFunctionType();
+
+    SmallVector<Type *> ArgTypes(FTy->param_begin(), FTy->param_end());
+    ArgTypes.push_back(Type::getInt8PtrTy(Ctx));
+
+    FunctionType *NFTy =
+        FunctionType::get(FTy->getReturnType(), ArgTypes, /*IsVarArgs*/ false);
+
+    Function *NF = Function::Create(NFTy, F.getLinkage(), F.getAddressSpace());
+
+    // Note - same attribute handling as DeadArgumentElimination
+    NF->copyAttributesFrom(&F);
+    NF->setComdat(F.getComdat());
+    F.getParent()->getFunctionList().insert(F.getIterator(), NF);
+    NF->takeName(&F);
+
+    AttrBuilder ParamAttrs(Ctx);
+    ParamAttrs.addAttribute(Attribute::NoAlias);
+    ParamAttrs.addAlignmentAttr(assumedStructAlignment(DL));
+
+    AttributeList Attrs = NF->getAttributes();
+    Attrs = Attrs.addParamAttributes(Ctx, NFTy->getNumParams() - 1, ParamAttrs);
+    NF->setAttributes(Attrs);
+
+    // Declared the new function, can now create calls to it
+    for (User *U : llvm::make_early_inc_range(F.users()))
+      if (CallBase *CB = dyn_cast<CallBase>(U))
+        ExpandCall(M, CB, NF);
+
+    // If it's a definition, move the implementation across
+    if (!F.isDeclaration()) {
+      NF->splice(NF->begin(), &F);
+
+      auto NewArg = NF->arg_begin();
+      for (Argument &Arg : F.args()) {
+        Arg.replaceAllUsesWith(NewArg);
+        NewArg->takeName(&Arg);
+        ++NewArg;
+      }
+      NewArg->setName("varargs");
+
+      for (BasicBlock &BB : *NF)
+        for (Instruction &I : llvm::make_early_inc_range(BB))
+          if (VAStartInst *II = dyn_cast<VAStartInst>(&I))
+            ExpandVAStart(II, NewArg);
+    }
+
+    SmallVector<std::pair<unsigned, MDNode *>, 1> MDs;
+    F.getAllMetadata(MDs);
+    for (auto [KindID, Node] : MDs)
+      NF->addMetadata(KindID, *Node);
+
+    // DAE bitcasts it, todo: check block addresses
+    // This fails to update call instructions, unfortunately
+    // It may therefore also fail to update globals
+    F.replaceAllUsesWith(NF);
+
+    F.eraseFromParent();
+  }
+};
+} // namespace
+
+char DesugarVariadics::ID = 0;
+
+INITIALIZE_PASS(DesugarVariadics, DEBUG_TYPE, "Desugar Variadics", false,
+                false)
+
+ModulePass *llvm::createDesugarVariadicsPass(bool ApplicableToAllFunctions) {
+  return new DesugarVariadics(ApplicableToAllFunctions);
+}
+
+PreservedAnalyses DesugarVariadicsPass::run(Module &M,
+                                              ModuleAnalysisManager &) {
+  return DesugarVariadics(false).runOnModule(M) ? PreservedAnalyses::none()
+                                                  : PreservedAnalyses::all();
+}

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

 #include "R600.h"
 #include "R600MachineFunctionInfo.h"
 #include "R600TargetMachine.h"
 #include "SIMachineFunctionInfo.h"
 #include "SIMachineScheduler.h"
 #include "TargetInfo/AMDGPUTargetInfo.h"
 #include "Utils/AMDGPUBaseInfo.h"
 #include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/CodeGen/DesugarVariadics.h"
 #include "llvm/CodeGen/GlobalISel/CSEInfo.h"
 #include "llvm/CodeGen/GlobalISel/IRTranslator.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelect.h"
 #include "llvm/CodeGen/GlobalISel/Legalizer.h"
 #include "llvm/CodeGen/GlobalISel/Localizer.h"
 #include "llvm/CodeGen/GlobalISel/RegBankSelect.h"
 #include "llvm/CodeGen/MIRParser/MIParser.h"
 #include "llvm/CodeGen/Passes.h"
PB.registerPipelineParsingCallback(
         if (PassName == "amdgpu-lower-module-lds") {
           PM.addPass(AMDGPULowerModuleLDSPass());
           return true;
         }
         if (PassName == "amdgpu-lower-ctor-dtor") {
           PM.addPass(AMDGPUCtorDtorLoweringPass());
           return true;
         }
+        if (PassName == "desugar-variadics") {
+          PM.addPass(DesugarVariadicsPass());
+          return true;
+        }
         return false;
       });
   PB.registerPipelineParsingCallback(
       [this](StringRef PassName, FunctionPassManager &PM,
              ArrayRef<PassBuilder::PipelineElement>) {
         if (PassName == "amdgpu-simplifylib") {
           PM.addPass(AMDGPUSimplifyLibCallsPass());
           return true;
if (EnableLowerModuleLDS) {
     addPass(createAMDGPULowerModuleLDSPass());
   }
 
   // AMDGPUAttributor infers lack of llvm.amdgcn.lds.kernel.id calls, so run
   // after their introduction
   if (TM.getOptLevel() > CodeGenOpt::None)
     addPass(createAMDGPUAttributorPass());
 
+  // TODO: Choose where to put this in pipeline
+  addPass(createDesugarVariadicsPass(true));
+
   if (TM.getOptLevel() > CodeGenOpt::None)
     addPass(createInferAddressSpacesPass());
 
   addPass(createAtomicExpandPass());
 
   if (TM.getOptLevel() > CodeGenOpt::None) {
     addPass(createAMDGPUPromoteAlloca());
 

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

 ; GCN-O0-NEXT:    AMDGPU Inline All Functions
 ; GCN-O0-NEXT:    Inliner for always_inline functions
 ; GCN-O0-NEXT:      FunctionPass Manager
 ; GCN-O0-NEXT:        Dominator Tree Construction
 ; GCN-O0-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O0-NEXT:        Function Alias Analysis Results
 ; GCN-O0-NEXT:    Lower OpenCL enqueued blocks
 ; GCN-O0-NEXT:    Lower uses of LDS variables from non-kernel functions
+; GCN-O0-NEXT:    Desugar Variadics
 ; GCN-O0-NEXT:    FunctionPass Manager
 ; GCN-O0-NEXT:      Expand Atomic instructions
 ; GCN-O0-NEXT:      Lower constant intrinsics
 ; GCN-O0-NEXT:      Remove unreachable blocks from the CFG
 ; GCN-O0-NEXT:      Expand vector predication intrinsics
 ; GCN-O0-NEXT:      Scalarize Masked Memory Intrinsics
 ; GCN-O0-NEXT:      Expand reduction intrinsics
 ; GCN-O0-NEXT:    CallGraph Construction

 ; GCN-O1-NEXT:        Dominator Tree Construction
 ; GCN-O1-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-NEXT:        Function Alias Analysis Results
 ; GCN-O1-NEXT:    Lower OpenCL enqueued blocks
 ; GCN-O1-NEXT:    Lower uses of LDS variables from non-kernel functions
 ; GCN-O1-NEXT:    AMDGPU Attributor
 ; GCN-O1-NEXT:      FunctionPass Manager
 ; GCN-O1-NEXT:        Cycle Info Analysis
+; GCN-O1-NEXT:    Desugar Variadics
 ; GCN-O1-NEXT:    FunctionPass Manager
 ; GCN-O1-NEXT:      Infer address spaces
 ; GCN-O1-NEXT:      Expand Atomic instructions
 ; GCN-O1-NEXT:      AMDGPU Promote Alloca
 ; GCN-O1-NEXT:      Dominator Tree Construction
 ; GCN-O1-NEXT:      Cycle Info Analysis
 ; GCN-O1-NEXT:      Uniformity Analysis
 ; GCN-O1-NEXT:      AMDGPU IR optimizations

 ; GCN-O1-OPTS-NEXT:        Dominator Tree Construction
 ; GCN-O1-OPTS-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O1-OPTS-NEXT:        Function Alias Analysis Results
 ; GCN-O1-OPTS-NEXT:    Lower OpenCL enqueued blocks
 ; GCN-O1-OPTS-NEXT:    Lower uses of LDS variables from non-kernel functions
 ; GCN-O1-OPTS-NEXT:    AMDGPU Attributor
 ; GCN-O1-OPTS-NEXT:      FunctionPass Manager
 ; GCN-O1-OPTS-NEXT:        Cycle Info Analysis
+; GCN-O1-OPTS-NEXT:    Desugar Variadics
 ; GCN-O1-OPTS-NEXT:    FunctionPass Manager
 ; GCN-O1-OPTS-NEXT:      Infer address spaces
 ; GCN-O1-OPTS-NEXT:      Expand Atomic instructions
 ; GCN-O1-OPTS-NEXT:      AMDGPU Promote Alloca
 ; GCN-O1-OPTS-NEXT:      Dominator Tree Construction
 ; GCN-O1-OPTS-NEXT:      Natural Loop Information
 ; GCN-O1-OPTS-NEXT:      Split GEPs to a variadic base and a constant offset for better CSE
 ; GCN-O1-OPTS-NEXT:      Scalar Evolution Analysis

 ; GCN-O2-NEXT:        Dominator Tree Construction
 ; GCN-O2-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O2-NEXT:        Function Alias Analysis Results
 ; GCN-O2-NEXT:    Lower OpenCL enqueued blocks
 ; GCN-O2-NEXT:    Lower uses of LDS variables from non-kernel functions
 ; GCN-O2-NEXT:    AMDGPU Attributor
 ; GCN-O2-NEXT:      FunctionPass Manager
 ; GCN-O2-NEXT:        Cycle Info Analysis
+; GCN-O2-NEXT:    Desugar Variadics
 ; GCN-O2-NEXT:    FunctionPass Manager
 ; GCN-O2-NEXT:      Infer address spaces
 ; GCN-O2-NEXT:      Expand Atomic instructions
 ; GCN-O2-NEXT:      AMDGPU Promote Alloca
 ; GCN-O2-NEXT:      Dominator Tree Construction
 ; GCN-O2-NEXT:      Natural Loop Information
 ; GCN-O2-NEXT:      Split GEPs to a variadic base and a constant offset for better CSE
 ; GCN-O2-NEXT:      Scalar Evolution Analysis

 ; GCN-O3-NEXT:        Dominator Tree Construction
 ; GCN-O3-NEXT:        Basic Alias Analysis (stateless AA impl)
 ; GCN-O3-NEXT:        Function Alias Analysis Results
 ; GCN-O3-NEXT:    Lower OpenCL enqueued blocks
 ; GCN-O3-NEXT:    Lower uses of LDS variables from non-kernel functions
 ; GCN-O3-NEXT:    AMDGPU Attributor
 ; GCN-O3-NEXT:      FunctionPass Manager
 ; GCN-O3-NEXT:        Cycle Info Analysis
+; GCN-O3-NEXT:    Desugar Variadics
 ; GCN-O3-NEXT:    FunctionPass Manager
 ; GCN-O3-NEXT:      Infer address spaces
 ; GCN-O3-NEXT:      Expand Atomic instructions
 ; GCN-O3-NEXT:      AMDGPU Promote Alloca
 ; GCN-O3-NEXT:      Dominator Tree Construction
 ; GCN-O3-NEXT:      Natural Loop Information
 ; GCN-O3-NEXT:      Split GEPs to a variadic base and a constant offset for better CSE
 ; GCN-O3-NEXT:      Scalar Evolution Analysis

llvm/test/CodeGen/AMDGPU/unsupported-calls.ll

define i32 @test_tail_call(ptr addrspace(1) %out, ptr addrspace(1) %in) {
   %a = load i32, ptr addrspace(1) %in
   %b = load i32, ptr addrspace(1) %b_ptr
   %c = tail call i32 @defined_function(i32 %b)
   ret i32 %c
 }
 
 declare void @external.varargs(i32, double, i64, ...)
 
-; GCN: error: <unknown>:0:0: in function test_call_varargs void (): unsupported call to variadic function external.varargs
 ; R600: in function test_call_varargs{{.*}}: unsupported call to function external.varargs
 define void @test_call_varargs() {
   call void (i32, double, i64, ...) @external.varargs(i32 42, double 1.0, i64 12, i8 3, i16 1, i32 4, float 1.0, double 2.0)
   ret void
 }
 
 declare i32 @extern_variadic(...)
 
 ; GCN: in function test_tail_call_bitcast_extern_variadic{{.*}}: unsupported required tail call to function extern_variadic
-; R600: in function test_tail_call_bitcast_extern_variadic{{.*}}: unsupported call to function extern_variadic
 define i32 @test_tail_call_bitcast_extern_variadic(<4 x float> %arg0, <4 x float> %arg1, i32 %arg2) {
   %add = fadd <4 x float> %arg0, %arg1
-  %call = tail call i32 @extern_variadic(<4 x float> %add)
+  %call = tail call i32 (...) @extern_variadic(i32 0, <4 x float> %add)
   ret i32 %call
 }
 
 ; R600: in function test_c_call{{.*}}: unsupported call to function defined_function
 define amdgpu_ps i32 @test_c_call_from_shader() {
   %call = call i32 @defined_function(i32 0)
   ret i32 %call
 }
 
 ; GCN-NOT: in function test_gfx_call{{.*}}unsupported
 ; R600: in function test_gfx_call{{.*}}: unsupported call to function defined_function
 define amdgpu_ps i32 @test_gfx_call_from_shader() {
   %call = call amdgpu_gfx i32 @defined_function(i32 0)
   ret i32 %call
 }
 

llvm/test/CodeGen/Generic/expand-variadic-intrinsics.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -desugar-variadics -desugar-variadics-all=true -S < %s | FileCheck %s
+
+target datalayout = "e-m:e-S8"
+
+; CHECK: %variadic_to_call_middle.vararg = type { i32, [4 x i8], double, i32 }
+
+define dso_local void @variadic_to_call_middle(double %d, ...) {
+; CHECK-LABEL: @variadic_to_call_middle(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[VA:%.*]] = alloca ptr, align 8
+; CHECK-NEXT:    call void @llvm.lifetime.start.p0(i64 8, ptr nonnull [[VA]])
+; CHECK-NEXT:    store ptr [[VARARGS:%.*]], ptr [[VA]], align 8
+; CHECK-NEXT:    [[ARGLIST_CURRENT:%.*]] = load ptr, ptr [[VA]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds i8, ptr [[ARGLIST_CURRENT]], i32 3
+; CHECK-NEXT:    [[TMP1:%.*]] = ptrtoint ptr [[TMP0]] to i64
+; CHECK-NEXT:    [[TMP2:%.*]] = and i64 [[TMP1]], -4
+; CHECK-NEXT:    [[TMP3:%.*]] = inttoptr i64 [[TMP2]] to ptr
+; CHECK-NEXT:    [[TMP4:%.*]] = load i32, ptr [[TMP3]], align 4
+; CHECK-NEXT:    [[ARGLIST_NEXT:%.*]] = getelementptr inbounds i32, ptr [[TMP3]], i64 1
+; CHECK-NEXT:    store ptr [[ARGLIST_NEXT]], ptr [[VA]], align 8
+; CHECK-NEXT:    [[ARGLIST_CURRENT1:%.*]] = load ptr, ptr [[VA]], align 8
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr inbounds i8, ptr [[ARGLIST_CURRENT1]], i32 7
+; CHECK-NEXT:    [[TMP6:%.*]] = ptrtoint ptr [[TMP5]] to i64
+; CHECK-NEXT:    [[TMP7:%.*]] = and i64 [[TMP6]], -8
+; CHECK-NEXT:    [[TMP8:%.*]] = inttoptr i64 [[TMP7]] to ptr
+; CHECK-NEXT:    [[TMP9:%.*]] = load double, ptr [[TMP8]], align 8
+; CHECK-NEXT:    [[ARGLIST_NEXT2:%.*]] = getelementptr inbounds double, ptr [[TMP8]], i64 1
+; CHECK-NEXT:    store ptr [[ARGLIST_NEXT2]], ptr [[VA]], align 8
+; CHECK-NEXT:    [[ARGLIST_CURRENT3:%.*]] = load ptr, ptr [[VA]], align 8
+; CHECK-NEXT:    [[TMP10:%.*]] = getelementptr inbounds i8, ptr [[ARGLIST_CURRENT3]], i32 3
+; CHECK-NEXT:    [[TMP11:%.*]] = ptrtoint ptr [[TMP10]] to i64
+; CHECK-NEXT:    [[TMP12:%.*]] = and i64 [[TMP11]], -4
+; CHECK-NEXT:    [[TMP13:%.*]] = inttoptr i64 [[TMP12]] to ptr
+; CHECK-NEXT:    [[TMP14:%.*]] = load i32, ptr [[TMP13]], align 4
+; CHECK-NEXT:    [[ARGLIST_NEXT4:%.*]] = getelementptr inbounds i32, ptr [[TMP13]], i64 1
+; CHECK-NEXT:    store ptr [[ARGLIST_NEXT4]], ptr [[VA]], align 8
+; CHECK-NEXT:    call void @_Z3erri(i32 [[TMP4]])
+; CHECK-NEXT:    call void @_Z3errd(double [[TMP9]])
+; CHECK-NEXT:    call void @_Z3erri(i32 [[TMP14]])
+; CHECK-NEXT:    call void @llvm.lifetime.end.p0(i64 8, ptr nonnull [[VA]])
+; CHECK-NEXT:    ret void
+;
+entry:
+  %va = alloca ptr, align 8
+  call void @llvm.lifetime.start.p0(i64 8, ptr nonnull %va)
+  call void @llvm.va_start(ptr nonnull %va)
+  %0 = va_arg ptr %va, i32
+  %1 = va_arg ptr %va, double
+  %2 = va_arg ptr %va, i32
+  call void @llvm.va_end(ptr %va)
+  call void @_Z3erri(i32 %0)
+  call void @_Z3errd(double %1)
+  call void @_Z3erri(i32 %2)
+  call void @llvm.lifetime.end.p0(i64 8, ptr nonnull %va)
+  ret void
+}
+
+define dso_local void @variadic_to_call_entry(i32 %x0, double %x1, i32 %x2) {
+; CHECK-LABEL: @variadic_to_call_entry(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[VARARG_BUFFER:%.*]] = alloca [[VARIADIC_TO_CALL_MIDDLE_VARARG:%.*]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = getelementptr inbounds [[VARIADIC_TO_CALL_MIDDLE_VARARG]], ptr [[VARARG_BUFFER]], i32 0, i32 0
+; CHECK-NEXT:    store i32 [[X0:%.*]], ptr [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr inbounds [[VARIADIC_TO_CALL_MIDDLE_VARARG]], ptr [[VARARG_BUFFER]], i32 0, i32 2
+; CHECK-NEXT:    store double [[X1:%.*]], ptr [[TMP1]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = getelementptr inbounds [[VARIADIC_TO_CALL_MIDDLE_VARARG]], ptr [[VARARG_BUFFER]], i32 0, i32 3
+; CHECK-NEXT:    store i32 [[X2:%.*]], ptr [[TMP2]], align 4
+; CHECK-NEXT:    tail call void @variadic_to_call_middle(double 1.000000e+00, ptr [[VARARG_BUFFER]])
+; CHECK-NEXT:    ret void
+;
+entry:
+  tail call void (double, ...) @variadic_to_call_middle(double 1.0, i32 %x0, double %x1, i32 %x2)
+  ret void
+}
+
+
+declare void @llvm.va_start(ptr)
+declare void @llvm.va_end(ptr)
+
+declare void @llvm.lifetime.start.p0(i64 immarg, ptr nocapture)
+declare void @llvm.lifetime.end.p0(i64 immarg, ptr nocapture)
+
+declare void @_Z3erri(i32)
+declare void @_Z3errd(double)

llvm/tools/opt/opt.cpp

std::vector<StringRef> PassNameExact = {
       "verify-safepoint-ir",
       "atomic-expand",
       "expandvp",
       "mve-tail-predication",
       "interleaved-access",
       "global-merge",
       "pre-isel-intrinsic-lowering",
       "expand-reductions",
+      "desugar-variadics",
       "indirectbr-expand",
       "generic-to-nvvm",
       "expandmemcmp",
       "loop-reduce",
       "lower-amx-type",
       "pre-amx-config",
       "lower-amx-intrinsics",
       "polyhedral-info",
int main(int argc, char **argv) {
   initializeInterleavedAccessPass(Registry);
   initializeUnreachableBlockElimLegacyPassPass(Registry);
   initializeExpandReductionsPass(Registry);
   initializeExpandVectorPredicationPass(Registry);
   initializeWasmEHPreparePass(Registry);
   initializeWriteBitcodePassPass(Registry);
   initializeReplaceWithVeclibLegacyPass(Registry);
   initializeJMCInstrumenterPass(Registry);
+  initializeDesugarVariadicsPass(Registry);
 
   SmallVector<PassPlugin, 1> PluginList;
   PassPlugins.setCallback([&](const std::string &PluginPath) {
     auto Plugin = PassPlugin::Load(PluginPath);
     if (!Plugin) {
       errs() << "Failed to load passes from '" << PluginPath
              << "'. Request ignored.\n";
       return;

llvm/utils/gn/secondary/llvm/lib/CodeGen/BUILD.gn

sources = [
     "EarlyIfConversion.cpp",
     "EdgeBundles.cpp",
     "ExecutionDomainFix.cpp",
     "ExpandLargeDivRem.cpp",
     "ExpandLargeFpConvert.cpp",
     "ExpandMemCmp.cpp",
     "ExpandPostRAPseudos.cpp",
     "ExpandReductions.cpp",
+    "DesugarVariadics.cpp",
     "ExpandVectorPredication.cpp",
     "FEntryInserter.cpp",
     "FaultMaps.cpp",
     "FinalizeISel.cpp",
     "FixupStatepointCallerSaved.cpp",
     "FuncletLayout.cpp",
     "GCMetadata.cpp",
     "GCMetadataPrinter.cpp",