[Clang] support for outputs along indirect edges of asm goto

[nickdesaulniers]

Summary

Initial support for asm goto w/ outputs (D69876) only supported outputs
along the "default" (aka "fallthrough") edge.

We can support outputs along all edges by repeating the same pattern of
stores along the indirect edges that we allready do for the default
edge. One complication is that these indirect edges may be critical
edges which would need to be split. Another issue is that mid-codgen of
LLVM IR, the control flow graph might not reflect the control flow of
the final function.

To avoid this "chicken and the egg" problem assume that any given
indirect edge may become a critical edge, and pro-actively split it.
This is unnecessary if the edge does not become critical, but LLVM will
optimize such cases via tail duplication.

Fixes: https://github.com/llvm/llvm-project/issues/53562

[nickdesaulniers]

• add note to commit message (let's see if phab captures what I wrote, used arc diff --verbatim)

[nickdesaulniers]

• git clang-format HEAD~

[nickdesaulniers]

Note to reviewers: I expect this to be one of a 3 part progression.

1. Fix DomTree D135997
2. Fix front end D136497
3. Fix back end (TODO)

Hopefully that's it; famous last words...

[nickdesaulniers]

• update LanguageExtensions doc

[void]

It might be easier to see the main changes here if you submit the (very nice) refactoring of EmitAsmStores first.

clang/lib/CodeGen/CGStmt.cpp

2358

s/ResultTruncRegTypes[i]/TruncTy/

2362

This looks like a direct copy from below (which is fine). I'm a bit iffy on whether this covers *all* of the value types that could come through here...

2848

Should these asserts (or some of them) be moved into EmitAsmStores()?

2860

nit: Could you add a comment explaining that we're calling EmitAsmStores for asm goto's indirect branches? I was slightly confused about why we were calling EmitAsmStores more than once. :)

[nickdesaulniers]

In D136497#3889938, @void wrote:

It might be easier to see the main changes here if you submit the (very nice) refactoring of EmitAsmStores first.

D137113 TODO(Nick): rebase on D137113

clang/lib/CodeGen/CGStmt.cpp

2358

Done in D137113

2362

Note: this code was simply moved wholesale. If value types are missing, that is a pre-existing bug and should be a prerequisite to fix for this patch.

I've broken out D137113 as a child patch to make that clearer.

2362

s/should/shouldn't/

2848

Ah, yeah, let me add those to D137113.

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2848

Now done in D137113.

[nickdesaulniers]

• rebase on D137113, add comment

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2855

Oh, I can remove this now, I moved it into EmitAsmStores.

2856–2861

I'm not sure that I need to handle this case (of a callbr with the same indirect destination as the default destination). That might result from optimizations, but I don't think clang can or will generate such IR. Maybe I should turn this into an: assert(CBR->getIndirectDest(i) != CBR->getDefaultDest(i) && "We already emitted asm stores in the default dest");?

[void]

clang/lib/CodeGen/CGStmt.cpp

2856–2861

It could only happen with something like this:

asm goto (... :::: indirect);

indirect:;

In that simplified case, it looks like clang emits this:

define dso_local void @foo() #0 {
  callbr void asm sideeffect "", "i,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@foo, %2)) #1
          to label %1 [label %2], !srcloc !6

1:                                                ; preds = %0
  br label %2

2:                                                ; preds = %1, %0
  ret void
}

So it's probably safe to make this an assert.

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2856–2861

Yeah, it's impossible. While a CallBrInst can have a same indirect destination as the default destination as a result of optimizations, above we always create an "asm.fallthrough" block that is distinct from any of the named labels from the asm goto.

[nickdesaulniers]

• remove unnecssary asserts

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2856–2861

Oh, it's not just the default and indirects sharing, but what if the same indirect is listed twice? I should add a test for that...i.e.

asm goto (... :::: indirect,indirect);

[nickdesaulniers]

• reroll for a new sha so that phab unmarks this as "changes planned"

clang/lib/CodeGen/CGStmt.cpp

2856–2861

Oh, right, we don't allow that.

error: duplicate use of asm operand name "indirect"
note: asm operand name "indirect" first referenced here

I'm going to leave these asserts out and mark this thread done, but please reopen this thread with a new comment if you feel strongly that I should add it back (assert(CBR->getIndirectDest(i) != CBR->getDefaultDest(i) && "We already emitted asm stores in the default dest");).

[nickdesaulniers]

• rebase, use new callbrpad inst

[nickdesaulniers]

Probably should copy the release note change from https://reviews.llvm.org/D138078.

[nickdesaulniers]

• rebase, drop callbrpad inst

[nickdesaulniers]

• add clang release notes from D138078 and new one for new feature.

[nickdesaulniers]

Ready for review:

1. https://reviews.llvm.org/D135997
2. https://reviews.llvm.org/D140166
3. https://reviews.llvm.org/D139861
4. https://reviews.llvm.org/D139872
5. https://reviews.llvm.org/D139883
6. https://reviews.llvm.org/D139970
7. https://reviews.llvm.org/D140160
8. https://reviews.llvm.org/D137113
9. https://reviews.llvm.org/D136497

[nickdesaulniers]

• rebase, format

[nickdesaulniers]

• rebase

[void]

clang/docs/ReleaseNotes.rst

199 ↗

(On Diff #490246)

Is it worth it to mention that this change makes Clang's behavior with regards to your example here consistent with GCC's behavior?

clang/lib/CodeGen/CGStmt.cpp

2803

It seems as if you really want a map indexed by the BasicBlock*. So something like this:

DenseMap<BasicBlock *, SmallVector<Value *, 4>> // 4 should be enough for anyone.

That way you can iterate over CBR->getIndirectDests() instead of having to use an iterator.

2853

Could probably use !CBRRegResults.empty() instead of .size(). The first should be O(1) (granted .size() probably should be too, but .empty() is more explicit anyway).

[nickdesaulniers]

• use densemap, update release note, use !empty rather than size as per @void

[efriedma]

clang/docs/LanguageExtensions.rst

1584 ↗

(On Diff #491857)

Maybe put a note about earlier releases, so people don't get confused if they use newer documentation than their version of clang.

Is there any chance we want a dedicated __has_extension flag?

[void]

One small comment about the documentation but L:GTM.

clang/docs/LanguageExtensions.rst

1584 ↗

(On Diff #491857)

Is there any chance we want a dedicated __has_extension flag?

Might not be a bad idea just in case they have more than one compiler version they're using. Then again, if they add code to use this feature, they're fairly committed to the assumption that it'll "just work" even with a __has_extension flag...(Remember that we currently don't actually warn if they're using the outputs on the indirect branch. We just specify that such behavior is undefined.)

I think a strongly worded note here that it's available with Clang 16+ (or whatever) is going to be slightly better. It'll force them to use macros to support multiple compiler versions, making their code wicked ugly, and them possibly coming up with better ways to ensure that the correct compiler is used for this feature.

[nickdesaulniers]

clang/docs/LanguageExtensions.rst

1584 ↗

(On Diff #491857)

I've added that as a separate commit https://reviews.llvm.org/D143205 to not hold this up. PTAL. Open to better names for the identifier.

[nickdesaulniers]

• rebase

[nickdesaulniers]

• final rebase

[uabelho]

I noticed that gvn-hoist doesn't handle this very nicely, I just wrote https://github.com/llvm/llvm-project/issues/61023

clang/lib/CodeGen/CGStmt.cpp

 #include "clang/AST/Expr.h"
 #include "clang/AST/Stmt.h"
 #include "clang/AST/StmtVisitor.h"
 #include "clang/Basic/Builtins.h"
 #include "clang/Basic/DiagnosticSema.h"
 #include "clang/Basic/PrettyStackTrace.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
+#include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/IR/Assumptions.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/Support/SaveAndRestore.h"
static void UpdateAsmCallInst(llvm::CallBase &Result, bool HasSideEffect,
   } else {
     for (unsigned i = 0, e = ResultRegTypes.size(); i != e; ++i) {
       llvm::Value *Tmp = CGF.Builder.CreateExtractValue(&Result, i, "asmresult");
       RegResults.push_back(Tmp);
     }
   }
 }
 
+static void EmitAsmStores(
+    CodeGenFunction &CGF,
+    const AsmStmt &S,
+    const llvm::ArrayRef<llvm::Value *> RegResults,
+    const llvm::ArrayRef<llvm::Type *> ResultRegTypes,
+    const llvm::ArrayRef<llvm::Type *> ResultTruncRegTypes,
+    const llvm::ArrayRef<LValue> ResultRegDests,
+    const llvm::ArrayRef<QualType> ResultRegQualTys,
+    const llvm::BitVector &ResultTypeRequiresCast,
+    const llvm::BitVector &ResultRegIsFlagReg) {
+  CGBuilderTy &Builder = CGF.Builder;
+  CodeGenModule &CGM = CGF.CGM;
+  llvm::LLVMContext &CTX = CGF.getLLVMContext();
+
+  for (unsigned i = 0, e = RegResults.size(); i != e; ++i) {
+    llvm::Value *Tmp = RegResults[i];
+    llvm::Type *TruncTy = ResultTruncRegTypes[i];
+
+    if ((i < ResultRegIsFlagReg.size()) && ResultRegIsFlagReg[i]) {
+      // Target must guarantee the Value `Tmp` here is lowered to a boolean
+      // value.
+      llvm::Constant *Two = llvm::ConstantInt::get(Tmp->getType(), 2);
+      llvm::Value *IsBooleanValue =
+          Builder.CreateCmp(llvm::CmpInst::ICMP_ULT, Tmp, Two);
+      llvm::Function *FnAssume = CGM.getIntrinsic(llvm::Intrinsic::assume);
+      Builder.CreateCall(FnAssume, IsBooleanValue);
+    }
+
+    // If the result type of the LLVM IR asm doesn't match the result type of
+    // the expression, do the conversion.
+    if (ResultRegTypes[i] != ResultTruncRegTypes[i]) {
+
+      // Truncate the integer result to the right size, note that TruncTy can be
+      // a pointer.
+      if (TruncTy->isFloatingPointTy())
+        Tmp = Builder.CreateFPTrunc(Tmp, TruncTy);
+      else if (TruncTy->isPointerTy() && Tmp->getType()->isIntegerTy()) {
+        uint64_t ResSize = CGM.getDataLayout().getTypeSizeInBits(TruncTy);
+        Tmp = Builder.CreateTrunc(Tmp,
+                   llvm::IntegerType::get(CTX, (unsigned)ResSize));
+        Tmp = Builder.CreateIntToPtr(Tmp, TruncTy);
+      } else if (Tmp->getType()->isPointerTy() && TruncTy->isIntegerTy()) {
+        uint64_t TmpSize =CGM.getDataLayout().getTypeSizeInBits(Tmp->getType());
+        Tmp = Builder.CreatePtrToInt(Tmp,
+                   llvm::IntegerType::get(CTX, (unsigned)TmpSize));
+        Tmp = Builder.CreateTrunc(Tmp, TruncTy);
+      } else if (TruncTy->isIntegerTy()) {
+        Tmp = Builder.CreateZExtOrTrunc(Tmp, TruncTy);
+      } else if (TruncTy->isVectorTy()) {
+        Tmp = Builder.CreateBitCast(Tmp, TruncTy);
+      }
+    }
+
+    LValue Dest = ResultRegDests[i];
+    // ResultTypeRequiresCast elements correspond to the first
+    // ResultTypeRequiresCast.size() elements of RegResults.
+    if ((i < ResultTypeRequiresCast.size()) && ResultTypeRequiresCast[i]) {
+      unsigned Size = CGF.getContext().getTypeSize(ResultRegQualTys[i]);
+      Address A = Builder.CreateElementBitCast(Dest.getAddress(CGF),
+                                               ResultRegTypes[i]);
+      if (CGF.getTargetHooks().isScalarizableAsmOperand(CGF, TruncTy)) {
+        Builder.CreateStore(Tmp, A);
+        continue;
+      }
+
+      QualType Ty = CGF.getContext().getIntTypeForBitwidth(Size, /*Signed*/ false);
+      if (Ty.isNull()) {
+        const Expr *OutExpr = S.getOutputExpr(i);
+        CGM.getDiags().Report(OutExpr->getExprLoc(),
+                              diag::err_store_value_to_reg);
+        return;
+      }
+      Dest = CGF.MakeAddrLValue(A, Ty);
+    }
+    CGF.EmitStoreThroughLValue(RValue::get(Tmp), Dest);
+  }
+}
+
 void CodeGenFunction::EmitAsmStmt(const AsmStmt &S) {
   // Pop all cleanup blocks at the end of the asm statement.
   CodeGenFunction::RunCleanupsScope Cleanups(*this);
 
   // Assemble the final asm string.
   std::string AsmString = S.generateAsmString(getContext());
 
   // Get all the output and input constraints together.
for (unsigned i = 0, e = InOutArgs.size(); i != e; i++) {
     Args.push_back(InOutArgs[i]);
   }
   Constraints += InOutConstraints;
 
   // Labels
   SmallVector<llvm::BasicBlock *, 16> Transfer;
   llvm::BasicBlock *Fallthrough = nullptr;
   bool IsGCCAsmGoto = false;
-  if (const auto *GS =  dyn_cast<GCCAsmStmt>(&S)) {
+  if (const auto *GS = dyn_cast<GCCAsmStmt>(&S)) {
     IsGCCAsmGoto = GS->isAsmGoto();
     if (IsGCCAsmGoto) {
       for (const auto *E : GS->labels()) {
         JumpDest Dest = getJumpDestForLabel(E->getLabel());
         Transfer.push_back(Dest.getBlock());
         if (!Constraints.empty())
           Constraints += ',';
         Constraints += "!i";
llvm::InlineAsm::AsmDialect GnuAsmDialect =
           : llvm::InlineAsm::AD_Intel;
   llvm::InlineAsm::AsmDialect AsmDialect = isa<MSAsmStmt>(&S) ?
     llvm::InlineAsm::AD_Intel : GnuAsmDialect;
 
   llvm::InlineAsm *IA = llvm::InlineAsm::get(
       FTy, AsmString, Constraints, HasSideEffect,
       /* IsAlignStack */ false, AsmDialect, HasUnwindClobber);
   std::vector<llvm::Value*> RegResults;
+  llvm::CallBrInst *CBR;
+  // A 2D vector, indexed by indirect destination, then ResultReg.
+  std::vector<std::vector<llvm::Value*>> CBRRegResults;
   if (IsGCCAsmGoto) {
-    llvm::CallBrInst *Result =
-        Builder.CreateCallBr(IA, Fallthrough, Transfer, Args);
+    CBR = Builder.CreateCallBr(IA, Fallthrough, Transfer, Args);
     EmitBlock(Fallthrough);
-    UpdateAsmCallInst(cast<llvm::CallBase>(*Result), HasSideEffect, false,
-                      ReadOnly, ReadNone, InNoMergeAttributedStmt, S,
-                      ResultRegTypes, ArgElemTypes, *this, RegResults);
+    UpdateAsmCallInst(*CBR, HasSideEffect, false, ReadOnly, ReadNone,
+                      InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
+                      *this, RegResults);
+    // Because we are emitting code top to bottom, we don't have enough
+    // information at this point to know precisely whether we have a critical
+    // edge. If we have outputs, split all indirect destinations.
+    if (RegResults.size()) {
+      CBRRegResults.resize(CBR->getNumIndirectDests());
+      for (unsigned i = 0, e = CBR->getNumIndirectDests(); i != e; ++i) {
+        CBRRegResults[i].resize(ResultRegTypes.size());
+        llvm::BasicBlock *Dest = CBR->getIndirectDest(i);
+        llvm::Twine SynthName = Dest->getName() + ".split";
+        llvm::BasicBlock *SynthBB = createBasicBlock(SynthName);
+        llvm::IRBuilderBase::InsertPointGuard IPG(Builder);
+        Builder.SetInsertPoint(SynthBB);
+
+        if (ResultRegTypes.size() == 1) {
+          CBRRegResults[i][0] = CBR;
+        } else {
+          for (unsigned j = 0, e = ResultRegTypes.size(); j != e; ++j) {
+            llvm::Value *Tmp = Builder.CreateExtractValue(CBR, j, "asmresult");
+            CBRRegResults[i][j] = Tmp;
+          }
+        }
+        EmitBranch(Dest);
+        EmitBlock(SynthBB);
+        CBR->setIndirectDest(i, SynthBB);
+      }
+    }
   } else if (HasUnwindClobber) {
     llvm::CallBase *Result = EmitCallOrInvoke(IA, Args, "");
     UpdateAsmCallInst(*Result, HasSideEffect, true, ReadOnly, ReadNone,
                       InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
                       *this, RegResults);
   } else {
-    llvm::CallInst *Result =
-        Builder.CreateCall(IA, Args, getBundlesForFunclet(IA));
-    UpdateAsmCallInst(cast<llvm::CallBase>(*Result), HasSideEffect, false,
-                      ReadOnly, ReadNone, InNoMergeAttributedStmt, S,
-                      ResultRegTypes, ArgElemTypes, *this, RegResults);
+    llvm::CallInst *Result = Builder.CreateCall(IA, Args, getBundlesForFunclet(IA));
+    UpdateAsmCallInst(*Result, HasSideEffect, false, ReadOnly, ReadNone,
+                      InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
+                      *this, RegResults);
   }
 
   assert(RegResults.size() == ResultRegTypes.size());
   assert(RegResults.size() == ResultTruncRegTypes.size());
   assert(RegResults.size() == ResultRegDests.size());
   // ResultRegDests can be also populated by addReturnRegisterOutputs() above,
   // in which case its size may grow.
   assert(ResultTypeRequiresCast.size() <= ResultRegDests.size());
   assert(ResultRegIsFlagReg.size() <= ResultRegDests.size());
-  for (unsigned i = 0, e = RegResults.size(); i != e; ++i) {
-    llvm::Value *Tmp = RegResults[i];
-    llvm::Type *TruncTy = ResultTruncRegTypes[i];
-
-    if ((i < ResultRegIsFlagReg.size()) && ResultRegIsFlagReg[i]) {
-      // Target must guarantee the Value `Tmp` here is lowered to a boolean
-      // value.
-      llvm::Constant *Two = llvm::ConstantInt::get(Tmp->getType(), 2);
-      llvm::Value *IsBooleanValue =
-          Builder.CreateCmp(llvm::CmpInst::ICMP_ULT, Tmp, Two);
-      llvm::Function *FnAssume = CGM.getIntrinsic(llvm::Intrinsic::assume);
-      Builder.CreateCall(FnAssume, IsBooleanValue);
-    }
 
-    // If the result type of the LLVM IR asm doesn't match the result type of
-    // the expression, do the conversion.
-    if (ResultRegTypes[i] != ResultTruncRegTypes[i]) {
-
-      // Truncate the integer result to the right size, note that TruncTy can be
-      // a pointer.
-      if (TruncTy->isFloatingPointTy())
-        Tmp = Builder.CreateFPTrunc(Tmp, TruncTy);
-      else if (TruncTy->isPointerTy() && Tmp->getType()->isIntegerTy()) {
-        uint64_t ResSize = CGM.getDataLayout().getTypeSizeInBits(TruncTy);
-        Tmp = Builder.CreateTrunc(Tmp,
-                   llvm::IntegerType::get(getLLVMContext(), (unsigned)ResSize));
-        Tmp = Builder.CreateIntToPtr(Tmp, TruncTy);
-      } else if (Tmp->getType()->isPointerTy() && TruncTy->isIntegerTy()) {
-        uint64_t TmpSize =CGM.getDataLayout().getTypeSizeInBits(Tmp->getType());
-        Tmp = Builder.CreatePtrToInt(Tmp,
-                   llvm::IntegerType::get(getLLVMContext(), (unsigned)TmpSize));
-        Tmp = Builder.CreateTrunc(Tmp, TruncTy);
-      } else if (TruncTy->isIntegerTy()) {
-        Tmp = Builder.CreateZExtOrTrunc(Tmp, TruncTy);
-      } else if (TruncTy->isVectorTy()) {
-        Tmp = Builder.CreateBitCast(Tmp, TruncTy);
-      }
-    }
-
-    LValue Dest = ResultRegDests[i];
-    // ResultTypeRequiresCast elements correspond to the first
-    // ResultTypeRequiresCast.size() elements of RegResults.
-    if ((i < ResultTypeRequiresCast.size()) && ResultTypeRequiresCast[i]) {
-      unsigned Size = getContext().getTypeSize(ResultRegQualTys[i]);
-      Address A = Builder.CreateElementBitCast(Dest.getAddress(*this),
-                                               ResultRegTypes[i]);
-      if (getTargetHooks().isScalarizableAsmOperand(*this, TruncTy)) {
-        Builder.CreateStore(Tmp, A);
+  EmitAsmStores(*this, S, RegResults, ResultRegTypes, ResultTruncRegTypes,
+      ResultRegDests, ResultRegQualTys, ResultTypeRequiresCast,
+      ResultRegIsFlagReg);
+
+  if (IsGCCAsmGoto && CBRRegResults.size()) {
+    for (unsigned i = 0, e = CBR->getNumIndirectDests(); i != e; ++i) {
+      assert(CBRRegResults[i].size() == ResultRegTypes.size());
+      // If we happen to share the same indirect and default dest, don't re-add
+      // stores. That was done for the default destination in the above call to
+      // EmitAsmStores.
+      llvm::BasicBlock *Succ = CBR->getIndirectDest(i);
+      if (Succ == CBR->getDefaultDest())
         continue;
-      }
-
-      QualType Ty = getContext().getIntTypeForBitwidth(Size, /*Signed*/ false);
-      if (Ty.isNull()) {
-        const Expr *OutExpr = S.getOutputExpr(i);
-        CGM.getDiags().Report(OutExpr->getExprLoc(),
-                              diag::err_store_value_to_reg);
-        return;
-      }
-      Dest = MakeAddrLValue(A, Ty);
-    }
-    EmitStoreThroughLValue(RValue::get(Tmp), Dest);
+      llvm::IRBuilderBase::InsertPointGuard IPG(Builder);
+      Builder.SetInsertPoint(Succ, --(Succ->end()));
+      EmitAsmStores(*this, S, CBRRegResults[i], ResultRegTypes, ResultTruncRegTypes,
+          ResultRegDests, ResultRegQualTys, ResultTypeRequiresCast,
+          ResultRegIsFlagReg);
   }
 }
 
 LValue CodeGenFunction::InitCapturedStruct(const CapturedStmt &S) {
   const RecordDecl *RD = S.getCapturedRecordDecl();
   QualType RecordTy = getContext().getRecordType(RD);
 
   // Initialize the captured struct.

clang/test/CodeGen/asm-goto.c

loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test2(int cond) {
   // CHECK-LABEL: define{{.*}} i32 @test2(
   // CHECK: callbr i32 asm sideeffect
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   asm volatile goto("testl %0, %0; jne %l2;" : "=r"(cond) : "r"(cond) :: label_true, loop);
   asm volatile goto("testl %0, %0; jne %l3;" : "=r"(cond) : "r"(cond) :: label_true, loop);
   // CHECK: callbr i32 asm sideeffect
-  // CHECK: to label %asm.fallthrough1 [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough1 [label %label_true.split2, label %loop.split3]
   // CHECK-LABEL: asm.fallthrough1:
   return 0;
 loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test3(int out1, int out2) {
   // CHECK-LABEL: define{{.*}} i32 @test3(
   // CHECK: callbr { i32, i32 } asm sideeffect
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   asm volatile goto("testl %0, %0; jne %l3;" : "=r"(out1), "=r"(out2) : "r"(out1) :: label_true, loop);
   asm volatile goto("testl %0, %0; jne %l4;" : "=r"(out1), "=r"(out2) : "r"(out1) :: label_true, loop);
   // CHECK: callbr { i32, i32 } asm sideeffect
-  // CHECK: to label %asm.fallthrough2 [label %label_true, label %loop]
-  // CHECK-LABEL: asm.fallthrough2:
+  // CHECK: to label %asm.fallthrough6 [label %label_true.split11, label %loop.split14]
+  // CHECK-LABEL: asm.fallthrough6:
   return 0;
 loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test4(int out1, int out2) {
   // CHECK-LABEL: define{{.*}} i32 @test4(
   // CHECK: callbr { i32, i32 } asm sideeffect "jne ${5:l}", "={si},={di},r,0,1,!i,!i
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   if (out1 < out2)
     asm volatile goto("jne %l5" : "+S"(out1), "+D"(out2) : "r"(out1) :: label_true, loop);
   else
     asm volatile goto("jne %l7" : "+S"(out1), "+D"(out2) : "r"(out1), "r"(out2) :: label_true, loop);
   // CHECK: callbr { i32, i32 } asm sideeffect "jne ${7:l}", "={si},={di},r,r,0,1,!i,!i
-  // CHECK: to label %asm.fallthrough2 [label %label_true, label %loop]
-  // CHECK-LABEL: asm.fallthrough2:
+  // CHECK: to label %asm.fallthrough6 [label %label_true.split11, label %loop.split14]
+  // CHECK-LABEL: asm.fallthrough6:
   return out1 + out2;
 loop:
   return -1;
 label_true:
   return -2;
 }
 
 int test5(int addr, int size, int limit) {
   // CHECK-LABEL: define{{.*}} i32 @test5(
   // CHECK: callbr i32 asm "add $1,$0 ; jc ${4:l} ; cmp $2,$0 ; ja ${4:l} ; ", "=r,imr,imr,0,!i
-  // CHECK: to label %asm.fallthrough [label %t_err]
+  // CHECK: to label %asm.fallthrough [label %t_err.split]
   // CHECK-LABEL: asm.fallthrough:
   asm goto(
       "add %1,%0 ; "
       "jc %l[t_err] ; "
       "cmp %2,%0 ; "
       "ja %l[t_err] ; "
       : "+r" (addr)
       : "g" (size), "g" (limit)
       : : t_err);
   return 0;
 t_err:
   return 1;
 }
 
 int test6(int out1) {
   // CHECK-LABEL: define{{.*}} i32 @test6(
   // CHECK: callbr i32 asm sideeffect "testl $0, $0; testl $1, $1; jne ${3:l}", "={si},r,0,!i,!i,{{.*}}
-  // CHECK: to label %asm.fallthrough [label %label_true, label %landing]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %landing.split]
   // CHECK-LABEL: asm.fallthrough:
   // CHECK-LABEL: landing:
   int out2 = 42;
   asm volatile goto("testl %0, %0; testl %1, %1; jne %l3" : "+S"(out2) : "r"(out1) :: label_true, landing);
 landing:
   return out1 + out2;
 label_true:
   return -2;
 }
 
 // test7 - For the output templates in the asm string (%0, %l2), GCC places
 // hidden inputs tied to outputs ("+r" constraint) BEFORE labels. Test that foo
 // is $2 (or rather ${2:l} because of the l output template) in the emitted asm
 // string, not $1.
 void *test7(void) {
   // CHECK-LABEL: define{{.*}} ptr @test7(
   // CHECK: %1 = callbr ptr asm "# $0\0A\09# ${2:l}", "=r,0,!i,~{dirflag},~{fpsr},~{flags}"(ptr %0)
-  // CHECK-NEXT: to label %asm.fallthrough [label %foo]
+  // CHECK-NEXT: to label %asm.fallthrough [label %foo.split]
   void *p = &&foo;
   asm goto ("# %0\n\t# %l2":"+r"(p):::foo);
 foo:
   return p;
 }
 
 // test8 - the same as test7, but this time we use symbolic names rather than
 // numbered outputs.
 void *test8(void) {
   // CHECK-LABEL: define{{.*}} ptr @test8(
   // CHECK: %1 = callbr ptr asm "# $0\0A\09# ${2:l}", "=r,0,!i,~{dirflag},~{fpsr},~{flags}"(ptr %0)
-  // CHECK-NEXT: to label %asm.fallthrough [label %foo]
+  // CHECK-NEXT: to label %asm.fallthrough [label %foo.split]
   void *p = &&foo;
   asm goto ("# %0\n\t# %l[foo]":"+r"(p):::foo);
 foo:
   return p;
 }

clang/test/CodeGen/asm-goto2.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: x86-registered-target
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -O0 -emit-llvm %s -o - | FileCheck %s
+
+// CHECK-LABEL: @test0(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1:[0-9]+]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !2
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 42, ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+// CHECK:       z.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test0 (void) {
+  int ret;
+  asm goto ("" : "=r"(ret):::z);
+  ret = 42;
+z:
+  return ret;
+}
+
+// CHECK-LABEL: @test1(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !3
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT1:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT1]], ptr [[B]], align 4
+// CHECK-NEXT:    store i32 42, ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+// CHECK:       z.split:
+// CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT3:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT2]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT3]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test1 (void) {
+  int ret, b;
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+  ret = 42;
+z:
+  return ret;
+}
+
+// CHECK-LABEL: @test2(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !4
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT1:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT1]], ptr [[B]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH4:%.*]] [label %z.split9], !srcloc !5
+// CHECK:       asm.fallthrough4:
+// CHECK-NEXT:    [[ASMRESULT5:%.*]] = extractvalue { i32, i32 } [[TMP1]], 0
+// CHECK-NEXT:    [[ASMRESULT6:%.*]] = extractvalue { i32, i32 } [[TMP1]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT5]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT6]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP2]]
+// CHECK:       z.split:
+// CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT3:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT2]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT3]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+// CHECK:       z.split9:
+// CHECK-NEXT:    [[ASMRESULT7:%.*]] = extractvalue { i32, i32 } [[TMP1]], 0
+// CHECK-NEXT:    [[ASMRESULT8:%.*]] = extractvalue { i32, i32 } [[TMP1]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT7]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT8]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test2 (void) {
+  int ret, b;
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+z:
+  return ret;
+}
+// CHECK-LABEL: @test3(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[OUT1_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i32 [[OUT1:%.*]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label [[LABEL_TRUE_SPLIT:%.*]], label %loop.split], !srcloc !6
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN:%.*]]
+// CHECK:       label_true.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    br label [[LABEL_TRUE:%.*]]
+// CHECK:       loop.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    br label [[LOOP:%.*]]
+// CHECK:       loop:
+// CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN]]
+// CHECK:       label_true:
+// CHECK-NEXT:    store i32 1, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN]]
+// CHECK:       return:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+//
+int test3 (int out1) {
+  asm goto("" : "=r"(out1)::: label_true, loop);
+  return 0;
+loop:
+  return 0;
+label_true:
+  return 1;
+}
+
+// CHECK-LABEL: @test4(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[X:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    br label [[FOO:%.*]]
+// CHECK:       foo:
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %foo.split], !srcloc !7
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[X]], align 4
+// CHECK-NEXT:    ret void
+// CHECK:       foo.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[X]], align 4
+// CHECK-NEXT:    br label [[FOO]]
+//
+void test4 (void) {
+  int x;
+foo:
+  asm goto ("" : "=r"(x):::foo);
+}

clang/test/Modules/asm-goto.c

 // RUN: rm -rf %t
 // RUN: %clang_cc1 -fmodules -fno-implicit-modules -x c -I%S/Inputs/asm-goto -emit-module %S/Inputs/asm-goto/module.modulemap -fmodule-name=a -o %t/a.pcm
 // RUN: %clang_cc1 -fmodules -fno-implicit-modules -x c -I%S/Inputs/asm-goto -emit-llvm -o - %s -fmodule-file=%t/a.pcm | FileCheck %s
 #include "a.h"
 
 // CHECK-LABEL: define {{.*}} @foo(
 // CHECK: callbr {{.*}} "=r,!i{{.*}}()
-// CHECK-NEXT: to label %asm.fallthrough [label %indirect]
+// CHECK-NEXT: to label %asm.fallthrough [label %indirect.split]
 
 int bar(void) {
   return foo();
 }