[Clang] support for outputs along indirect edges of asm goto

[nickdesaulniers]

Summary

Initial support for asm goto w/ outputs (D69876) only supported outputs
along the "default" (aka "fallthrough") edge.

We can support outputs along all edges by repeating the same pattern of
stores along the indirect edges that we allready do for the default
edge. One complication is that these indirect edges may be critical
edges which would need to be split. Another issue is that mid-codgen of
LLVM IR, the control flow graph might not reflect the control flow of
the final function.

To avoid this "chicken and the egg" problem assume that any given
indirect edge may become a critical edge, and pro-actively split it.
This is unnecessary if the edge does not become critical, but LLVM will
optimize such cases via tail duplication.

Fixes: https://github.com/llvm/llvm-project/issues/53562

[nickdesaulniers]

• add note to commit message (let's see if phab captures what I wrote, used arc diff --verbatim)

[nickdesaulniers]

• git clang-format HEAD~

[nickdesaulniers]

Note to reviewers: I expect this to be one of a 3 part progression.

1. Fix DomTree D135997
2. Fix front end D136497
3. Fix back end (TODO)

Hopefully that's it; famous last words...

[nickdesaulniers]

• update LanguageExtensions doc

[void]

It might be easier to see the main changes here if you submit the (very nice) refactoring of EmitAsmStores first.

clang/lib/CodeGen/CGStmt.cpp

2361

s/ResultTruncRegTypes[i]/TruncTy/

2365

This looks like a direct copy from below (which is fine). I'm a bit iffy on whether this covers *all* of the value types that could come through here...

2859–2860

Should these asserts (or some of them) be moved into EmitAsmStores()?

2863

nit: Could you add a comment explaining that we're calling EmitAsmStores for asm goto's indirect branches? I was slightly confused about why we were calling EmitAsmStores more than once. :)

[nickdesaulniers]

In D136497#3889938, @void wrote:

It might be easier to see the main changes here if you submit the (very nice) refactoring of EmitAsmStores first.

D137113 TODO(Nick): rebase on D137113

clang/lib/CodeGen/CGStmt.cpp

2361

Done in D137113

2365

Note: this code was simply moved wholesale. If value types are missing, that is a pre-existing bug and should be a prerequisite to fix for this patch.

I've broken out D137113 as a child patch to make that clearer.

2365

s/should/shouldn't/

2859–2860

Ah, yeah, let me add those to D137113.

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2859–2860

Now done in D137113.

[nickdesaulniers]

• rebase on D137113, add comment

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2869

Oh, I can remove this now, I moved it into EmitAsmStores.

2870–2875

I'm not sure that I need to handle this case (of a callbr with the same indirect destination as the default destination). That might result from optimizations, but I don't think clang can or will generate such IR. Maybe I should turn this into an: assert(CBR->getIndirectDest(i) != CBR->getDefaultDest(i) && "We already emitted asm stores in the default dest");?

[void]

clang/lib/CodeGen/CGStmt.cpp

2870–2875

It could only happen with something like this:

asm goto (... :::: indirect);

indirect:;

In that simplified case, it looks like clang emits this:

define dso_local void @foo() #0 {
  callbr void asm sideeffect "", "i,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@foo, %2)) #1
          to label %1 [label %2], !srcloc !6

1:                                                ; preds = %0
  br label %2

2:                                                ; preds = %1, %0
  ret void
}

So it's probably safe to make this an assert.

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2870–2875

Yeah, it's impossible. While a CallBrInst can have a same indirect destination as the default destination as a result of optimizations, above we always create an "asm.fallthrough" block that is distinct from any of the named labels from the asm goto.

[nickdesaulniers]

• remove unnecssary asserts

[nickdesaulniers]

clang/lib/CodeGen/CGStmt.cpp

2870–2875

Oh, it's not just the default and indirects sharing, but what if the same indirect is listed twice? I should add a test for that...i.e.

asm goto (... :::: indirect,indirect);

[nickdesaulniers]

• reroll for a new sha so that phab unmarks this as "changes planned"

clang/lib/CodeGen/CGStmt.cpp

2870–2875

Oh, right, we don't allow that.

error: duplicate use of asm operand name "indirect"
note: asm operand name "indirect" first referenced here

I'm going to leave these asserts out and mark this thread done, but please reopen this thread with a new comment if you feel strongly that I should add it back (assert(CBR->getIndirectDest(i) != CBR->getDefaultDest(i) && "We already emitted asm stores in the default dest");).

[nickdesaulniers]

• rebase, use new callbrpad inst

[nickdesaulniers]

Probably should copy the release note change from https://reviews.llvm.org/D138078.

[nickdesaulniers]

• rebase, drop callbrpad inst

[nickdesaulniers]

• add clang release notes from D138078 and new one for new feature.

[nickdesaulniers]

Ready for review:

1. https://reviews.llvm.org/D135997
2. https://reviews.llvm.org/D140166
3. https://reviews.llvm.org/D139861
4. https://reviews.llvm.org/D139872
5. https://reviews.llvm.org/D139883
6. https://reviews.llvm.org/D139970
7. https://reviews.llvm.org/D140160
8. https://reviews.llvm.org/D137113
9. https://reviews.llvm.org/D136497

[nickdesaulniers]

• rebase, format

[nickdesaulniers]

• rebase

[void]

clang/docs/ReleaseNotes.rst

64

Is it worth it to mention that this change makes Clang's behavior with regards to your example here consistent with GCC's behavior?

clang/lib/CodeGen/CGStmt.cpp

2815

It seems as if you really want a map indexed by the BasicBlock*. So something like this:

DenseMap<BasicBlock *, SmallVector<Value *, 4>> // 4 should be enough for anyone.

That way you can iterate over CBR->getIndirectDests() instead of having to use an iterator.

2867

Could probably use !CBRRegResults.empty() instead of .size(). The first should be O(1) (granted .size() probably should be too, but .empty() is more explicit anyway).

[nickdesaulniers]

• use densemap, update release note, use !empty rather than size as per @void

[efriedma]

clang/docs/LanguageExtensions.rst

1592

Maybe put a note about earlier releases, so people don't get confused if they use newer documentation than their version of clang.

Is there any chance we want a dedicated __has_extension flag?

[void]

One small comment about the documentation but L:GTM.

clang/docs/LanguageExtensions.rst

1592

Is there any chance we want a dedicated __has_extension flag?

Might not be a bad idea just in case they have more than one compiler version they're using. Then again, if they add code to use this feature, they're fairly committed to the assumption that it'll "just work" even with a __has_extension flag...(Remember that we currently don't actually warn if they're using the outputs on the indirect branch. We just specify that such behavior is undefined.)

I think a strongly worded note here that it's available with Clang 16+ (or whatever) is going to be slightly better. It'll force them to use macros to support multiple compiler versions, making their code wicked ugly, and them possibly coming up with better ways to ensure that the correct compiler is used for this feature.

[nickdesaulniers]

clang/docs/LanguageExtensions.rst

1592

I've added that as a separate commit https://reviews.llvm.org/D143205 to not hold this up. PTAL. Open to better names for the identifier.

[nickdesaulniers]

• rebase

[nickdesaulniers]

• final rebase

[uabelho]

I noticed that gvn-hoist doesn't handle this very nicely, I just wrote https://github.com/llvm/llvm-project/issues/61023

clang/docs/LanguageExtensions.rst

 :doc:`BlockLanguageSpec<BlockLanguageSpec>`. Implementation and ABI details for
 the clang implementation are in :doc:`Block-ABI-Apple<Block-ABI-Apple>`.
 
 Query for this feature with ``__has_extension(blocks)``.
 
 ASM Goto with Output Constraints
 ================================
 
-.. note::
-
-  Clang's implementation of ASM goto differs from `GCC's
-  implementation <https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html>`_ in
-  that Clang doesn't yet support outputs on the indirect branch. Use of an
-  output on the indirect branch may result in undefined behavior and should be
-  avoided. E.g., in the following `z` isn't valid when used and may have
-  different values depending on optimization levels. (Clang may not warn about
-  such constructs.)
-
-  .. code-block:: c++
-
-    int foo(int x) {
-      int y, z;
-      asm goto(... : "=r"(y), "=r"(z): "r"(x) : : err);
-      return y;
-    err:
-      return z;
-    }
+Outputs may be used along any branches from the ``asm goto`` whether the
+branches are taken or not.
 
 When using tied-outputs (i.e. outputs that are inputs and outputs, not just
 outputs) with the `+r` constraint, there is a hidden input that's created
 before the label, so numeric references to operands must account for that.
 
 .. code-block:: c++
 
   int foo(int x) {

clang/docs/ReleaseNotes.rst

 ============================
 These changes are ones which we think may surprise users when upgrading to
 Clang |release| because of the opportunity they pose for disruption to existing
 code bases.
 
 
 C/C++ Language Potentially Breaking Changes
 -------------------------------------------
+- Indirect edges of asm goto statements under certain circumstances may now be
+  split. In previous releases of clang, that means for the following code the
+  two inputs may have compared equal in the inline assembly.  This is no longer
+  guaranteed (and necessary to support outputs along indirect edges, which is
+  now supported as of this release). This change is more consistent with the
+  behavior of GCC.
+
+  .. code-block:: c
+
+    foo: asm goto ("# %0 %1"::"i"(&&foo)::foo);
 
 C++ Specific Potentially Breaking Changes
 -----------------------------------------
 
 ABI Changes in This Version
 ---------------------------
 
 What's New in Clang |release|?
 ==============================
 Some of the major new features and improvements to Clang are listed
 here. Generic improvements to Clang as a whole or to its underlying
 infrastructure are described first, followed by language-specific
 sections with improvements to Clang's support for those languages.
 
 C++ Language Changes
 --------------------
 - Improved ``-O0`` code generation for calls to ``std::forward_like``. Similarly to
   ``std::move, std::forward`` et al. it is now treated as a compiler builtin and implemented
   directly rather than instantiating the definition from the standard library.
 
 C++20 Feature Support
 ^^^^^^^^^^^^^^^^^^^^^
 
 C++2b Feature Support
 ^^^^^^^^^^^^^^^^^^^^^
 
 Resolutions to C++ Defect Reports
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 C Language Changes
 ------------------
+- Support for outputs from asm goto statements along indirect edges has been
+  added. This fixes
+  `Issue 53562 <https://github.com/llvm/llvm-project/issues/53562`_.
 
 C2x Feature Support
 ^^^^^^^^^^^^^^^^^^^
 - Implemented the ``unreachable`` macro in freestanding ``<stddef.h>`` for
   `WG14 N2826 <https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2826.pdf>`_
 
 Non-comprehensive list of changes in this release
 -------------------------------------------------

clang/lib/CodeGen/CGStmt.cpp

 #include "clang/AST/Stmt.h"
 #include "clang/AST/StmtVisitor.h"
 #include "clang/Basic/Builtins.h"
 #include "clang/Basic/DiagnosticSema.h"
 #include "clang/Basic/PrettyStackTrace.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TargetInfo.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/IR/Assumptions.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/Support/SaveAndRestore.h"
EmitAsmStores(CodeGenFunction &CGF, const AsmStmt &S,
   for (unsigned i = 0, e = RegResults.size(); i != e; ++i) {
     llvm::Value *Tmp = RegResults[i];
     llvm::Type *TruncTy = ResultTruncRegTypes[i];
 
     if ((i < ResultRegIsFlagReg.size()) && ResultRegIsFlagReg[i]) {
       // Target must guarantee the Value `Tmp` here is lowered to a boolean
       // value.
       llvm::Constant *Two = llvm::ConstantInt::get(Tmp->getType(), 2);
       llvm::Value *IsBooleanValue =
           Builder.CreateCmp(llvm::CmpInst::ICMP_ULT, Tmp, Two);
       llvm::Function *FnAssume = CGM.getIntrinsic(llvm::Intrinsic::assume);
       Builder.CreateCall(FnAssume, IsBooleanValue);
     }
 
     // If the result type of the LLVM IR asm doesn't match the result type of
     // the expression, do the conversion.
     if (ResultRegTypes[i] != TruncTy) {
 
       // Truncate the integer result to the right size, note that TruncTy can be
       // a pointer.
       if (TruncTy->isFloatingPointTy())
llvm::InlineAsm::AsmDialect GnuAsmDialect =
           : llvm::InlineAsm::AD_Intel;
   llvm::InlineAsm::AsmDialect AsmDialect = isa<MSAsmStmt>(&S) ?
     llvm::InlineAsm::AD_Intel : GnuAsmDialect;
 
   llvm::InlineAsm *IA = llvm::InlineAsm::get(
       FTy, AsmString, Constraints, HasSideEffect,
       /* IsAlignStack */ false, AsmDialect, HasUnwindClobber);
   std::vector<llvm::Value*> RegResults;
+  llvm::CallBrInst *CBR;
+  llvm::DenseMap<llvm::BasicBlock *, SmallVector<llvm::Value *, 4>>
+      CBRRegResults;
   if (IsGCCAsmGoto) {
-    llvm::CallBrInst *Result =
-        Builder.CreateCallBr(IA, Fallthrough, Transfer, Args);
+    CBR = Builder.CreateCallBr(IA, Fallthrough, Transfer, Args);
     EmitBlock(Fallthrough);
-    UpdateAsmCallInst(*Result, HasSideEffect, false, ReadOnly, ReadNone,
+    UpdateAsmCallInst(*CBR, HasSideEffect, false, ReadOnly, ReadNone,
                       InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
                       *this, RegResults);
+    // Because we are emitting code top to bottom, we don't have enough
+    // information at this point to know precisely whether we have a critical
+    // edge. If we have outputs, split all indirect destinations.
+    if (!RegResults.empty()) {
+      unsigned i = 0;
+      for (llvm::BasicBlock *Dest : CBR->getIndirectDests()) {
+        llvm::Twine SynthName = Dest->getName() + ".split";
+        llvm::BasicBlock *SynthBB = createBasicBlock(SynthName);
+        llvm::IRBuilderBase::InsertPointGuard IPG(Builder);
+        Builder.SetInsertPoint(SynthBB);
+
+        if (ResultRegTypes.size() == 1) {
+          CBRRegResults[SynthBB].push_back(CBR);
+        } else {
+          for (unsigned j = 0, e = ResultRegTypes.size(); j != e; ++j) {
+            llvm::Value *Tmp = Builder.CreateExtractValue(CBR, j, "asmresult");
+            CBRRegResults[SynthBB].push_back(Tmp);
+          }
+        }
+
+        EmitBranch(Dest);
+        EmitBlock(SynthBB);
+        CBR->setIndirectDest(i++, SynthBB);
+      }
+    }
   } else if (HasUnwindClobber) {
     llvm::CallBase *Result = EmitCallOrInvoke(IA, Args, "");
     UpdateAsmCallInst(*Result, HasSideEffect, true, ReadOnly, ReadNone,
                       InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
                       *this, RegResults);
   } else {
     llvm::CallInst *Result =
         Builder.CreateCall(IA, Args, getBundlesForFunclet(IA));
     UpdateAsmCallInst(*Result, HasSideEffect, false, ReadOnly, ReadNone,
                       InNoMergeAttributedStmt, S, ResultRegTypes, ArgElemTypes,
                       *this, RegResults);
   }
 
   EmitAsmStores(*this, S, RegResults, ResultRegTypes, ResultTruncRegTypes,
                 ResultRegDests, ResultRegQualTys, ResultTypeRequiresCast,
                 ResultRegIsFlagReg);
+
+  // If this is an asm goto with outputs, repeat EmitAsmStores, but with a
+  // different insertion point; one for each indirect destination and with
+  // CBRRegResults rather than RegResults.
+  if (IsGCCAsmGoto && !CBRRegResults.empty()) {
+    for (llvm::BasicBlock *Succ : CBR->getIndirectDests()) {
+      llvm::IRBuilderBase::InsertPointGuard IPG(Builder);
+      Builder.SetInsertPoint(Succ, --(Succ->end()));
+      EmitAsmStores(*this, S, CBRRegResults[Succ], ResultRegTypes,
+                    ResultTruncRegTypes, ResultRegDests, ResultRegQualTys,
+                    ResultTypeRequiresCast, ResultRegIsFlagReg);
+    }
+  }
 }
 
 LValue CodeGenFunction::InitCapturedStruct(const CapturedStmt &S) {
   const RecordDecl *RD = S.getCapturedRecordDecl();
   QualType RecordTy = getContext().getRecordType(RD);
 
   // Initialize the captured struct.
   LValue SlotLV =

clang/test/CodeGen/asm-goto.c

loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test2(int cond) {
   // CHECK-LABEL: define{{.*}} i32 @test2(
   // CHECK: callbr i32 asm sideeffect
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   asm volatile goto("testl %0, %0; jne %l2;" : "=r"(cond) : "r"(cond) :: label_true, loop);
   asm volatile goto("testl %0, %0; jne %l3;" : "=r"(cond) : "r"(cond) :: label_true, loop);
   // CHECK: callbr i32 asm sideeffect
-  // CHECK: to label %asm.fallthrough1 [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough1 [label %label_true.split2, label %loop.split3]
   // CHECK-LABEL: asm.fallthrough1:
   return 0;
 loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test3(int out1, int out2) {
   // CHECK-LABEL: define{{.*}} i32 @test3(
   // CHECK: callbr { i32, i32 } asm sideeffect
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   asm volatile goto("testl %0, %0; jne %l3;" : "=r"(out1), "=r"(out2) : "r"(out1) :: label_true, loop);
   asm volatile goto("testl %0, %0; jne %l4;" : "=r"(out1), "=r"(out2) : "r"(out1) :: label_true, loop);
   // CHECK: callbr { i32, i32 } asm sideeffect
-  // CHECK: to label %asm.fallthrough2 [label %label_true, label %loop]
-  // CHECK-LABEL: asm.fallthrough2:
+  // CHECK: to label %asm.fallthrough6 [label %label_true.split11, label %loop.split14]
+  // CHECK-LABEL: asm.fallthrough6:
   return 0;
 loop:
   return 0;
 label_true:
   return 1;
 }
 
 int test4(int out1, int out2) {
   // CHECK-LABEL: define{{.*}} i32 @test4(
   // CHECK: callbr { i32, i32 } asm sideeffect "jne ${5:l}", "={si},={di},r,0,1,!i,!i
-  // CHECK: to label %asm.fallthrough [label %label_true, label %loop]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %loop.split]
   // CHECK-LABEL: asm.fallthrough:
   if (out1 < out2)
     asm volatile goto("jne %l5" : "+S"(out1), "+D"(out2) : "r"(out1) :: label_true, loop);
   else
     asm volatile goto("jne %l7" : "+S"(out1), "+D"(out2) : "r"(out1), "r"(out2) :: label_true, loop);
   // CHECK: callbr { i32, i32 } asm sideeffect "jne ${7:l}", "={si},={di},r,r,0,1,!i,!i
-  // CHECK: to label %asm.fallthrough2 [label %label_true, label %loop]
-  // CHECK-LABEL: asm.fallthrough2:
+  // CHECK: to label %asm.fallthrough6 [label %label_true.split11, label %loop.split14]
+  // CHECK-LABEL: asm.fallthrough6:
   return out1 + out2;
 loop:
   return -1;
 label_true:
   return -2;
 }
 
 int test5(int addr, int size, int limit) {
   // CHECK-LABEL: define{{.*}} i32 @test5(
   // CHECK: callbr i32 asm "add $1,$0 ; jc ${4:l} ; cmp $2,$0 ; ja ${4:l} ; ", "=r,imr,imr,0,!i
-  // CHECK: to label %asm.fallthrough [label %t_err]
+  // CHECK: to label %asm.fallthrough [label %t_err.split]
   // CHECK-LABEL: asm.fallthrough:
   asm goto(
       "add %1,%0 ; "
       "jc %l[t_err] ; "
       "cmp %2,%0 ; "
       "ja %l[t_err] ; "
       : "+r" (addr)
       : "g" (size), "g" (limit)
       : : t_err);
   return 0;
 t_err:
   return 1;
 }
 
 int test6(int out1) {
   // CHECK-LABEL: define{{.*}} i32 @test6(
   // CHECK: callbr i32 asm sideeffect "testl $0, $0; testl $1, $1; jne ${3:l}", "={si},r,0,!i,!i,{{.*}}
-  // CHECK: to label %asm.fallthrough [label %label_true, label %landing]
+  // CHECK: to label %asm.fallthrough [label %label_true.split, label %landing.split]
   // CHECK-LABEL: asm.fallthrough:
   // CHECK-LABEL: landing:
   int out2 = 42;
   asm volatile goto("testl %0, %0; testl %1, %1; jne %l3" : "+S"(out2) : "r"(out1) :: label_true, landing);
 landing:
   return out1 + out2;
 label_true:
   return -2;
 }
 
 // test7 - For the output templates in the asm string (%0, %l2), GCC places
 // hidden inputs tied to outputs ("+r" constraint) BEFORE labels. Test that foo
 // is $2 (or rather ${2:l} because of the l output template) in the emitted asm
 // string, not $1.
 void *test7(void) {
   // CHECK-LABEL: define{{.*}} ptr @test7(
   // CHECK: %1 = callbr ptr asm "# $0\0A\09# ${2:l}", "=r,0,!i,~{dirflag},~{fpsr},~{flags}"(ptr %0)
-  // CHECK-NEXT: to label %asm.fallthrough [label %foo]
+  // CHECK-NEXT: to label %asm.fallthrough [label %foo.split]
   void *p = &&foo;
   asm goto ("# %0\n\t# %l2":"+r"(p):::foo);
 foo:
   return p;
 }
 
 // test8 - the same as test7, but this time we use symbolic names rather than
 // numbered outputs.
 void *test8(void) {
   // CHECK-LABEL: define{{.*}} ptr @test8(
   // CHECK: %1 = callbr ptr asm "# $0\0A\09# ${2:l}", "=r,0,!i,~{dirflag},~{fpsr},~{flags}"(ptr %0)
-  // CHECK-NEXT: to label %asm.fallthrough [label %foo]
+  // CHECK-NEXT: to label %asm.fallthrough [label %foo.split]
   void *p = &&foo;
   asm goto ("# %0\n\t# %l[foo]":"+r"(p):::foo);
 foo:
   return p;
 }

clang/test/CodeGen/asm-goto2.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: x86-registered-target
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -O0 -emit-llvm %s -o - | FileCheck %s
+
+// CHECK-LABEL: @test0(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1:[0-9]+]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !2
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 42, ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+// CHECK:       z.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test0 (void) {
+  int ret;
+  asm goto ("" : "=r"(ret):::z);
+  ret = 42;
+z:
+  return ret;
+}
+
+// CHECK-LABEL: @test1(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !3
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT1:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT1]], ptr [[B]], align 4
+// CHECK-NEXT:    store i32 42, ptr [[RET]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+// CHECK:       z.split:
+// CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT3:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT2]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT3]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test1 (void) {
+  int ret, b;
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+  ret = 42;
+z:
+  return ret;
+}
+
+// CHECK-LABEL: @test2(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RET:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[B:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %z.split], !srcloc !4
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    [[ASMRESULT:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT1:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT1]], ptr [[B]], align 4
+// CHECK-NEXT:    [[TMP1:%.*]] = callbr { i32, i32 } asm "", "=r,=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH4:%.*]] [label %z.split9], !srcloc !5
+// CHECK:       asm.fallthrough4:
+// CHECK-NEXT:    [[ASMRESULT5:%.*]] = extractvalue { i32, i32 } [[TMP1]], 0
+// CHECK-NEXT:    [[ASMRESULT6:%.*]] = extractvalue { i32, i32 } [[TMP1]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT5]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT6]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z:%.*]]
+// CHECK:       z:
+// CHECK-NEXT:    [[TMP2:%.*]] = load i32, ptr [[RET]], align 4
+// CHECK-NEXT:    ret i32 [[TMP2]]
+// CHECK:       z.split:
+// CHECK-NEXT:    [[ASMRESULT2:%.*]] = extractvalue { i32, i32 } [[TMP0]], 0
+// CHECK-NEXT:    [[ASMRESULT3:%.*]] = extractvalue { i32, i32 } [[TMP0]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT2]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT3]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+// CHECK:       z.split9:
+// CHECK-NEXT:    [[ASMRESULT7:%.*]] = extractvalue { i32, i32 } [[TMP1]], 0
+// CHECK-NEXT:    [[ASMRESULT8:%.*]] = extractvalue { i32, i32 } [[TMP1]], 1
+// CHECK-NEXT:    store i32 [[ASMRESULT7]], ptr [[RET]], align 4
+// CHECK-NEXT:    store i32 [[ASMRESULT8]], ptr [[B]], align 4
+// CHECK-NEXT:    br label [[Z]]
+//
+int test2 (void) {
+  int ret, b;
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+  asm goto ("" : "=r"(ret), "=r"(b):::z);
+z:
+  return ret;
+}
+// CHECK-LABEL: @test3(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    [[OUT1_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i32 [[OUT1:%.*]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label [[LABEL_TRUE_SPLIT:%.*]], label %loop.split], !srcloc !6
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN:%.*]]
+// CHECK:       label_true.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    br label [[LABEL_TRUE:%.*]]
+// CHECK:       loop.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[OUT1_ADDR]], align 4
+// CHECK-NEXT:    br label [[LOOP:%.*]]
+// CHECK:       loop:
+// CHECK-NEXT:    store i32 0, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN]]
+// CHECK:       label_true:
+// CHECK-NEXT:    store i32 1, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    br label [[RETURN]]
+// CHECK:       return:
+// CHECK-NEXT:    [[TMP1:%.*]] = load i32, ptr [[RETVAL]], align 4
+// CHECK-NEXT:    ret i32 [[TMP1]]
+//
+int test3 (int out1) {
+  asm goto("" : "=r"(out1)::: label_true, loop);
+  return 0;
+loop:
+  return 0;
+label_true:
+  return 1;
+}
+
+// CHECK-LABEL: @test4(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[X:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    br label [[FOO:%.*]]
+// CHECK:       foo:
+// CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,~{dirflag},~{fpsr},~{flags}"() #[[ATTR1]]
+// CHECK-NEXT:    to label [[ASM_FALLTHROUGH:%.*]] [label %foo.split], !srcloc !7
+// CHECK:       asm.fallthrough:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[X]], align 4
+// CHECK-NEXT:    ret void
+// CHECK:       foo.split:
+// CHECK-NEXT:    store i32 [[TMP0]], ptr [[X]], align 4
+// CHECK-NEXT:    br label [[FOO]]
+//
+void test4 (void) {
+  int x;
+foo:
+  asm goto ("" : "=r"(x):::foo);
+}

clang/test/Modules/asm-goto.c

 // RUN: rm -rf %t
 // RUN: %clang_cc1 -fmodules -fno-implicit-modules -x c -I%S/Inputs/asm-goto -emit-module %S/Inputs/asm-goto/module.modulemap -fmodule-name=a -o %t/a.pcm
 // RUN: %clang_cc1 -fmodules -fno-implicit-modules -x c -I%S/Inputs/asm-goto -emit-llvm -o - %s -fmodule-file=%t/a.pcm | FileCheck %s
 #include "a.h"
 
 // CHECK-LABEL: define {{.*}} @foo(
 // CHECK: callbr {{.*}} "=r,!i{{.*}}()
-// CHECK-NEXT: to label %asm.fallthrough [label %indirect]
+// CHECK-NEXT: to label %asm.fallthrough [label %indirect.split]
 
 int bar(void) {
   return foo();
 }