Add an @llvm.sideeffect intrinsic

[sunfish]

Summary

This patch implements Chandler's idea for supporting languages that require support for infinite loops with side effects, such as Rust, providing a solution to bug 965.

Specifically, it adds an llvm.sideeffect() intrinsic, which has no actual effect, but which appears to optimization passes to have obscure side effects, such that they don't optimize away loops containing it. It also teaches several optimization passes to ignore this intrinsic, so that it doesn't significantly impact optimization in most cases.

[efriedma]

docs/LangRef.rst

14208

A language that needs these is going to be inserting a lot of them; assuming a language doesn't have goto, I think the frontend needs to insert one at the beginning of every function and every loop. Is there some rule for coalescing and/or eliminating llvm.sideeffect calls which aren't necessary?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667

Why is it safe to drop the llvm.sideeffect call in SelectionDAG?

[sunfish]

docs/LangRef.rst

14208

Eliminating requires seeing a volatile access, or similar, nearby, because LLVM doesn't know when calls are guaranteed to perform side effects. Coalescing two @llvm.sideeffect calls would be straightforward. I haven't implemented these yet because if there's already a volatile access or an @llvm.sideeffect call nearby, deleting another @llvm.sideeffect call doesn't usually make much difference (to eg. functionattrs).

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667

Just that SelectionDAG and CodeGen after it don't currently have any optimizations that rely on forward-progress guarantees, and probably won't add them. Those kinds of optimizations are the optimizer's job.

If there's any situation where there's an actual loop in the user code that is at all possible to eliminate, it's always better to find a way to have the optimizer do it, because that's an enabling optimization that will make other parts of the optimizer more powerful.

[• dberlin]

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

[hfinkel]

Maybe we should do this in follow-up, but I'd also like to see some general text added to the LangRef about the semantics of loops (that makes reference to this intrinsic). Maybe we can say something like:

Loops that contains no calls that access inaccessible memory, perform no atomic or volatile memory access, nor otherwise synchronizes with other threads, are assumed to eventually terminate. If the source-level language has well-defined infinite loops, the LLVM IR representation of such loops should contain at least one call to `@llvm.sideeffect`.

We'll then need to update Clang to insert this intrinsic into loops (in C) that have a constant controlling condition (per C11 6.8.5p6).

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667

I'm not sure the situation is that straightforward. We can also use this information to assume things about the behavior of expressions that determine loop strip counts, and we might want to do that later in the pipeline too.

We don't have anything that does that now, however, so we'll just need to preserve this somehow if need be.

[sunfish]

In D38336#882813, @dberlin wrote:

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Another potential use was mentioned here.

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

[• dberlin]

In D38336#882864, @sunfish wrote:

In D38336#882813, @dberlin wrote:

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Another potential use was mentioned here.

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

(If it was, i'd request we either build an analysis like assume analysis, or circularly link these into a separate instruction list in basic blocks so you can tell in O(1) whether they are there)

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

SGTM

[sunfish]

In D38336#882873, @dberlin wrote:

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667

It wouldn't be complicated to add an ISD::SIDEEFFECT and TargetOpcode::SIDEEFFECT and carry them all the way through codegen, and to fix the various places that they pessimize, if it turns out to be important.

[• dberlin]

In D38336#882970, @sunfish wrote:

In D38336#882873, @dberlin wrote:

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.
Those mechanisms keep loops that are useful alive. They are definition of what makes a loop useful in the first place.
You said that llvm.sideeffect, by itself, should not be a reason to keep it alive if say, you can prove it finite or unreachable. It's only there to keep otherwise infinite empty loops alive.
But if you use the same mechanisms, it will always keep the loop alive, no matter what. Even if it's finite, etc.
In particular, those mechanisms are used to mark things that are still not able to be eliminated, for example, despite the loop being provably finite and otherwise empty.

So if llvm.sideeffect is not like those things, and does not mean the loop should stay alive if it's provably finite and otherwise empty (For example), i fail to see how the existence of those mechanisms helps.

[sunfish]

This update revises the documentation to clarify that the intrinsic doesn't inherently prevent loops from being optimized away.

In D38336#884104, @dberlin wrote:

In D38336#882970, @sunfish wrote:

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.

The basic idea of llvm.sideeffect is to just be a thing that gets treated like it has side effects. It's just like a volatile store through a non-escaped memory address.

I may have been unclear when you asked

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

and I said

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

I interpreted your question to be just about loops where we can prove the backedge is not taken. llvm.sideeffect does keep finite loops alive under the same conditions that other operations with side effects would.

In the future, we may want to optimize away redundant llvm.sideeffect calls, and an advanced form of this could allow optimizing away provably-finite loops containing llvm.sideeffect, however, the current patch doesn't depend on this.

[• dberlin]

In D38336#884469, @sunfish wrote:

This update revises the documentation to clarify that the intrinsic doesn't inherently prevent loops from being optimized away.

In D38336#884104, @dberlin wrote:

In D38336#882970, @sunfish wrote:

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.

The basic idea of llvm.sideeffect is to just be a thing that gets treated like it has side effects. It's just like a volatile store through a non-escaped memory address.

I may have been unclear when you asked

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

and I said

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

I interpreted your question to be just about loops where we can prove the backedge is not taken. llvm.sideeffect does keep finite loops alive under the same conditions that other operations with side effects would.

In the future, we may want to optimize away redundant llvm.sideeffect calls, and an advanced form of this could allow optimizing away provably-finite loops containing llvm.sideeffect, however, the current patch doesn't depend on this.

Okay, that helps, thanks.

[hfinkel]

LGTM

docs/LangRef.rst

 a constant initializer folded into a function body. This intrinsic can be
 used to avoid the possibility of overflows when loading from such a constant.
 
+'``llvm.sideeffect``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare void @llvm.sideeffect() inaccessiblememonly nounwind
+
+Overview:
+"""""""""
+
+The ``llvm.sideeffect`` intrinsic doesn't perform any operation. Optimizers
+treat it as having side effects, so it can be inserted into a loop to
+indicate that the loop shouldn't be assumed to terminate (which could
+potentially lead to the loop being optimized away entirely), even if it's
+an infinite loop with no other side effects.
+
+Arguments:
+""""""""""
+
+None.
+
+Semantics:
+""""""""""
+
+This intrinsic actually does nothing, but optimizers must assume that it
+has externally observable side effects.
+
 Stack Map Intrinsics
 --------------------
 

include/llvm/Analysis/TargetTransformInfoImpl.h

 
     case Intrinsic::annotation:
     case Intrinsic::assume:
+    case Intrinsic::sideeffect:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_value:
     case Intrinsic::invariant_start:

include/llvm/CodeGen/BasicTTIImpl.h

     // FIXME: We should return 0 whenever getIntrinsicCost == TCC_Free.
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
+    case Intrinsic::sideeffect:
       return 0;
     case Intrinsic::masked_store:
       return static_cast<T *>(this)

include/llvm/IR/Intrinsics.td

 // NOP: calls/invokes to this intrinsic are removed by codegen
 def int_donothing : Intrinsic<[], [], [IntrNoMem]>;
 
+// This instruction has no actual effect, though it is treated by the optimizer
+// has having opaque side effects. This may be inserted into loops to ensure
+// that they are not removed even if they turn out to be empty, for languages
+// which specify that infinite loops must be preserved.
+def int_sideeffect : Intrinsic<[], [], [IntrInaccessibleMemOnly]>;
+
 // Intrisics to support half precision floating point format
 let IntrProperties = [IntrNoMem] in {
 def int_convert_to_fp16   : Intrinsic<[llvm_i16_ty], [llvm_anyfloat_ty]>;

lib/Analysis/AliasSetTracker.cpp

       break;
       // FIXME: Add lifetime/invariant intrinsics (See: PR30807).
     case Intrinsic::assume:
+    case Intrinsic::sideeffect:
       return;
     }
   }

lib/Analysis/ValueTracking.cpp

       default: break;
       // FIXME: This list is repeated from NoTTI::getIntrinsicCost.
       case Intrinsic::assume:
+      case Intrinsic::sideeffect:
       case Intrinsic::dbg_declare:
       case Intrinsic::dbg_value:
       case Intrinsic::invariant_start:

     // FIXME: This isn't aggressive enough; a call which only writes to a global
     // is guaranteed to return.
     return CS.onlyReadsMemory() || CS.onlyAccessesArgMemory() ||
-           match(I, m_Intrinsic<Intrinsic::assume>());
+           match(I, m_Intrinsic<Intrinsic::assume>()) ||
+           match(I, m_Intrinsic<Intrinsic::sideeffect>());
   }
 
   // Other instructions return normally.

lib/Analysis/VectorUtils.cpp

     return Intrinsic::not_intrinsic;
 
   if (isTriviallyVectorizable(ID) || ID == Intrinsic::lifetime_start ||
-      ID == Intrinsic::lifetime_end || ID == Intrinsic::assume)
+      ID == Intrinsic::lifetime_end || ID == Intrinsic::assume ||
+      ID == Intrinsic::sideeffect)
     return ID;
   return Intrinsic::not_intrinsic;
 }

lib/CodeGen/SelectionDAG/FastISel.cpp

   case Intrinsic::lifetime_end:
   // The donothing intrinsic does, well, nothing.
   case Intrinsic::donothing:
+  // Neither does the sideeffect intrinsic.
+  case Intrinsic::sideeffect:
   // Neither does the assume intrinsic; it's also OK not to codegen its operand.
   case Intrinsic::assume:
     return true;

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

     return nullptr;
   case Intrinsic::assume:
   case Intrinsic::var_annotation:
-    // Discard annotate attributes and assumptions
+  case Intrinsic::sideeffect:
+    // Discard annotate attributes, assumptions, and artificial side-effects.
     return nullptr;
 
   case Intrinsic::codeview_annotation: {

lib/Transforms/Scalar/EarlyCSE.cpp

       continue;
     }
 
+    // Skip sideeffect intrinsics, for the same reason as assume intrinsics.
+    if (match(Inst, m_Intrinsic<Intrinsic::sideeffect>())) {
+      DEBUG(dbgs() << "EarlyCSE skipping sideeffect: " << *Inst << '\n');
+      continue;
+    }
+
     // Skip invariant.start intrinsics since they only read memory, and we can
     // forward values across it. Also, we dont need to consume the last store
     // since the semantics of invariant.start allow us to perform DSE of the

lib/Transforms/Scalar/GVNHoist.cpp

         else if (auto *Call = dyn_cast<CallInst>(&I1)) {
           if (auto *Intr = dyn_cast<IntrinsicInst>(Call)) {
             if (isa<DbgInfoIntrinsic>(Intr) ||
-                Intr->getIntrinsicID() == Intrinsic::assume)
+                Intr->getIntrinsicID() == Intrinsic::assume ||
+                Intr->getIntrinsicID() == Intrinsic::sideeffect)
               continue;
           }
           if (Call->mayHaveSideEffects())

lib/Transforms/Utils/Evaluator.cpp

           DEBUG(dbgs() << "Skipping assume intrinsic.\n");
           ++CurInst;
           continue;
+        } else if (II->getIntrinsicID() == Intrinsic::sideeffect) {
+          DEBUG(dbgs() << "Skipping sideeffect intrinsic.\n");
+          ++CurInst;
+          continue;
         }
 
         DEBUG(dbgs() << "Unknown intrinsic. Can not evaluate.\n");

lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Value.h"

         MemoryInstrs.push_back(&I);
       else
         ChainInstrs.push_back(&I);
+    } else if (isa<IntrinsicInst>(&I) &&
+               cast<IntrinsicInst>(&I)->getIntrinsicID() ==
+                   Intrinsic::sideeffect) {
+      // Ignore llvm.sideeffect calls.
     } else if (IsLoadChain && (I.mayWriteToMemory() || I.mayThrow())) {
       DEBUG(dbgs() << "LSV: Found may-write/throw operation: " << I << '\n');
       break;

lib/Transforms/Vectorize/LoopVectorize.cpp

   if (CallInst *CI = dyn_cast<CallInst>(I)) {
     Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
     if (ID && (ID == Intrinsic::assume || ID == Intrinsic::lifetime_end ||
-               ID == Intrinsic::lifetime_start))
+               ID == Intrinsic::lifetime_start || ID == Intrinsic::sideeffect))
       return nullptr;
   }
 

lib/Transforms/Vectorize/SLPVectorizer.cpp

            "new ScheduleData already in scheduling region");
     SD->init(SchedulingRegionID, I);
 
-    if (I->mayReadOrWriteMemory()) {
+    if (I->mayReadOrWriteMemory() &&
+        (!isa<IntrinsicInst>(I) ||
+         cast<IntrinsicInst>(I)->getIntrinsicID() != Intrinsic::sideeffect)) {
       // Update the linked list of memory accessing instructions.
       if (CurrentLoadStore) {
         CurrentLoadStore->NextLoadStore = SD;

test/CodeGen/Generic/intrinsics.ll

         %q = call i8* @llvm.invariant.group.barrier(i8* %p)
         ret i8* %q
 }
+
+; sideeffect
+
+declare void @llvm.sideeffect()
+
+define void @test_sideeffect() {
+    call void @llvm.sideeffect()
+    ret void
+}

test/Transforms/DCE/int_sideeffect.ll

+; RUN: opt -S < %s -instcombine | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Don't DCE llvm.sideeffect calls.
+
+; CHECK-LABEL: dce
+; CHECK: call void @llvm.sideeffect()
+define void @dce() {
+    call void @llvm.sideeffect()
+    ret void
+}

test/Transforms/DeadStoreElimination/int_sideeffect.ll

+; RUN: opt -S < %s -dse | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Dead store elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: dse
+; CHECK: store
+; CHECK-NOT: store
+define void @dse(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    store float 0.0, float* %p
+    ret void
+}

test/Transforms/EarlyCSE/int_sideeffect.ll

+; RUN: opt -S < %s -early-cse | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}

test/Transforms/FunctionAttrs/int_sideeffect.ll

+; RUN: opt -S < %s -functionattrs | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Don't add readnone or similar attributes when an @llvm.sideeffect() intrinsic
+; is present.
+
+; CHECK: define void @test() {
+define void @test() {
+    call void @llvm.sideeffect()
+    ret void
+}
+
+; CHECK: define void @loop() {
+define void @loop() {
+    br label %loop
+
+loop:
+    call void @llvm.sideeffect()
+    br label %loop
+}

test/Transforms/GVN/int_sideeffect.ll

+; RUN: opt -S < %s -gvn | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}
+
+; LICM across a @llvm.sideeffect.
+
+; CHECK-LABEL: licm
+; CHECK: load
+; CHECK: loop:
+; CHECK-NOT: load
+define float @licm(i64 %n, float* nocapture readonly %p) #0 {
+bb0:
+  br label %loop
+
+loop:
+  %i = phi i64 [ 0, %bb0 ], [ %t5, %loop ]
+  %sum = phi float [ 0.000000e+00, %bb0 ], [ %t4, %loop ]
+  call void @llvm.sideeffect()
+  %t3 = load float, float* %p
+  %t4 = fadd float %sum, %t3
+  %t5 = add i64 %i, 1
+  %t6 = icmp ult i64 %t5, %n
+  br i1 %t6, label %loop, label %bb2
+
+bb2:
+  ret float %t4
+}

test/Transforms/GVNHoist/int_sideeffect.ll

+; RUN: opt -S < %s -gvn-hoist | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; GVN hoisting across a @llvm.sideeffect.
+
+; CHECK-LABEL: scalarsHoisting
+; CHECK: = fsub
+; CHECK: br i1 %cmp,
+; CHECK-NOT: fsub
+define float @scalarsHoisting(float %d, float %m, float %a, i1 %cmp) {
+entry:
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:
+  call void @llvm.sideeffect()
+  %sub0 = fsub float %m, %a
+  %mul = fmul float %sub0, %d
+  br label %if.end
+
+if.else:
+  %sub1 = fsub float %m, %a
+  %div = fdiv float %sub1, %d
+  br label %if.end
+
+if.end:
+  %phi = phi float [ %mul, %if.then ], [ %div, %if.else ]
+  ret float %phi
+}
+

test/Transforms/GVNSink/int_sideeffect.ll

+; RUN: opt -S < %s -gvn-sink | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; GVN sinking across a @llvm.sideeffect.
+
+; CHECK-LABEL: scalarsSinking
+; CHECK-NOT: fmul
+; CHECK: = phi
+; CHECK: = fmul
+define float @scalarsSinking(float %d, float %m, float %a, i1 %cmp) {
+entry:
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:
+  call void @llvm.sideeffect()
+  %sub = fsub float %m, %a
+  %mul0 = fmul float %sub, %d
+  br label %if.end
+
+if.else:
+  %add = fadd float %m, %a
+  %mul1 = fmul float %add, %d
+  br label %if.end
+
+if.end:
+  %phi = phi float [ %mul0, %if.then ], [ %mul1, %if.else ]
+  ret float %phi
+}
+

test/Transforms/GlobalOpt/int_sideeffect.ll

+; RUN: opt -S < %s -globalopt | FileCheck %s
+
+; Static evaluation across a @llvm.sideeffect.
+
+; CHECK-NOT: store
+
+declare void @llvm.sideeffect()
+
[e-mail]_ctors = appending global [1 x { i32, void ()* }] [ { i32, void ()* } { i32 65535, void ()* @ctor } ]
+@G = global i32 0
+
+define internal void @ctor() {
+    store i32 1, i32* @G
+    call void @llvm.sideeffect()
+    ret void
+}

test/Transforms/InstCombine/int_sideeffect.ll

+; RUN: opt -S < %s -instcombine | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}

test/Transforms/LICM/int_sideeffect.ll

+; RUN: opt -S < %s -licm | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; LICM across a @llvm.sideeffect.
+
+; CHECK-LABEL: licm
+; CHECK: load
+; CHECK: loop:
+; CHECK-NOT: load
+define float @licm(i64 %n, float* nocapture readonly %p) #0 {
+bb0:
+  br label %loop
+
+loop:
+  %i = phi i64 [ 0, %bb0 ], [ %t5, %loop ]
+  %sum = phi float [ 0.000000e+00, %bb0 ], [ %t4, %loop ]
+  call void @llvm.sideeffect()
+  %t3 = load float, float* %p
+  %t4 = fadd float %sum, %t3
+  %t5 = add i64 %i, 1
+  %t6 = icmp ult i64 %t5, %n
+  br i1 %t6, label %loop, label %bb2
+
+bb2:
+  ret float %t4
+}

test/Transforms/LoadStoreVectorizer/int_sideeffect.ll

+; RUN: opt -S < %s -load-store-vectorizer | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; load-store vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: test
+; CHECK: load <4 x float>
+; CHECK: store <4 x float>
+define void @test(float* %p) {
+    %p0 = getelementptr float, float* %p, i64 0
+    %p1 = getelementptr float, float* %p, i64 1
+    %p2 = getelementptr float, float* %p, i64 2
+    %p3 = getelementptr float, float* %p, i64 3
+    %l0 = load float, float* %p0, align 16
+    %l1 = load float, float* %p1
+    %l2 = load float, float* %p2
+    call void @llvm.sideeffect()
+    %l3 = load float, float* %p3
+    store float %l0, float* %p0, align 16
+    call void @llvm.sideeffect()
+    store float %l1, float* %p1
+    store float %l2, float* %p2
+    store float %l3, float* %p3
+    ret void
+}

test/Transforms/LoopIdiom/int_sideeffect.ll

+; RUN: opt -S < %s -loop-idiom | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Loop idiom recognition across a @llvm.sideeffect.
+
+; CHECK-LABEL: zero
+; CHECK: llvm.memset
+define void @zero(float* %p, i64 %n) nounwind {
+bb7.lr.ph:
+  br label %bb7
+
+bb7:
+  %i.02 = phi i64 [ 0, %bb7.lr.ph ], [ %tmp13, %bb7 ]
+  %tmp10 = getelementptr inbounds float, float* %p, i64 %i.02
+  store float 0.000000e+00, float* %tmp10, align 4
+  %tmp13 = add i64 %i.02, 1
+  %tmp6 = icmp ult i64 %tmp13, %n
+  br i1 %tmp6, label %bb7, label %bb14
+
+bb14:
+  ret void
+}

test/Transforms/LoopVectorize/int_sideeffect.ll

+; RUN: opt -S < %s -loop-vectorize -force-vector-width=4 | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: store_ones
+; CHECK: store <4 x float>
+define void @store_ones(float* %p, i64 %n) nounwind {
+bb7.lr.ph:
+  br label %bb7
+
+bb7:
+  %i.02 = phi i64 [ 0, %bb7.lr.ph ], [ %tmp13, %bb7 ]
+  call void @llvm.sideeffect()
+  %tmp10 = getelementptr inbounds float, float* %p, i64 %i.02
+  store float 1.0, float* %tmp10, align 4
+  %tmp13 = add i64 %i.02, 1
+  %tmp6 = icmp ult i64 %tmp13, %n
+  br i1 %tmp6, label %bb7, label %bb14
+
+bb14:
+  ret void
+}

test/Transforms/NewGVN/int_sideeffect.ll

+; RUN: opt -S < %s -newgvn | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}

test/Transforms/SLPVectorizer/int_sideeffect.ll

+; RUN: opt -S < %s -slp-vectorizer -slp-max-reg-size=128 -slp-min-reg-size=128 | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; SLP vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: test
+; CHECK: store <4 x float>
+define void @test(float* %p) {
+    %p0 = getelementptr float, float* %p, i64 0
+    %p1 = getelementptr float, float* %p, i64 1
+    %p2 = getelementptr float, float* %p, i64 2
+    %p3 = getelementptr float, float* %p, i64 3
+    %l0 = load float, float* %p0
+    %l1 = load float, float* %p1
+    %l2 = load float, float* %p2
+    call void @llvm.sideeffect()
+    %l3 = load float, float* %p3
+    store float %l0, float* %p0
+    call void @llvm.sideeffect()
+    store float %l1, float* %p1
+    store float %l2, float* %p2
+    store float %l3, float* %p3
+    ret void
+}