Add an @llvm.sideeffect intrinsic

[sunfish]

Summary

This patch implements Chandler's idea for supporting languages that require support for infinite loops with side effects, such as Rust, providing a solution to bug 965.

Specifically, it adds an llvm.sideeffect() intrinsic, which has no actual effect, but which appears to optimization passes to have obscure side effects, such that they don't optimize away loops containing it. It also teaches several optimization passes to ignore this intrinsic, so that it doesn't significantly impact optimization in most cases.

[efriedma]

docs/LangRef.rst

14208 ↗

(On Diff #116888)

A language that needs these is going to be inserting a lot of them; assuming a language doesn't have goto, I think the frontend needs to insert one at the beginning of every function and every loop. Is there some rule for coalescing and/or eliminating llvm.sideeffect calls which aren't necessary?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667 ↗

(On Diff #116888)

Why is it safe to drop the llvm.sideeffect call in SelectionDAG?

[sunfish]

docs/LangRef.rst

14208 ↗

(On Diff #116888)

Eliminating requires seeing a volatile access, or similar, nearby, because LLVM doesn't know when calls are guaranteed to perform side effects. Coalescing two @llvm.sideeffect calls would be straightforward. I haven't implemented these yet because if there's already a volatile access or an @llvm.sideeffect call nearby, deleting another @llvm.sideeffect call doesn't usually make much difference (to eg. functionattrs).

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667 ↗

(On Diff #116888)

Just that SelectionDAG and CodeGen after it don't currently have any optimizations that rely on forward-progress guarantees, and probably won't add them. Those kinds of optimizations are the optimizer's job.

If there's any situation where there's an actual loop in the user code that is at all possible to eliminate, it's always better to find a way to have the optimizer do it, because that's an enabling optimization that will make other parts of the optimizer more powerful.

[• dberlin]

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

[hfinkel]

Maybe we should do this in follow-up, but I'd also like to see some general text added to the LangRef about the semantics of loops (that makes reference to this intrinsic). Maybe we can say something like:

Loops that contains no calls that access inaccessible memory, perform no atomic or volatile memory access, nor otherwise synchronizes with other threads, are assumed to eventually terminate. If the source-level language has well-defined infinite loops, the LLVM IR representation of such loops should contain at least one call to `@llvm.sideeffect`.

We'll then need to update Clang to insert this intrinsic into loops (in C) that have a constant controlling condition (per C11 6.8.5p6).

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667 ↗

(On Diff #116888)

I'm not sure the situation is that straightforward. We can also use this information to assume things about the behavior of expressions that determine loop strip counts, and we might want to do that later in the pipeline too.

We don't have anything that does that now, however, so we'll just need to preserve this somehow if need be.

[sunfish]

In D38336#882813, @dberlin wrote:

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Another potential use was mentioned here.

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

[• dberlin]

In D38336#882864, @sunfish wrote:

In D38336#882813, @dberlin wrote:

Is literally the only purpose to handle infinite loops?
If so, please tighten up the description a bit.
(Right now "indicate that the loop shouldn't be optimized away even if it's an infinite
loop with no other side effects." would cover unreachable loops, etc).

Another potential use was mentioned here.

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

(If it was, i'd request we either build an analysis like assume analysis, or circularly link these into a separate instruction list in basic blocks so you can tell in O(1) whether they are there)

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

SGTM

[sunfish]

In D38336#882873, @dberlin wrote:

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

5667 ↗

(On Diff #116888)

It wouldn't be complicated to add an ISD::SIDEEFFECT and TargetOpcode::SIDEEFFECT and carry them all the way through codegen, and to fix the various places that they pessimize, if it turns out to be important.

[• dberlin]

In D38336#882970, @sunfish wrote:

In D38336#882873, @dberlin wrote:

One reason i ask is because ADCE/etc wouldn't even see it. they won't process the block in such a case.
Hence, if the intrinsic relies on always being looked at and then deciding to keep everything containing it, i'd be opposed (since that adds O(N) walks to everything :P).
Right now, it doesn't sound like that's the case.

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.
Those mechanisms keep loops that are useful alive. They are definition of what makes a loop useful in the first place.
You said that llvm.sideeffect, by itself, should not be a reason to keep it alive if say, you can prove it finite or unreachable. It's only there to keep otherwise infinite empty loops alive.
But if you use the same mechanisms, it will always keep the loop alive, no matter what. Even if it's finite, etc.
In particular, those mechanisms are used to mark things that are still not able to be eliminated, for example, despite the loop being provably finite and otherwise empty.

So if llvm.sideeffect is not like those things, and does not mean the loop should stay alive if it's provably finite and otherwise empty (For example), i fail to see how the existence of those mechanisms helps.

[sunfish]

This update revises the documentation to clarify that the intrinsic doesn't inherently prevent loops from being optimized away.

In D38336#884104, @dberlin wrote:

In D38336#882970, @sunfish wrote:

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.

The basic idea of llvm.sideeffect is to just be a thing that gets treated like it has side effects. It's just like a volatile store through a non-escaped memory address.

I may have been unclear when you asked

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

and I said

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

I interpreted your question to be just about loops where we can prove the backedge is not taken. llvm.sideeffect does keep finite loops alive under the same conditions that other operations with side effects would.

In the future, we may want to optimize away redundant llvm.sideeffect calls, and an advanced form of this could allow optimizing away provably-finite loops containing llvm.sideeffect, however, the current patch doesn't depend on this.

[• dberlin]

In D38336#884469, @sunfish wrote:

This update revises the documentation to clarify that the intrinsic doesn't inherently prevent loops from being optimized away.

In D38336#884104, @dberlin wrote:

In D38336#882970, @sunfish wrote:

If it supports C++, ADCE needs to preserve calls to a library I/O functions, volatile operations, thread synchronizations, and atomic operations, and calls to functions which may perform any of these. Whatever it does to handle those should handle llvm.sideeffect as well.

This seems pretty incongruous to me with your answers to my earlier questions, so i feel like i'm missing something.

The basic idea of llvm.sideeffect is to just be a thing that gets treated like it has side effects. It's just like a volatile store through a non-escaped memory address.

I may have been unclear when you asked

Do you expect it to keep provably finite loops that have it alive.
IE a loop where we can prove the backedge is not taken?

and I said

No; a backedge can still be folded away in the normal ways. The llvm.sideeffect call would remain, no longer needed, but not invalid. If it were important, we could add code to remove it in that case, but the patch here doesn't have that.

I interpreted your question to be just about loops where we can prove the backedge is not taken. llvm.sideeffect does keep finite loops alive under the same conditions that other operations with side effects would.

In the future, we may want to optimize away redundant llvm.sideeffect calls, and an advanced form of this could allow optimizing away provably-finite loops containing llvm.sideeffect, however, the current patch doesn't depend on this.

Okay, that helps, thanks.

[hfinkel]

LGTM

llvm/trunk/docs/LangRef.rst

 ``i32 trunc(x - %ptr)``, the intrinsic call is folded to ``x``.
 
 LLVM provides that the calculation of such a constant initializer will
 not overflow at link time under the medium code model if ``x`` is an
 ``unnamed_addr`` function. However, it does not provide this guarantee for
 a constant initializer folded into a function body. This intrinsic can be
 used to avoid the possibility of overflows when loading from such a constant.
 
+'``llvm.sideeffect``' Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      declare void @llvm.sideeffect() inaccessiblememonly nounwind
+
+Overview:
+"""""""""
+
+The ``llvm.sideeffect`` intrinsic doesn't perform any operation. Optimizers
+treat it as having side effects, so it can be inserted into a loop to
+indicate that the loop shouldn't be assumed to terminate (which could
+potentially lead to the loop being optimized away entirely), even if it's
+an infinite loop with no other side effects.
+
+Arguments:
+""""""""""
+
+None.
+
+Semantics:
+""""""""""
+
+This intrinsic actually does nothing, but optimizers must assume that it
+has externally observable side effects.
+
 Stack Map Intrinsics
 --------------------
 
 LLVM provides experimental intrinsics to support runtime patching
 mechanisms commonly desired in dynamic language JITs. These intrinsics
 are described in :doc:`StackMaps`.
 
 Element Wise Atomic Memory Intrinsics

llvm/trunk/include/llvm/Analysis/TargetTransformInfoImpl.h

unsigned getIntrinsicCost(Intrinsic::ID IID, Type *RetTy,
     default:
       // Intrinsics rarely (if ever) have normal argument setup constraints.
       // Model them as having a basic instruction cost.
       // FIXME: This is wrong for libc intrinsics.
       return TTI::TCC_Basic;
 
     case Intrinsic::annotation:
     case Intrinsic::assume:
+    case Intrinsic::sideeffect:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_value:
     case Intrinsic::invariant_start:
     case Intrinsic::invariant_end:
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
     case Intrinsic::objectsize:
     case Intrinsic::ptr_annotation:

llvm/trunk/include/llvm/CodeGen/BasicTTIImpl.h

case Intrinsic::fma:
       ISDs.push_back(ISD::FMA);
       break;
     case Intrinsic::fmuladd:
       ISDs.push_back(ISD::FMA);
       break;
     // FIXME: We should return 0 whenever getIntrinsicCost == TCC_Free.
     case Intrinsic::lifetime_start:
     case Intrinsic::lifetime_end:
+    case Intrinsic::sideeffect:
       return 0;
     case Intrinsic::masked_store:
       return static_cast<T *>(this)
           ->getMaskedMemoryOpCost(Instruction::Store, Tys[0], 0, 0);
     case Intrinsic::masked_load:
       return static_cast<T *>(this)
           ->getMaskedMemoryOpCost(Instruction::Load, RetTy, 0, 0);
     case Intrinsic::ctpop:

llvm/trunk/include/llvm/IR/Intrinsics.td

 
 // Support for speculative runtime guards
 def int_experimental_guard : Intrinsic<[], [llvm_i1_ty, llvm_vararg_ty],
                                        [Throws]>;
 
 // NOP: calls/invokes to this intrinsic are removed by codegen
 def int_donothing : Intrinsic<[], [], [IntrNoMem]>;
 
+// This instruction has no actual effect, though it is treated by the optimizer
+// has having opaque side effects. This may be inserted into loops to ensure
+// that they are not removed even if they turn out to be empty, for languages
+// which specify that infinite loops must be preserved.
+def int_sideeffect : Intrinsic<[], [], [IntrInaccessibleMemOnly]>;
+
 // Intrisics to support half precision floating point format
 let IntrProperties = [IntrNoMem] in {
 def int_convert_to_fp16   : Intrinsic<[llvm_i16_ty], [llvm_anyfloat_ty]>;
 def int_convert_from_fp16 : Intrinsic<[llvm_anyfloat_ty], [llvm_i16_ty]>;
 }
 
 // Clear cache intrinsic, default to ignore (ie. emit nothing)
 // maps to void __clear_cache() on supporting platforms

llvm/trunk/lib/Analysis/AliasSetTracker.cpp

void AliasSetTracker::addUnknown(Instruction *Inst) {
   if (auto *II = dyn_cast<IntrinsicInst>(Inst)) {
     // These intrinsics will show up as affecting memory, but they are just
     // markers.
     switch (II->getIntrinsicID()) {
     default:
       break;
       // FIXME: Add lifetime/invariant intrinsics (See: PR30807).
     case Intrinsic::assume:
+    case Intrinsic::sideeffect:
       return;
     }
   }
   if (!Inst->mayReadOrWriteMemory())
     return; // doesn't alias anything
 
   AliasSet *AS = findAliasSetForUnknownInst(Inst);
   if (AS) {

llvm/trunk/lib/Analysis/ValueTracking.cpp

 // Is this an intrinsic that cannot be speculated but also cannot trap?
 static bool isAssumeLikeIntrinsic(const Instruction *I) {
   if (const CallInst *CI = dyn_cast<CallInst>(I))
     if (Function *F = CI->getCalledFunction())
       switch (F->getIntrinsicID()) {
       default: break;
       // FIXME: This list is repeated from NoTTI::getIntrinsicCost.
       case Intrinsic::assume:
+      case Intrinsic::sideeffect:
       case Intrinsic::dbg_declare:
       case Intrinsic::dbg_value:
       case Intrinsic::invariant_start:
       case Intrinsic::invariant_end:
       case Intrinsic::lifetime_start:
       case Intrinsic::lifetime_end:
       case Intrinsic::objectsize:
       case Intrinsic::ptr_annotation:
if (auto CS = ImmutableCallSite(I)) {
     //    the program.
     //
     // We rely on those assumptions here, and use the memory effects of the call
     // target as a proxy for checking that it always returns.
 
     // FIXME: This isn't aggressive enough; a call which only writes to a global
     // is guaranteed to return.
     return CS.onlyReadsMemory() || CS.onlyAccessesArgMemory() ||
-           match(I, m_Intrinsic<Intrinsic::assume>());
+           match(I, m_Intrinsic<Intrinsic::assume>()) ||
+           match(I, m_Intrinsic<Intrinsic::sideeffect>());
   }
 
   // Other instructions return normally.
   return true;
 }
 
 bool llvm::isGuaranteedToExecuteForEveryIteration(const Instruction *I,
                                                   const Loop *L) {

llvm/trunk/lib/Analysis/VectorUtils.cpp

 /// its ID, in case it does not found it return not_intrinsic.
 Intrinsic::ID llvm::getVectorIntrinsicIDForCall(const CallInst *CI,
                                                 const TargetLibraryInfo *TLI) {
   Intrinsic::ID ID = getIntrinsicForCallSite(CI, TLI);
   if (ID == Intrinsic::not_intrinsic)
     return Intrinsic::not_intrinsic;
 
   if (isTriviallyVectorizable(ID) || ID == Intrinsic::lifetime_start ||
-      ID == Intrinsic::lifetime_end || ID == Intrinsic::assume)
+      ID == Intrinsic::lifetime_end || ID == Intrinsic::assume ||
+      ID == Intrinsic::sideeffect)
     return ID;
   return Intrinsic::not_intrinsic;
 }
 
 /// \brief Find the operand of the GEP that should be checked for consecutive
 /// stores. This ignores trailing indices that have no effect on the final
 /// pointer.
 unsigned llvm::getGEPInductionOperand(const GetElementPtrInst *Gep) {

llvm/trunk/lib/CodeGen/SelectionDAG/FastISel.cpp

bool FastISel::selectIntrinsicCall(const IntrinsicInst *II) {
   switch (II->getIntrinsicID()) {
   default:
     break;
   // At -O0 we don't care about the lifetime intrinsics.
   case Intrinsic::lifetime_start:
   case Intrinsic::lifetime_end:
   // The donothing intrinsic does, well, nothing.
   case Intrinsic::donothing:
+  // Neither does the sideeffect intrinsic.
+  case Intrinsic::sideeffect:
   // Neither does the assume intrinsic; it's also OK not to codegen its operand.
   case Intrinsic::assume:
     return true;
   case Intrinsic::dbg_declare: {
     const DbgDeclareInst *DI = cast<DbgDeclareInst>(II);
     assert(DI->getVariable() && "Missing variable");
     if (!FuncInfo.MF->getMMI().hasDebugInfo()) {
       DEBUG(dbgs() << "Dropping debug info for " << *DI << "\n");

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

SelectionDAGBuilder::visitIntrinsicCall(const CallInst &I, unsigned Intrinsic) {
   case Intrinsic::annotation:
   case Intrinsic::ptr_annotation:
   case Intrinsic::invariant_group_barrier:
     // Drop the intrinsic, but forward the value
     setValue(&I, getValue(I.getOperand(0)));
     return nullptr;
   case Intrinsic::assume:
   case Intrinsic::var_annotation:
-    // Discard annotate attributes and assumptions
+  case Intrinsic::sideeffect:
+    // Discard annotate attributes, assumptions, and artificial side-effects.
     return nullptr;
 
   case Intrinsic::codeview_annotation: {
     // Emit a label associated with this metadata.
     MachineFunction &MF = DAG.getMachineFunction();
     MCSymbol *Label =
         MF.getMMI().getContext().createTempSymbol("annotation", true);
     Metadata *MD = cast<MetadataAsValue>(I.getArgOperand(0))->getMetadata();

llvm/trunk/lib/Transforms/Scalar/EarlyCSE.cpp

if (match(Inst, m_Intrinsic<Intrinsic::assume>())) {
       if (CondI && SimpleValue::canHandle(CondI)) {
         DEBUG(dbgs() << "EarlyCSE considering assumption: " << *Inst << '\n');
         AvailableValues.insert(CondI, ConstantInt::getTrue(BB->getContext()));
       } else
         DEBUG(dbgs() << "EarlyCSE skipping assumption: " << *Inst << '\n');
       continue;
     }
 
+    // Skip sideeffect intrinsics, for the same reason as assume intrinsics.
+    if (match(Inst, m_Intrinsic<Intrinsic::sideeffect>())) {
+      DEBUG(dbgs() << "EarlyCSE skipping sideeffect: " << *Inst << '\n');
+      continue;
+    }
+
     // Skip invariant.start intrinsics since they only read memory, and we can
     // forward values across it. Also, we dont need to consume the last store
     // since the semantics of invariant.start allow us to perform DSE of the
     // last store, if there was a store following invariant.start. Consider:
     //
     // store 30, i8* p
     // invariant.start(p)
     // store 40, i8* p

llvm/trunk/lib/Transforms/Scalar/GVNHoist.cpp

for (BasicBlock *BB : depth_first(&F.getEntryBlock())) {
 
         if (auto *Load = dyn_cast<LoadInst>(&I1))
           LI.insert(Load, VN);
         else if (auto *Store = dyn_cast<StoreInst>(&I1))
           SI.insert(Store, VN);
         else if (auto *Call = dyn_cast<CallInst>(&I1)) {
           if (auto *Intr = dyn_cast<IntrinsicInst>(Call)) {
             if (isa<DbgInfoIntrinsic>(Intr) ||
-                Intr->getIntrinsicID() == Intrinsic::assume)
+                Intr->getIntrinsicID() == Intrinsic::assume ||
+                Intr->getIntrinsicID() == Intrinsic::sideeffect)
               continue;
           }
           if (Call->mayHaveSideEffects())
             break;
 
           if (Call->isConvergent())
             break;
 

llvm/trunk/lib/Transforms/Utils/Evaluator.cpp

if (StoreInst *SI = dyn_cast<StoreInst>(CurInst)) {
           }
           // Continue even if we do nothing.
           ++CurInst;
           continue;
         } else if (II->getIntrinsicID() == Intrinsic::assume) {
           DEBUG(dbgs() << "Skipping assume intrinsic.\n");
           ++CurInst;
           continue;
+        } else if (II->getIntrinsicID() == Intrinsic::sideeffect) {
+          DEBUG(dbgs() << "Skipping sideeffect intrinsic.\n");
+          ++CurInst;
+          continue;
         }
 
         DEBUG(dbgs() << "Unknown intrinsic. Can not evaluate.\n");
         return false;
       }
 
       // Resolve function pointers.
       Function *Callee = dyn_cast<Function>(getVal(CS.getCalledValue()));

llvm/trunk/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/KnownBits.h"
Vectorizer::getVectorizablePrefix(ArrayRef<Instruction *> Chain) {
   });
 
   for (Instruction &I : make_range(getBoundaryInstrs(Chain))) {
     if (isa<LoadInst>(I) || isa<StoreInst>(I)) {
       if (!is_contained(Chain, &I))
         MemoryInstrs.push_back(&I);
       else
         ChainInstrs.push_back(&I);
+    } else if (isa<IntrinsicInst>(&I) &&
+               cast<IntrinsicInst>(&I)->getIntrinsicID() ==
+                   Intrinsic::sideeffect) {
+      // Ignore llvm.sideeffect calls.
     } else if (IsLoadChain && (I.mayWriteToMemory() || I.mayThrow())) {
       DEBUG(dbgs() << "LSV: Found may-write/throw operation: " << I << '\n');
       break;
     } else if (!IsLoadChain && (I.mayReadOrWriteMemory() || I.mayThrow())) {
       DEBUG(dbgs() << "LSV: Found may-read/write/throw operation: " << I
                    << '\n');
       break;
     }

llvm/trunk/lib/Transforms/Vectorize/LoopVectorize.cpp

bool LoopVectorizationPlanner::tryToWiden(Instruction *I, VPBasicBlock *VPBB,
   };
 
   if (!IsVectorizableOpcode(I->getOpcode()))
     return false;
 
   if (CallInst *CI = dyn_cast<CallInst>(I)) {
     Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
     if (ID && (ID == Intrinsic::assume || ID == Intrinsic::lifetime_end ||
-               ID == Intrinsic::lifetime_start))
+               ID == Intrinsic::lifetime_start || ID == Intrinsic::sideeffect))
       return false;
   }
 
   auto willWiden = [&](unsigned VF) -> bool {
     if (!isa<PHINode>(I) && (CM.isScalarAfterVectorization(I, VF) ||
                              CM.isProfitableToScalarize(I, VF)))
       return false;
     if (CallInst *CI = dyn_cast<CallInst>(I)) {

llvm/trunk/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (!SD) {
       SD = allocateScheduleDataChunks();
       ScheduleDataMap[I] = SD;
       SD->Inst = I;
     }
     assert(!isInSchedulingRegion(SD) &&
            "new ScheduleData already in scheduling region");
     SD->init(SchedulingRegionID, I);
 
-    if (I->mayReadOrWriteMemory()) {
+    if (I->mayReadOrWriteMemory() &&
+        (!isa<IntrinsicInst>(I) ||
+         cast<IntrinsicInst>(I)->getIntrinsicID() != Intrinsic::sideeffect)) {
       // Update the linked list of memory accessing instructions.
       if (CurrentLoadStore) {
         CurrentLoadStore->NextLoadStore = SD;
       } else {
         FirstLoadStoreInRegion = SD;
       }
       CurrentLoadStore = SD;
     }

llvm/trunk/test/CodeGen/Generic/intrinsics.ll

 }
 
 declare i8* @llvm.invariant.group.barrier(i8*)
 
 define i8* @barrier(i8* %p) {
         %q = call i8* @llvm.invariant.group.barrier(i8* %p)
         ret i8* %q
 }
+
+; sideeffect
+
+declare void @llvm.sideeffect()
+
+define void @test_sideeffect() {
+    call void @llvm.sideeffect()
+    ret void
+}

llvm/trunk/test/Transforms/DCE/int_sideeffect.ll

+; RUN: opt -S < %s -instcombine | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Don't DCE llvm.sideeffect calls.
+
+; CHECK-LABEL: dce
+; CHECK: call void @llvm.sideeffect()
+define void @dce() {
+    call void @llvm.sideeffect()
+    ret void
+}

llvm/trunk/test/Transforms/DeadStoreElimination/int_sideeffect.ll

+; RUN: opt -S < %s -dse | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Dead store elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: dse
+; CHECK: store
+; CHECK-NOT: store
+define void @dse(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    store float 0.0, float* %p
+    ret void
+}

llvm/trunk/test/Transforms/EarlyCSE/int_sideeffect.ll

+; RUN: opt -S < %s -early-cse | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}

llvm/trunk/test/Transforms/FunctionAttrs/int_sideeffect.ll

+; RUN: opt -S < %s -functionattrs | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Don't add readnone or similar attributes when an @llvm.sideeffect() intrinsic
+; is present.
+
+; CHECK: define void @test() {
+define void @test() {
+    call void @llvm.sideeffect()
+    ret void
+}
+
+; CHECK: define void @loop() {
+define void @loop() {
+    br label %loop
+
+loop:
+    call void @llvm.sideeffect()
+    br label %loop
+}

llvm/trunk/test/Transforms/GVN/int_sideeffect.ll

+; RUN: opt -S < %s -gvn | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}
+
+; LICM across a @llvm.sideeffect.
+
+; CHECK-LABEL: licm
+; CHECK: load
+; CHECK: loop:
+; CHECK-NOT: load
+define float @licm(i64 %n, float* nocapture readonly %p) #0 {
+bb0:
+  br label %loop
+
+loop:
+  %i = phi i64 [ 0, %bb0 ], [ %t5, %loop ]
+  %sum = phi float [ 0.000000e+00, %bb0 ], [ %t4, %loop ]
+  call void @llvm.sideeffect()
+  %t3 = load float, float* %p
+  %t4 = fadd float %sum, %t3
+  %t5 = add i64 %i, 1
+  %t6 = icmp ult i64 %t5, %n
+  br i1 %t6, label %loop, label %bb2
+
+bb2:
+  ret float %t4
+}

llvm/trunk/test/Transforms/GVNHoist/int_sideeffect.ll

+; RUN: opt -S < %s -gvn-hoist | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; GVN hoisting across a @llvm.sideeffect.
+
+; CHECK-LABEL: scalarsHoisting
+; CHECK: = fsub
+; CHECK: br i1 %cmp,
+; CHECK-NOT: fsub
+define float @scalarsHoisting(float %d, float %m, float %a, i1 %cmp) {
+entry:
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:
+  call void @llvm.sideeffect()
+  %sub0 = fsub float %m, %a
+  %mul = fmul float %sub0, %d
+  br label %if.end
+
+if.else:
+  %sub1 = fsub float %m, %a
+  %div = fdiv float %sub1, %d
+  br label %if.end
+
+if.end:
+  %phi = phi float [ %mul, %if.then ], [ %div, %if.else ]
+  ret float %phi
+}
+

llvm/trunk/test/Transforms/GVNSink/int_sideeffect.ll

+; RUN: opt -S < %s -gvn-sink | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; GVN sinking across a @llvm.sideeffect.
+
+; CHECK-LABEL: scalarsSinking
+; CHECK-NOT: fmul
+; CHECK: = phi
+; CHECK: = fmul
+define float @scalarsSinking(float %d, float %m, float %a, i1 %cmp) {
+entry:
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:
+  call void @llvm.sideeffect()
+  %sub = fsub float %m, %a
+  %mul0 = fmul float %sub, %d
+  br label %if.end
+
+if.else:
+  %add = fadd float %m, %a
+  %mul1 = fmul float %add, %d
+  br label %if.end
+
+if.end:
+  %phi = phi float [ %mul0, %if.then ], [ %mul1, %if.else ]
+  ret float %phi
+}
+

llvm/trunk/test/Transforms/GlobalOpt/int_sideeffect.ll

+; RUN: opt -S < %s -globalopt | FileCheck %s
+
+; Static evaluation across a @llvm.sideeffect.
+
+; CHECK-NOT: store
+
+declare void @llvm.sideeffect()
+
[e-mail]_ctors = appending global [1 x { i32, void ()* }] [ { i32, void ()* } { i32 65535, void ()* @ctor } ]
+@G = global i32 0
+
+define internal void @ctor() {
+    store i32 1, i32* @G
+    call void @llvm.sideeffect()
+    ret void
+}

llvm/trunk/test/Transforms/InstCombine/int_sideeffect.ll

+; RUN: opt -S < %s -instcombine | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}

llvm/trunk/test/Transforms/LICM/int_sideeffect.ll

+; RUN: opt -S < %s -licm | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; LICM across a @llvm.sideeffect.
+
+; CHECK-LABEL: licm
+; CHECK: load
+; CHECK: loop:
+; CHECK-NOT: load
+define float @licm(i64 %n, float* nocapture readonly %p) #0 {
+bb0:
+  br label %loop
+
+loop:
+  %i = phi i64 [ 0, %bb0 ], [ %t5, %loop ]
+  %sum = phi float [ 0.000000e+00, %bb0 ], [ %t4, %loop ]
+  call void @llvm.sideeffect()
+  %t3 = load float, float* %p
+  %t4 = fadd float %sum, %t3
+  %t5 = add i64 %i, 1
+  %t6 = icmp ult i64 %t5, %n
+  br i1 %t6, label %loop, label %bb2
+
+bb2:
+  ret float %t4
+}

llvm/trunk/test/Transforms/LoadStoreVectorizer/int_sideeffect.ll

+; RUN: opt -S < %s -load-store-vectorizer | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; load-store vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: test
+; CHECK: load <4 x float>
+; CHECK: store <4 x float>
+define void @test(float* %p) {
+    %p0 = getelementptr float, float* %p, i64 0
+    %p1 = getelementptr float, float* %p, i64 1
+    %p2 = getelementptr float, float* %p, i64 2
+    %p3 = getelementptr float, float* %p, i64 3
+    %l0 = load float, float* %p0, align 16
+    %l1 = load float, float* %p1
+    %l2 = load float, float* %p2
+    call void @llvm.sideeffect()
+    %l3 = load float, float* %p3
+    store float %l0, float* %p0, align 16
+    call void @llvm.sideeffect()
+    store float %l1, float* %p1
+    store float %l2, float* %p2
+    store float %l3, float* %p3
+    ret void
+}

llvm/trunk/test/Transforms/LoopIdiom/int_sideeffect.ll

+; RUN: opt -S < %s -loop-idiom | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Loop idiom recognition across a @llvm.sideeffect.
+
+; CHECK-LABEL: zero
+; CHECK: llvm.memset
+define void @zero(float* %p, i64 %n) nounwind {
+bb7.lr.ph:
+  br label %bb7
+
+bb7:
+  %i.02 = phi i64 [ 0, %bb7.lr.ph ], [ %tmp13, %bb7 ]
+  %tmp10 = getelementptr inbounds float, float* %p, i64 %i.02
+  store float 0.000000e+00, float* %tmp10, align 4
+  %tmp13 = add i64 %i.02, 1
+  %tmp6 = icmp ult i64 %tmp13, %n
+  br i1 %tmp6, label %bb7, label %bb14
+
+bb14:
+  ret void
+}

llvm/trunk/test/Transforms/LoopVectorize/int_sideeffect.ll

+; RUN: opt -S < %s -loop-vectorize -force-vector-width=4 | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: store_ones
+; CHECK: store <4 x float>
+define void @store_ones(float* %p, i64 %n) nounwind {
+bb7.lr.ph:
+  br label %bb7
+
+bb7:
+  %i.02 = phi i64 [ 0, %bb7.lr.ph ], [ %tmp13, %bb7 ]
+  call void @llvm.sideeffect()
+  %tmp10 = getelementptr inbounds float, float* %p, i64 %i.02
+  store float 1.0, float* %tmp10, align 4
+  %tmp13 = add i64 %i.02, 1
+  %tmp6 = icmp ult i64 %tmp13, %n
+  br i1 %tmp6, label %bb7, label %bb14
+
+bb14:
+  ret void
+}

llvm/trunk/test/Transforms/NewGVN/int_sideeffect.ll

+; RUN: opt -S < %s -newgvn | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; Store-to-load forwarding across a @llvm.sideeffect.
+
+; CHECK-LABEL: s2l
+; CHECK-NOT: load
+define float @s2l(float* %p) {
+    store float 0.0, float* %p
+    call void @llvm.sideeffect()
+    %t = load float, float* %p
+    ret float %t
+}
+
+; Redundant load elimination across a @llvm.sideeffect.
+
+; CHECK-LABEL: rle
+; CHECK: load
+; CHECK-NOT: load
+define float @rle(float* %p) {
+    %r = load float, float* %p
+    call void @llvm.sideeffect()
+    %s = load float, float* %p
+    %t = fadd float %r, %s
+    ret float %t
+}

llvm/trunk/test/Transforms/SLPVectorizer/int_sideeffect.ll

+; RUN: opt -S < %s -slp-vectorizer -slp-max-reg-size=128 -slp-min-reg-size=128 | FileCheck %s
+
+declare void @llvm.sideeffect()
+
+; SLP vectorization across a @llvm.sideeffect.
+
+; CHECK-LABEL: test
+; CHECK: store <4 x float>
+define void @test(float* %p) {
+    %p0 = getelementptr float, float* %p, i64 0
+    %p1 = getelementptr float, float* %p, i64 1
+    %p2 = getelementptr float, float* %p, i64 2
+    %p3 = getelementptr float, float* %p, i64 3
+    %l0 = load float, float* %p0
+    %l1 = load float, float* %p1
+    %l2 = load float, float* %p2
+    call void @llvm.sideeffect()
+    %l3 = load float, float* %p3
+    store float %l0, float* %p0
+    call void @llvm.sideeffect()
+    store float %l1, float* %p1
+    store float %l2, float* %p2
+    store float %l3, float* %p3
+    ret void
+}