Correct dwarf unwind information in function epilogue

[violetav]

Summary

This patch aims to provide correct dwarf unwind information in function epilogue for X86.
It consists of two parts. The first part inserts CFI instructions that set appropriate cfa offset and cfa register in emitEpilogue() in X86FrameLowering. This part is X86 specific.

The second part is platform independent and ensures that:

• CFI instructions do not affect code generation
• Unwind information remains correct when a function is modified by different passes. This is done in a late pass by analyzing information about cfa offset and cfa register in BBs and inserting additional CFI directives where necessary.

Changed CFI instructions so that they:

• are duplicable
• are not counted as instructions when tail duplicating or tail merging
• can be compared as equal

Added CFIInstrInserter pass:

• analyzes each basic block to determine cfa offset and register valid at its entry and exit
• verifies that outgoing cfa offset and register of predecessor blocks match incoming values of their successors
• inserts additional CFI directives at basic block beginning to correct the rule for calculating CFA

Having CFI instructions in function epilogue can cause incorrect CFA calculation rule for some basic blocks. This can happen if, due to basic block reordering, or the existence of multiple epilogue blocks, some of the blocks have wrong cfa offset and register values set by the epilogue block above them.
CFIInstrInserter is currently run only on X86, but can be used by any target that implements support for adding CFI instructions in epilogue.

[violetav]

This patch has been reviewed in D18046 and commited. It was reverted because it caused a number of sanitizer test failures on PPC64 buildbot, and also, it reported incorrect cfa offset values in recursive LLVM builds. These issues have been resolved in this patch.

The following was changed:

Changed CFIInfoVerifier to report errors if it finds incorrect offset/register (not just to print them out).
Added support for unhandled cases that were the cause of incorrect offset values in recursive builds.
There were two issues.
The first one is the issue of a MBB having an adjust_cfa_offset and being its own successor. Because of this, the mechanism for updating CFI info was changed. Added AdjustIncomingCFAOffset and AdjustOutgoingCFAOffset information to a basic block. These values present the amount of adjustment (originating from adjust_cfa_offset directives) in effect at basic block entry and exit. These values get updated during different passes. IncomingCFAOffset and OutgoingCFAOffset now contain only values set by CFI instructions that set absolute offset (def_cfa, def_cfa_offset). In CFIInfoVerifier adjust in/out cfa offset values are added to in/out cfa offsets before checking this information for correctness.
The second issue was the case of creating a new MBB in SplitCriticalEdge() and not updating its CFI info.
Added maintainsCFIInfo() in TargetFrameLowering so that CFIInfoVerifier and CFIInstrInserter run only for targets that maintain CFI information (only X86 for now).

[thegameg]

While trying to make this work with AArch64 and add the support for .cfi_offset / .cfi_restore directives, I came across a few questions and comments:

• Do you plan on adding MIR support? I guess you would have to (de)serialize the state that was added in MachineBasicBlock.h.
• This pass is required to have correct unwind information, but it's disabled on Darwin, because of compact unwinding. Compact unwinding has a mode where it can fallback on DWARF CFI (X86, AArch64), and it is used for cases where the information can't be encoded in that format. I guess this can take a similar path.

lib/CodeGen/MachineBasicBlock.cpp

1489 ↗

(On Diff #108098)

SmallPtrSet<MachineBasicBlock *> ?

1546 ↗

(On Diff #108098)

SmallPtrSet<MachineBasicBlock *> ?

lib/Target/X86/X86CallFrameOptimization.cpp

241 ↗

(On Diff #108098)

Looks like this is already called in PEI::insertPrologEpilogCode, which sounds like a better idea than having it in target specific passes.

[violetav]

Do you plan on adding MIR support?

Not at the moment.

This pass is required to have correct unwind information, but it's disabled on Darwin, because of compact unwinding. Compact unwinding has a mode where it can fallback on DWARF CFI (X86, AArch64), and it is used for cases where the information can't be encoded in that format. I guess this can take a similar path.

When CFI instructions are inserted in epilogue, wrong compact unwind encoding is generated. I have previously posted a question to llvm-dev about this http://lists.llvm.org/pipermail/llvm-dev/2016-March/097037.html and received an answer that "Darwin compact unwinder is never going to learn anything to its benefit from a CFI instruction in the epilogue". I have tried the suggested approach (marking CFI instructions with FrameSetup/FrameDestroy, and then disregarding the ones from epilogue, marked with FrameDestroy). However, there was a problem when you wanted to generate compact unwind section from an existing .s file; there is no info about FrameSetup/FrameDestroy flags, so again you end up generating the wrong encoding.
In the end I disabled this support for Darwin.

I am not sure what you are suggesting here. Doesn't that mean that compact unwind would practically be disabled (would always fallback to DWARF)?

lib/Target/X86/X86CallFrameOptimization.cpp

241 ↗

(On Diff #108098)

You are right. This was needed for previous version of the patch, and I haven't removed it. This pass runs before PEI and needed to have initial values set, because cfa offset and cfa adjustment were both contained in cfa offset field (in the previous patch). Now that they are apart, and because this pass adds only adjust_cfa_offset directives, this initialization is no longer needed here; absolute cfa offsets can be initialized in PEI.

[thegameg]

I am not sure what you are suggesting here. Doesn't that mean that compact unwind would practically be disabled (would always fallback to DWARF)?

The difference here is that your pass is not even running if the target is Darwin. Compact unwinding is generated from CFI. The CFI generated without your pass can be incorrect, so the compact unwinding will be too.

What I'm suggesting is that when we have extra CFI directives that can't be encoded in the compact unwinding format, we use the fallback on DWARF (as we already do in other cases). That would generate correct unwinding information (since your pass fixed it), but won't use compact unwinding.

Let me know if there's something I missed here.

[violetav]

Removed unnecessary call to initializeCFIInfo() in X86CallFrameOptimization.cpp.
Using SmallPtrSet instead of std::set in MachineBasicBlock.cpp.

[iteratee]

Can you provide a description of what you had to change relative to the rollback, and how you're verifying that the issue that caused the rollback has been fixed?

[petarj]

In D35844#830951, @iteratee wrote:

Can you provide a description of what you had to change relative to the rollback

Violeta has provided that information in the first comment.

In D35844#830951, @iteratee wrote:

and how you're verifying that the issue that caused the rollback has been fixed?

She reproduced all the issues, and verified that none of these occur with this version of the patch.

[violetav]

Hi Kyle,
Here are a few more details, on top of what Petar said:

So, there were a few issues, the first one being that some sanitizer tests on PPC64 were failing due to this change (in stage 2). I saw that the CFIInfoVerifier reported incorrect offsets during stage2 build for PPC, and realised what was causing this. Even if the values for cfa offset and register are not initialized for platforms other that X86, cfa offset value can get set in recalculateCFIInfo(), so it doesn't remain uninitialized. I confirmed that CFIInstrInserter and CFIInfoVerifier were causing the failures on PPC, and not the changes made to CFI instructions in general. So, I added a target hook so that these two passes run only for platforms that maintain CFI info. This resolved the issue for PPC.
The second was the group of issues that caused incorrect offset reports with recursive LLVM builds (which I unfortunately didn't test before). I started debugging the example that Eli reported: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20170626/465290.html. This example had two issues, which I have described in the first comment. Resolving these two issues resulted in successfully building the CGStmtOpenMP.cpp.o example. I have also changed the CFIInfoVerifier to emit errors as Eli suggested, and not just to print them out. After that, I have built clang+llvm+compiler-rt+lld, ran make check-all, did a recursive build, ran make check with that, ran the llvm-test suite, and got no failures.

[iteratee]

include/llvm/CodeGen/MachineBasicBlock.h

781 ↗

(On Diff #109065)

I don't follow the comment about why these are necessary again.
Can you elaborate?

lib/CodeGen/TargetPassConfig.cpp

875

I thought part of the problem was that these passes should not be run generally.

[MatzeB]

In order to avoid the discussion being split between two places I added some comments to D18046

[violetav]

include/llvm/CodeGen/MachineBasicBlock.h

781 ↗

(On Diff #109065)

I will try. So, let's say there is a function that looks like this:

  BB_0
    | 
  BB_1 <-----|
  /  \       |
BB_2 BB_3    |
       |     |
     BB_4    |
       |     |
     BB_5 ___|

and that the following is true:

1. BB_1 has following cfi directives:

BB_1:

...
.cfi_adjust_cfa_offset 4
...
.cfi_adjust_cfa_offset -4
...

2. BB_4 has the same:

BB_4:

...
.cfi_adjust_cfa_offset 4
...
.cfi_adjust_cfa_offset -4
...

3. BB_1 is a successor of BB_5 (if it's not clear from the 'illustration', BB_0 has one successor, BB_1 has two, and BB_3,BB_4,BB_5 each have one).

4. For this example, first the cfi_adjust_cfa directives with positive values are inserted (in both BB_1 and BB_4) and then the ones with negative values.

What happens here with the previous approach is that, although BB_1 and BB_4 do not change cfa offset (their incoming and outgoing offsets are the same), they and their successors end up having the wrong offset set.

This was the way that in/out offset was set to successors in the previous approach (from updateCFIInfoSucc()):

// the difference remains the same, calculate it by using previous in/out offset values
int AdjustAmount = Succ->getOutgoingCFAOffset() - Succ->getIncomingCFAOffset();
// set outgoing offset to be sum of the outgoing offset of the predecessor and the amount that this successor adjusts the offset itself (with adjust_cfa_directives)
Succ->setOutgoingCFAOffset(CurrSucc->getOutgoingCFAOffset() + AdjustAmount);
// set incoming offset to be the outgoing offset of the predecessor
Succ->setIncomingCFAOffset(CurrSucc->getOutgoingCFAOffset());

Let's say that all blocks have in/out offset set to zero at the beginning (just to simplify the numbers).
When first 'adjust_cfa_offset 4' is inserted in BB_1, after updating the successors, the in/out cfa offsets of BBs in the function look like this:

BB_0, in 0, out 0
BB_1, in 0, out 4
BB_2, in 4, out 4
BB_3, in 4, out 4
BB_4, in 4, out 4
BB_5, in 4, out 4

Next, when 'adjust_cfa_offset 4' is inserted in BB_4:

BB_0, in 0, out 0
BB_1, in 8, out 12
BB_2, in 12, out 12
BB_3, in 12, out 12
BB_4, in 4, out 8
BB_5, in 8, out 8

Next, when 'adjust_cfa_offset -4' is inserted in BB_1:

BB_0, in 0, out 0
BB_1, in 8, out 8
BB_2, in 8, out 8
BB_3, in 8, out 8
BB_4, in 8, out 12
BB_5, in 12, out 12

Next, when 'adjust_cfa_offset -4' is inserted in BB_4:

BB_0, in 0, out 0
BB_1, in 8, out 8
BB_2, in 8, out 8
BB_3, in 8, out 8
BB_4, in 8, out 8
BB_5, in 8, out 8

And then you end up having wrong in/out cfa offset for blocks 1-5 because the adjustments are contained in OutgoingCFAOffset which is the value that gets propagated further. That is what is propagated to the successors, instead of the increment/decrement from adjust directives. This way of calculating in/out cfa offset values produces wrong results for the case where you have a cycle (like in the example above).

What is changed now is that each 'adjust_cfa_offset' directive is propagated to the successors when it's inserted and updates their incoming and outgoing adjust values appropriately. In the end, before verifying CFI info and inserting additional CFI instructions if needed, these values are added to incoming and outgoing cfa offset (that now contain only absolute offsets that are set).

lib/CodeGen/TargetPassConfig.cpp

875

I put the added target hook in the passes themselves. Should I move it to TargetPassConfig?

[thanm]

Remark: there are several outstanding LLVM bugs for this problem:

bug 25916 filed in 2015 and
bug 20774 filed in 2014
bug 13161 filed way back in 2012

My team also ran into the same problem this month. Looking in the Google internal bug tracker I can find reports there too.

It would be great to get this bugfix in -- seems as though it has caused problems for many folks over the years.

[violetav]

Here is the new patch written based on the comments received and the previously implemented solution.
What is changed:

• Removed all changes from MachineBasicBlock.
• Removed changes related to updating in/out CFI information from common passes (BranchFolding.cpp and TailDuplicator.cpp).
• Merged CFIInfoVerifier and CFIInstrInserter into one pass that now calculates in/out CFI info for BBs, verifies it and inserts additional CFI instructions if needed. This pass isn't added in TargetPassConfig.cpp if the target does not maintain CFA info.

These are the tests that still haven't been updated (and that currently fail with this patch):

O0-pipeline.ll
avx512vl-vec-masked-cmp.ll
cmp.ll
frame-lowering-debug-intrinsic-2.ll
mul-constant-result.ll
pr29061.ll
pr34088.ll
select-mmx.ll
x86-interleaved-access.ll

They need to have appropriate .cfi directives added.
I will add these changes when this patch gets approved.

[aprantl]

include/llvm/Target/TargetFrameLowering.h

346

/// please

lib/CodeGen/CFIInstrInserter.cpp

64

please use /// for all comments describing declarations.

[thegameg]

Hi Violeta,

Thanks a lot for working on this! This looks really nice! Seems a lot easier to adapt to other targets and other CFI state like CSRs.

lib/CodeGen/CFIInstrInserter.cpp

33

I think most of the members can be private here.

53

Should this be guarded by the EXPENSIVE_CHECKS macro?

62

In LLVM we usually use

struct MBBCFAInfo {
  [...]
};

87

Why not a reference instead of a pointer? It can never be null, right?

131

No need for struct here.

156

You can already initialize the variables here.

193

You could limit the scope of SuccInfo to the loop body.

254

I think you can assume the reference is always non-null here, right?

[MatzeB]

Thanks @violetav this looks a lot better as a self-contained pass!

include/llvm/CodeGen/MachineInstr.h

795

This name is unintuitive to me. And it feels like this isn't really a category of machine instructions but rather just a combination that BranchFolding is able to deal with after this patch.

Maybe it would be better to move this into BranchFolding.cpp then in the form of
static bool isDirective(const MachineInstr &MI) { return MI.isDebugValue() || MI.isCFIInstruction(); }.

include/llvm/CodeGen/Passes.h

420–424

Even though there is a lot of precedent in this file, I'm starting to wonder how useful it is to have a description of a pass at the beginning of the passes implementation file and another one in front of the corresponding function in Passes.h. Maybe it would be best to shorten this to:

/// Creates CFI Instruction Inserter pass. \see CFIInstrInserter.cpp

so we only have to maintain one description of the passes details.

include/llvm/Target/TargetFrameLowering.h

345–347

• Use doxygen /// comments.
• I guess const MachineFunction &MF should work as well?

349–351

ditto

include/llvm/Target/TargetMachine.h

268–272

I think we should rather add a TargetPassConfig method instead of another TargetMachine callback (see below).

lib/CodeGen/CFIInstrInserter.cpp

10–18

Use doxygen comments:

/// \file This pass verifies ...

35

This feels slightly out of place as a member variable. Why not make it a return value of verify()?

53

I wouldn't go as far as EXPENSIVE_CHECKS. But I think #ifndef NDEBUG would be good to skip the verification in release builds.

77–79

MachineBasicBlocks have a dense numbering, so you can use a more efficient std::vector of length MachineFunction::getNumBlockIDs() instead. (See MachineTraceMetrics::BlockInfo for an example)

114

We tend to use lowercase pass names here. So maybe "cfi-instr-inserter"?

130

Please avoid auto in cases where the actual type is only slightly longer. I think that is friendlier to readers of the code. (same for a number of other loops below).

164–165

You could use for (MachineInstr &MI : make_range(MBBInfo->MBB->instr_begin(), MBBInfo->MBB->instr_end()) { ... } here.

205

You could move this declaration down to the use (and combine it with the first assignment).

254

References must be non-null otherwise it's undefined behavior. This means the compiler will most likely optimize this assert away and it won't do anything even if the reference happened to illegally be nullptr.

266

The declarations could be move downwards to their first use.

lib/CodeGen/MachineInstr.cpp

331–347

This looks like a good candidate for a switch().

lib/CodeGen/TargetFrameLoweringImpl.cpp

109–112

I'd simply use llvm_unreachable("getInitialCFAOffset not implemented!"); and don't even bother checking the flag. Same in the next function.

lib/CodeGen/TargetPassConfig.cpp

877–881

• I'd vote to remove the maintainsCFAInfo() flag and simply let the targets that need it add the pass as part of addPreEmitPass().
• Is the , false really necessary? I wouldn't expect the pass to produce anything that fails the machine verifier.

[violetav]

Addressed comments. Added changes for tests that were previously failing.

[violetav]

include/llvm/CodeGen/MachineInstr.h

795

This is used in BranchFolding and TailDuplication. It was suggested previously to be done this way https://reviews.llvm.org/D18046?id=101198#inline-294697

[MatzeB]

This is starting to look good, a few more nitpicks below and hopefullly @iteratee will comment on the isDirective() thing.

include/llvm/CodeGen/MachineInstr.h

795

@iteratee Have you seen isMetaInstruction() and isTransient()? This feels like yet another variation of machine instructions that don't produce code. However I don't know why this specific combination is interesting and the generic name and no comment doesn't help understanding that either.

lib/CodeGen/MachineInstr.cpp

333–353

Keep case/default at the same indentation level as the switch.

354–355

As there are only 2 or 3 cases that are not handled yet, I would recommend to explicitely add cases for them and not add a default: at all. The benefit of this is that when someone adds new OpTypes in the future he will immediately get a compiler warning for a missing case when he forgets to add a case here.

lib/CodeGen/TargetFrameLoweringImpl.cpp

110

llvm_unreachable never returns so you can leave out the return statement behind it. Same in the next function.

[iteratee]

include/llvm/CodeGen/MachineInstr.h

795

I hadn't.
Violeta, are there any reasons those 2 categories won't work?

[thegameg]

lib/CodeGen/CFIInstrInserter.cpp

203

This could be a const MBBCFAInfo * to avoid copies.

211

And this const MBBCFAInfo &.

263

Same.

265

Same.

293

Same.

[violetav]

Addressed comments.

[violetav]

include/llvm/CodeGen/MachineInstr.h

795

Four regression tests fail when isDirective() is replaced with isMetaInstruction():
avx512-mask-op.ll
conditional-tailcall.ll
extra-toc-reg-deps.ll
tail-dup-merge-loop-headers.ll

Different code gets generated because of some tail merging and tail duplicating that don't happen with clean version of the code. At first glance, it seems that presence of IMPLICIT_DEF affects BranchFolding, and presence of KILL affects TailDuplicator. I haven't looked at this further than comparing .s files.

[iteratee]

include/llvm/CodeGen/MachineInstr.h

795

violetav:

For Tail duplication I'm certain that it's fine. It's just counting instructions, and so it should skip all the meta instructions.

If you post the diff for BranchFolding, I'll look at it. I'm guessing it's fine too, but I'm less certain.

It would be easier if you did that first in a separate patch.

[violetav]

include/llvm/CodeGen/MachineInstr.h

795

Hi,

Other than different code being generated in some tests, there are at least two problems when isMetaInstruction() is used instead of isDirective() in BranchFolding.
First of all, even without this patch (only when places where there's isDirective() in it are replaced with isMetaInstruction()) there is a test that fails with 'Bad machine code' errors. That's test/CodeGen/PowerPC/extra-toc-reg-deps.ll and it reports:

Bad machine code: MBB has more than one landing pad successor ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#49 lpad175 (0x16665d0)

Bad machine code: MBB exits via unconditional branch but doesn't have exactly one CFG successor! ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#49 lpad175 (0x16665d0)

Bad machine code: MBB has more than one landing pad successor ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#53 lpad230 (0x1666da0)

Bad machine code: MBB exits via unconditional branch but doesn't have exactly one CFG successor! ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#53 lpad230 (0x1666da0)

Bad machine code: MBB has more than one landing pad successor ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#63 _ZN4Foam4ListIiED2Ev.exit.i3073 (0x1668510)

Bad machine code: MBB exits via unconditional branch but doesn't have exactly one CFG successor! ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#63 _ZN4Foam4ListIiED2Ev.exit.i3073 (0x1668510)

Bad machine code: MBB has more than one landing pad successor ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#70 lpad905.loopexit.split-lp (0x1668e00)

Bad machine code: MBB exits via unconditional branch but doesn't have exactly one CFG successor! ***
function: _ZN4Foam13checkTopologyERKNS_8polyMeshEbb
basic block: BB#70 lpad905.loopexit.split-lp (0x1668e00)
LLVM ERROR: Found 8 machine code errors.

The second problem is seen in test/CodeGen/X86/conditional-tailcall.ll, which fails with following error:

The incoming offset of a successor is inconsistent. ***
function: pr31257
basic block: BB#12 if.else28 (0x903c28)
Successor BB#14 has incoming offset (32), while BB#12 has outgoing offset (48).

The outgoing offset of a predecessor is inconsistent. ***
function: pr31257
basic block: BB#14 cleanup.thread (0x903e98)
Predecessor BB#12 has outgoing offset (48), while BB#14 has incoming offset (32).
LLVM ERROR: Found 2 in/out CFI information errors.

What happens here is that a case appears where an iterator that points to the start of the common tail, points to a CFI instruction, and then that CFI instruction gets cut off, with a jump to the common tail, and gets lost. This then results in setting wrong outgoing CFA offset for that block (because one CFI instruction is missing), and the CFI Info Verifier reports a mismatch between outgoing offset of that block and incoming offset of its successor.

Errors about incorrect cfa offset are also reported with recursive LLVM build.

Considering these failures, I am not sure that isMetaInstruction() is safe to use here. I am not familiar with how these types of instructions should be handled.

It makes sense that isDirective() method is not added to MachineInstr, as it is used only for this purpose. If isMetaInstruction() turns out to be unsuitable for use here, we could, as Matthias suggested, add something such as shouldSkipInstr(MI) to BranchFolding and TailDuplication.

[violetav]

Is there anything else that should be done for this patch?

[MatzeB]

In D35844#899665, @violetav wrote:

Is there anything else that should be done for this patch?

Could you try changing it to not add MachineInstr::isDirective()? The rest looked good.

[violetav]

Could you try changing it to not add MachineInstr::isDirective()?

Would changing the name 'isDirective()' into something more sensible and adding a description be acceptable?
Or maybe moving it to BranchFolding and using isMetaInstruction() in TailDuplication?
Or leaving the 'isDebugValue() || isCFIInstruction()' condition without adding a special method for this particular case?

I know that isDirective() would currently be used only in BranchFolding, but I suspect that it will also need to be used elsewhere when support for CFI instructions in epilogue is added to other targets.

[violetav]

Removed MachineInstr::isDirective(). Added countsAsInstruction(const MachineInstr &MI) in BranchFolding. Replaced isDirective() with isMetaInstruction() in TailDuplication.

[rnk]

I'd like to see this go in so we can iterate on it in tree.

test/CodeGen/X86/2011-10-19-widen_vselect.ll

92

These are gone now. You will need to rebase this patch.

[MatzeB]

LGTM, thanks!

[violetav]

Rebased the patch.
Thank you all for the comments!

[violetav]

Hi,
As this change was reverted in r317726, is the way to solve this issue to leave CFI_INSTRUCTION to be notDuplicable, and allow TailDuplication to generate different code when cfi_directives are present (e.g. not to duplicate epilogue blocks, if they contain cfi_directives) ?

include/llvm/CodeGen/MachineInstr.h

public:
   /// A DBG_VALUE is indirect iff the first operand is a register and
   /// the second operand is an immediate.
   bool isIndirectDebugValue() const {
     return isDebugValue()
       && getOperand(0).isReg()
       && getOperand(1).isImm();
   }
 
+  bool isDirective() const { return isDebugValue() || isCFIInstruction(); }
   bool isPHI() const {
     return getOpcode() == TargetOpcode::PHI ||
            getOpcode() == TargetOpcode::G_PHI;
   }
   bool isKill() const { return getOpcode() == TargetOpcode::KILL; }
   bool isImplicitDef() const { return getOpcode()==TargetOpcode::IMPLICIT_DEF; }
   bool isInlineAsm() const { return getOpcode() == TargetOpcode::INLINEASM; }
 

include/llvm/CodeGen/Passes.h

/// MachineDominanaceFrontier - This pass is a machine dominators analysis pass.
   /// This pass performs outlining on machine instructions directly before
   /// printing assembly.
   ModulePass *createMachineOutlinerPass();
 
   /// This pass expands the experimental reduction intrinsics into sequences of
   /// shuffles.
   FunctionPass *createExpandReductionsPass();
 
+  /// This pass verifies that outgoing cfa offset and register of predecessor
+  /// blocks match incoming cfa offset and register of their successors. Then it
+  /// checks if blocks have correct CFA calculation rule set and inserts
+  /// required CFI instruction at their beginnings if they don't (due to
+  /// non-linear block layout).
+  FunctionPass *createCFIInstrInserter();
+
 } // End llvm namespace
 
 #endif

include/llvm/InitializePasses.h

 void initializeBranchProbabilityInfoWrapperPassPass(PassRegistry&);
 void initializeBranchRelaxationPass(PassRegistry&);
 void initializeBreakCriticalEdgesPass(PassRegistry&);
 void initializeCFGOnlyPrinterLegacyPassPass(PassRegistry&);
 void initializeCFGOnlyViewerLegacyPassPass(PassRegistry&);
 void initializeCFGPrinterLegacyPassPass(PassRegistry&);
 void initializeCFGSimplifyPassPass(PassRegistry&);
 void initializeCFGViewerLegacyPassPass(PassRegistry&);
+void initializeCFIInstrInserterPass(PassRegistry&);
 void initializeCFLAndersAAWrapperPassPass(PassRegistry&);
 void initializeCFLSteensAAWrapperPassPass(PassRegistry&);
 void initializeCallGraphDOTPrinterPass(PassRegistry&);
 void initializeCallGraphPrinterLegacyPassPass(PassRegistry&);
 void initializeCallGraphViewerPass(PassRegistry&);
 void initializeCallGraphWrapperPassPass(PassRegistry&);
 void initializeCodeGenPreparePass(PassRegistry&);
 void initializeConstantHoistingLegacyPassPass(PassRegistry&);

include/llvm/Target/Target.td

def INLINEASM : Instruction {
   let AsmString = "";
   let hasSideEffects = 0;  // Note side effect is encoded in an operand.
 }
 def CFI_INSTRUCTION : Instruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "";
   let hasCtrlDep = 1;
-  let isNotDuplicable = 1;
+  let isNotDuplicable = 0;
 }
 def EH_LABEL : Instruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "";
   let hasCtrlDep = 1;
   let isNotDuplicable = 1;
 }

include/llvm/Target/TargetFrameLowering.h

if (!F->hasLocalLinkage() || F->hasAddressTaken() ||
       return false;
     // Function should not be optimized as tail call.
     for (const User *U : F->users())
       if (auto CS = ImmutableCallSite(U))
         if (CS.isTailCall())
           return false;
     return true;
   }
+
+  // Return initial CFA offset value i.e. the one valid at the beginning of the
+  // function (before any stack operations).
+  virtual int getInitialCFAOffset(MachineFunction &MF) const;
+
+  // Return initial CFA register value i.e. the one valid at the beginning of
+  // the function (before any stack operations).
+  virtual unsigned getInitialCFARegister(MachineFunction &MF) const;
 };
 
 } // End llvm namespace
 
 #endif

include/llvm/Target/TargetMachine.h

public:
   /// spilling and scavenging are not needed or used.
   virtual bool usesPhysRegsForPEI() const { return true; }
 
   /// True if the target wants to use interprocedural register allocation by
   /// default. The -enable-ipra flag can be used to override this.
   virtual bool useIPRA() const {
     return false;
   }
+
+  /// Check whether CFA info is maintained for this target. If not, it shouldn't
+  /// be verified for consistency and additional CFI instructions should not be
+  /// inserted to correct CFA calculation rule.
+  virtual bool maintainsCFAInfo() const { return false; }
 };
 
 /// This class describes a target machine that is implemented with the LLVM
 /// target-independent code generator.
 ///
 class LLVMTargetMachine : public TargetMachine {
 protected: // Can only create subclasses.
   LLVMTargetMachine(const Target &T, StringRef DataLayoutString,

lib/CodeGen/BranchFolding.cpp

static unsigned ComputeCommonTailLength(MachineBasicBlock *MBB1,
                                         MachineBasicBlock::iterator &I2) {
   I1 = MBB1->end();
   I2 = MBB2->end();
 
   unsigned TailLen = 0;
   while (I1 != MBB1->begin() && I2 != MBB2->begin()) {
     --I1; --I2;
     // Skip debugging pseudos; necessary to avoid changing the code.
-    while (I1->isDebugValue()) {
+    while (I1->isDirective()) {
       if (I1==MBB1->begin()) {
-        while (I2->isDebugValue()) {
+        while (I2->isDirective()) {
           if (I2==MBB2->begin())
             // I1==DBG at begin; I2==DBG at begin
             return TailLen;
           --I2;
         }
         ++I2;
         // I1==DBG at begin; I2==non-DBG, or first of DBGs not at begin
         return TailLen;
       }
       --I1;
     }
     // I1==first (untested) non-DBG preceding known match
-    while (I2->isDebugValue()) {
+    while (I2->isDirective()) {
       if (I2==MBB2->begin()) {
         ++I1;
         // I1==non-DBG, or first of DBGs not at begin; I2==DBG at begin
         return TailLen;
       }
       --I2;
     }
     // I1, I2==first (untested) non-DBGs preceding known match
if (I2 == MBB2->begin() && I1 != MBB1->begin()) {
     --I1;
     while (I1->isDebugValue()) {
       if (I1 == MBB1->begin())
         return TailLen;
       --I1;
     }
     ++I1;
   }
+
+  // Ensure that I1 and I2 do not point to a CFI_INSTRUCTION. This can happen if
+  // I1 and I2 are non-identical when compared and then one or both of them ends
+  // up pointing to a CFI instruction after being incremented. For example:
+  /*
+    BB1:
+    ...
+    INSTRUCTION_A
+    ADD32ri8  <- last common instruction
+    ...
+    BB2:
+    ...
+    INSTRUCTION_B
+    CFI_INSTRUCTION
+    ADD32ri8  <- last common instruction
+    ...
+  */
+  // When INSTRUCTION_A and INSTRUCTION_B are compared as not equal, after
+  // incrementing the iterators, I1 will point to ADD, however I2 will point to
+  // the CFI instruction. Later on, this leads to BB2 being 'hacked off' at the
+  // wrong place (in ReplaceTailWithBranchTo()) which results in losing this CFI
+  // instruction.
+  while (I1 != MBB1->end() && I1->isCFIInstruction()) {
+    ++I1;
+  }
+
+  while (I2 != MBB2->end() && I2->isCFIInstruction()) {
+    ++I2;
+  }
   return TailLen;
 }
 
 void BranchFolder::replaceTailWithBranchTo(MachineBasicBlock::iterator OldInst,
                                            MachineBasicBlock &NewDest) {
   if (UpdateLiveIns) {
     // OldInst should always point to an instruction.
     MachineBasicBlock &OldMBB = *OldInst->getParent();

 }
 
 /// EstimateRuntime - Make a rough estimate for how long it will take to run
 /// the specified code.
 static unsigned EstimateRuntime(MachineBasicBlock::iterator I,
                                 MachineBasicBlock::iterator E) {
   unsigned Time = 0;
   for (; I != E; ++I) {
-    if (I->isDebugValue())
+    if (I->isDirective())
       continue;
     if (I->isCall())
       Time += 10;
     else if (I->mayLoad() || I->mayStore())
       Time += 2;
     else
       ++Time;
   }
mergeOperations(MachineBasicBlock::iterator MBBIStartPos,
   MachineBasicBlock::reverse_iterator MBBIE = MBB->rend();
   MachineBasicBlock::reverse_iterator MBBICommon = MBBCommon.rbegin();
   MachineBasicBlock::reverse_iterator MBBIECommon = MBBCommon.rend();
 
   while (CommonTailLen--) {
     assert(MBBI != MBBIE && "Reached BB end within common tail length!");
     (void)MBBIE;
 
-    if (MBBI->isDebugValue()) {
+    if (MBBI->isDirective()) {
       ++MBBI;
       continue;
     }
 
-    while ((MBBICommon != MBBIECommon) && MBBICommon->isDebugValue())
+    while ((MBBICommon != MBBIECommon) && MBBICommon->isDirective())
       ++MBBICommon;
 
     assert(MBBICommon != MBBIECommon &&
            "Reached BB end within common tail length!");
     assert(MBBICommon->isIdenticalTo(*MBBI) && "Expected matching MIIs!");
 
     // Merge MMOs from memory operations in the common block.
     if (MBBICommon->mayLoad() || MBBICommon->mayStore())
if (i != commonTailIndex) {
       mergeOperations(SameTails[i].getTailStartPos(), *MBB);
     } else {
       assert(SameTails[i].getTailStartPos() == MBB->begin() &&
           "MBB is not a common tail only block");
     }
   }
 
   for (auto &MI : *MBB) {
-    if (MI.isDebugValue())
+    if (MI.isDirective())
       continue;
     DebugLoc DL = MI.getDebugLoc();
     for (unsigned int i = 0 ; i < NextCommonInsts.size() ; i++) {
       if (i == commonTailIndex)
         continue;
 
       auto &Pos = NextCommonInsts[i];
       assert(Pos != SameTails[i].getBlock()->end() &&
           "Reached BB end within common tail");
-      while (Pos->isDebugValue()) {
+      while (Pos->isDirective()) {
         ++Pos;
         assert(Pos != SameTails[i].getBlock()->end() &&
             "Reached BB end within common tail");
       }
       assert(MI.isIdenticalTo(*Pos) && "Expected matching MIIs!");
       DL = DILocation::getMergedLocation(DL, Pos->getDebugLoc());
       NextCommonInsts[i] = ++Pos;
     }

lib/CodeGen/CFIInstrInserter.cpp

+//===------ CFIInstrInserter.cpp - Insert additional CFI instructions -----===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass verifies incoming and outgoing CFA information of basic blocks. CFA
+// information is information about offset and register set by CFI directives,
+// valid at the start and end of a basic block. This pass checks that outgoing
+// information of predecessors matches incoming information of their successors.
+// Then it checks if blocks have correct CFA calculation rule set and inserts
+// additional CFI instruction at their beginnings if they don't. CFI
+// instructions are inserted if basic blocks have incorrect offset or register
+// set by previous blocks, as a result of a non-linear layout of blocks in a
+// function.
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/Target/TargetFrameLowering.h"
+#include "llvm/Target/TargetInstrInfo.h"
+#include "llvm/Target/TargetMachine.h"
+#include "llvm/Target/TargetSubtargetInfo.h"
+using namespace llvm;
+
+namespace {
+class CFIInstrInserter : public MachineFunctionPass {
+ public:
+  static char ID;
+  int ErrorNum;
+
+  CFIInstrInserter() : MachineFunctionPass(ID) {
+    initializeCFIInstrInserterPass(*PassRegistry::getPassRegistry());
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesAll();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override {
+
+    if (!MF.getMMI().hasDebugInfo() &&
+        !MF.getFunction()->needsUnwindTableEntry())
+      return false;
+
+    calculateCFAInfo(MF);
+    verify(MF);
+    if (ErrorNum)
+      report_fatal_error("Found " + Twine(ErrorNum) +
+                         " in/out CFI information errors.");
+    bool insertedCFI = insertCFIInstrs(MF);
+    MBBMap.clear();
+    return insertedCFI;
+  }
+
+  typedef struct MBBCFAInfo {
+    MachineBasicBlock *MBB;
+    // Value of cfa offset valid at basic block entry.
+    int IncomingCFAOffset = -1;
+    // Value of cfa offset valid at basic block exit.
+    int OutgoingCFAOffset = -1;
+    // Value of cfa register valid at basic block entry.
+    unsigned IncomingCFARegister = 0;
+    // Value of cfa register valid at basic block exit.
+    unsigned OutgoingCFARegister = 0;
+    // If in/out cfa offset and register values for this block have already been
+    // set or not.
+    bool Processed = false;
+  } MBBCFAInfo;
+
+  // Contains cfa offset and register values valid at entry and exit of basic
+  // blocks.
+  std::map<unsigned, MBBCFAInfo> MBBMap;
+
+  // Calculate cfa offset and register values valid at entry and exit for all
+  // basic blocks in a function.
+  void calculateCFAInfo(MachineFunction &MF);
+  // Calculate cfa offset and register values valid at basic block exit by
+  // checking the block for CFI instructions. Block's incoming CFA info remains
+  // the same.
+  void calculateOutgoingCFAInfo(MBBCFAInfo *MBBInfo);
+  // Update in/out cfa offset and register values for successors of the basic
+  // block.
+  void updateSuccCFAInfo(MBBCFAInfo *MBBInfo);
+
+  // Check if incoming CFA information of a basic block matches outgoing CFA
+  // information of the previous block. If it doesn't, insert CFI instruction at
+  // the beginning of the block that corrects the CFA calculation rule for that
+  // block.
+  bool insertCFIInstrs(MachineFunction &MF);
+  // Return the cfa offset value that should be set at the beginning of a MBB if
+  // needed. The negated value is needed when creating CFI instructions that set
+  // absolute offset.
+  int getCorrectCFAOffset(MachineBasicBlock *MBB) {
+    return -MBBMap[MBB->getNumber()].IncomingCFAOffset;
+  }
+
+  void report(const char *msg, MachineBasicBlock &MBB);
+  // Go through each MBB in a function and check that outgoing offset and
+  // register of its predecessors match incoming offset and register of that
+  // MBB, as well as that incoming offset and register of its successors match
+  // outgoing offset and register of the MBB.
+  void verify(MachineFunction &MF);
+};
+}
+
+char CFIInstrInserter::ID = 0;
+INITIALIZE_PASS(CFIInstrInserter, "CFIInstrInserter",
+                "Check CFA info and insert CFI instructions if needed", false,
+                false)
+FunctionPass *llvm::createCFIInstrInserter() { return new CFIInstrInserter(); }
+
+void CFIInstrInserter::calculateCFAInfo(MachineFunction &MF) {
+  // Initial CFA offset value i.e. the one valid at the beginning of the
+  // function.
+  int InitialOffset =
+      MF.getSubtarget().getFrameLowering()->getInitialCFAOffset(MF);
+  // Initial CFA register value i.e. the one valid at the beginning of the
+  // function.
+  unsigned InitialRegister =
+      MF.getSubtarget().getFrameLowering()->getInitialCFARegister(MF);
+
+  // Initialize MBBMap.
+  for (auto &MBB : MF) {
+    struct MBBCFAInfo MBBInfo;
+    MBBInfo.MBB = &MBB;
+    MBBInfo.IncomingCFAOffset = InitialOffset;
+    MBBInfo.OutgoingCFAOffset = InitialOffset;
+    MBBInfo.IncomingCFARegister = InitialRegister;
+    MBBInfo.OutgoingCFARegister = InitialRegister;
+    std::pair<unsigned, MBBCFAInfo> MBBEntry(MBB.getNumber(), MBBInfo);
+    MBBMap.insert(MBBEntry);
+  }
+
+  // Set in/out cfa info for all blocks in the function. This traversal is based
+  // on the assumption that the first block in the function is the entry block
+  // i.e. that it has initial cfa offset and register values as incoming CFA
+  // information.
+  for (auto &MBB : MF) {
+    if (MBBMap[MBB.getNumber()].Processed) continue;
+    calculateOutgoingCFAInfo(&MBBMap[MBB.getNumber()]);
+    updateSuccCFAInfo(&MBBMap[MBB.getNumber()]);
+  }
+}
+
+void CFIInstrInserter::calculateOutgoingCFAInfo(MBBCFAInfo *MBBInfo) {
+  // Outgoing cfa offset set by the block.
+  int SetOffset;
+  // Outgoing cfa register set by the block.
+  unsigned SetRegister;
+  const std::vector<MCCFIInstruction> &Instrs =
+      MBBInfo->MBB->getParent()->getFrameInstructions();
+
+  SetOffset = MBBInfo->IncomingCFAOffset;
+  SetRegister = MBBInfo->IncomingCFARegister;
+
+  // Determine cfa offset and register set by the block.
+  for (MachineBasicBlock::instr_iterator MI = MBBInfo->MBB->instr_begin();
+       MI != MBBInfo->MBB->instr_end(); ++MI) {
+    if (MI->isCFIInstruction()) {
+      unsigned CFIIndex = MI->getOperand(0).getCFIIndex();
+      const MCCFIInstruction &CFI = Instrs[CFIIndex];
+      if (CFI.getOperation() == MCCFIInstruction::OpDefCfaRegister) {
+        SetRegister = CFI.getRegister();
+      } else if (CFI.getOperation() == MCCFIInstruction::OpDefCfaOffset) {
+        SetOffset = CFI.getOffset();
+      } else if (CFI.getOperation() == MCCFIInstruction::OpAdjustCfaOffset) {
+        SetOffset += CFI.getOffset();
+      } else if (CFI.getOperation() == MCCFIInstruction::OpDefCfa) {
+        SetRegister = CFI.getRegister();
+        SetOffset = CFI.getOffset();
+      }
+    }
+  }
+
+  MBBInfo->Processed = true;
+
+  // Update outgoing CFA info.
+  MBBInfo->OutgoingCFAOffset = SetOffset;
+  MBBInfo->OutgoingCFARegister = SetRegister;
+}
+
+void CFIInstrInserter::updateSuccCFAInfo(MBBCFAInfo *MBBInfo) {
+
+  MBBCFAInfo *SuccInfo;
+  for (auto Succ : MBBInfo->MBB->successors()) {
+    SuccInfo = &MBBMap[Succ->getNumber()];
+    if (SuccInfo->Processed) continue;
+    SuccInfo->IncomingCFAOffset = MBBInfo->OutgoingCFAOffset;
+    SuccInfo->IncomingCFARegister = MBBInfo->OutgoingCFARegister;
+    calculateOutgoingCFAInfo(SuccInfo);
+    updateSuccCFAInfo(SuccInfo);
+  }
+}
+
+bool CFIInstrInserter::insertCFIInstrs(MachineFunction &MF) {
+
+  MBBCFAInfo PrevMBBInfo = MBBMap[MF.front().getNumber()];
+  MBBCFAInfo MBBInfo;
+  const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
+  bool InsertedCFIInstr = false;
+
+  for (auto &MBB : MF) {
+    // Skip the first MBB in a function
+    if (MBB.getNumber() == MF.front().getNumber()) continue;
+
+    MBBInfo = MBBMap[MBB.getNumber()];
+    auto MBBI = MBBInfo.MBB->begin();
+    DebugLoc DL = MBBInfo.MBB->findDebugLoc(MBBI);
+
+    if (PrevMBBInfo.OutgoingCFAOffset != MBBInfo.IncomingCFAOffset) {
+      // If both outgoing offset and register of a previous block don't match
+      // incoming offset and register of this block, add a def_cfa instruction
+      // with the correct offset and register for this block.
+      if (PrevMBBInfo.OutgoingCFARegister != MBBInfo.IncomingCFARegister) {
+        unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createDefCfa(
+            nullptr, MBBInfo.IncomingCFARegister, getCorrectCFAOffset(&MBB)));
+        BuildMI(*MBBInfo.MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+            .addCFIIndex(CFIIndex);
+        // If outgoing offset of a previous block doesn't match incoming offset
+        // of this block, add a def_cfa_offset instruction with the correct
+        // offset for this block.
+      } else {
+        unsigned CFIIndex =
+            MF.addFrameInst(MCCFIInstruction::createDefCfaOffset(
+                nullptr, getCorrectCFAOffset(&MBB)));
+        BuildMI(*MBBInfo.MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+            .addCFIIndex(CFIIndex);
+      }
+      InsertedCFIInstr = true;
+      // If outgoing register of a previous block doesn't match incoming
+      // register of this block, add a def_cfa_register instruction with the
+      // correct register for this block.
+    } else if (PrevMBBInfo.OutgoingCFARegister != MBBInfo.IncomingCFARegister) {
+      unsigned CFIIndex =
+          MF.addFrameInst(MCCFIInstruction::createDefCfaRegister(
+              nullptr, MBBInfo.IncomingCFARegister));
+      BuildMI(*MBBInfo.MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
+          .addCFIIndex(CFIIndex);
+      InsertedCFIInstr = true;
+    }
+    PrevMBBInfo = MBBInfo;
+  }
+  return InsertedCFIInstr;
+}
+
+void CFIInstrInserter::report(const char *msg, MachineBasicBlock &MBB) {
+  assert(&MBB);
+  ErrorNum++;
+  errs() << '\n';
+  errs() << "*** " << msg << " ***\n"
+         << "- function:    " << MBB.getParent()->getName() << "\n";
+  errs() << "- basic block: BB#" << MBB.getNumber() << ' ' << MBB.getName()
+         << " (" << (const void *)&MBB << ')';
+  errs() << '\n';
+}
+
+void CFIInstrInserter::verify(MachineFunction &MF) {
+  ErrorNum = 0;
+  MBBCFAInfo CurrMBBInfo, PredMBBInfo, SuccMBBInfo;
+  for (auto &CurrMBB : MF) {
+    CurrMBBInfo = MBBMap[CurrMBB.getNumber()];
+    for (auto Pred : CurrMBB.predecessors()) {
+      PredMBBInfo = MBBMap[Pred->getNumber()];
+      // Check that outgoing offset values of predecessors match the incoming
+      // offset value of CurrMBB
+      if (PredMBBInfo.OutgoingCFAOffset != CurrMBBInfo.IncomingCFAOffset) {
+        report("The outgoing offset of a predecessor is inconsistent.",
+               CurrMBB);
+        errs() << "Predecessor BB#" << Pred->getNumber()
+               << " has outgoing offset (" << PredMBBInfo.OutgoingCFAOffset
+               << "), while BB#" << CurrMBB.getNumber()
+               << " has incoming offset (" << CurrMBBInfo.IncomingCFAOffset
+               << ").\n";
+      }
+      // Check that outgoing register values of predecessors match the incoming
+      // register value of CurrMBB
+      if (PredMBBInfo.OutgoingCFARegister != CurrMBBInfo.IncomingCFARegister) {
+        report("The outgoing register of a predecessor is inconsistent.",
+               CurrMBB);
+        errs() << "Predecessor BB#" << Pred->getNumber()
+               << " has outgoing register (" << PredMBBInfo.OutgoingCFARegister
+               << "), while BB#" << CurrMBB.getNumber()
+               << " has incoming register (" << CurrMBBInfo.IncomingCFARegister
+               << ").\n";
+      }
+    }
+
+    for (auto Succ : CurrMBB.successors()) {
+      SuccMBBInfo = MBBMap[Succ->getNumber()];
+      // Check that incoming offset values of successors match the outgoing
+      // offset value of CurrMBB
+      if (SuccMBBInfo.IncomingCFAOffset != CurrMBBInfo.OutgoingCFAOffset) {
+        report("The incoming offset of a successor is inconsistent.", CurrMBB);
+        errs() << "Successor BB#" << Succ->getNumber()
+               << " has incoming offset (" << SuccMBBInfo.IncomingCFAOffset
+               << "), while BB#" << CurrMBB.getNumber()
+               << " has outgoing offset (" << CurrMBBInfo.OutgoingCFAOffset
+               << ").\n";
+      }
+      // Check that incoming register values of successors match the outgoing
+      // register value of CurrMBB
+      if (SuccMBBInfo.IncomingCFARegister != CurrMBBInfo.OutgoingCFARegister) {
+        report("The incoming register of a successor is inconsistent.",
+               CurrMBB);
+        errs() << "Successor BB#" << Succ->getNumber()
+               << " has incoming register (" << SuccMBBInfo.IncomingCFARegister
+               << "), while BB#" << CurrMBB.getNumber()
+               << " has outgoing register (" << CurrMBBInfo.OutgoingCFARegister
+               << ").\n";
+      }
+    }
+  }
+}

lib/CodeGen/CMakeLists.txt

 add_llvm_library(LLVMCodeGen
   AggressiveAntiDepBreaker.cpp
   AllocationOrder.cpp
   Analysis.cpp
   AtomicExpandPass.cpp
   BasicTargetTransformInfo.cpp
   BranchFolding.cpp
   BranchRelaxation.cpp
   BuiltinGCs.cpp
   CalcSpillWeights.cpp
   CallingConvLower.cpp
+  CFIInstrInserter.cpp
   CodeGen.cpp
   CodeGenPrepare.cpp
   CountingFunctionInserter.cpp
   CriticalAntiDepBreaker.cpp
   DeadMachineInstructionElim.cpp
   DetectDeadLanes.cpp
   DFAPacketizer.cpp
   DwarfEHPrepare.cpp

lib/CodeGen/CodeGen.cpp

 
 using namespace llvm;
 
 /// initializeCodeGen - Initialize all passes linked into the CodeGen library.
 void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeAtomicExpandPass(Registry);
   initializeBranchFolderPassPass(Registry);
   initializeBranchRelaxationPass(Registry);
+  initializeCFIInstrInserterPass(Registry);
   initializeCodeGenPreparePass(Registry);
   initializeCountingFunctionInserterPass(Registry);
   initializeDeadMachineInstructionElimPass(Registry);
   initializeDetectDeadLanesPass(Registry);
   initializeDwarfEHPreparePass(Registry);
   initializeEarlyIfConverterPass(Registry);
   initializeExpandISelPseudosPass(Registry);
   initializeExpandPostRAPass(Registry);

lib/CodeGen/MachineInstr.cpp

case MachineOperand::MO_RegisterLiveOut: {
     const TargetRegisterInfo *TRI = MF->getSubtarget().getRegisterInfo();
     unsigned RegMaskSize = (TRI->getNumRegs() + 31) / 32;
 
     // Deep compare of the two RegMasks
     return std::equal(RegMask, RegMask + RegMaskSize, OtherRegMask);
   }
   case MachineOperand::MO_MCSymbol:
     return getMCSymbol() == Other.getMCSymbol();
-  case MachineOperand::MO_CFIIndex:
-    return getCFIIndex() == Other.getCFIIndex();
+  case MachineOperand::MO_CFIIndex: {
+    const MachineFunction *MF = getParent()->getParent()->getParent();
+    const MachineFunction *OtherMF =
+        Other.getParent()->getParent()->getParent();
+    MCCFIInstruction Inst = MF->getFrameInstructions()[getCFIIndex()];
+    MCCFIInstruction OtherInst =
+        OtherMF->getFrameInstructions()[Other.getCFIIndex()];
+    MCCFIInstruction::OpType op = Inst.getOperation();
+    if (op != OtherInst.getOperation()) return false;
+    if (op == MCCFIInstruction::OpDefCfa || op == MCCFIInstruction::OpOffset ||
+        op == MCCFIInstruction::OpRestore ||
+        op == MCCFIInstruction::OpUndefined ||
+        op == MCCFIInstruction::OpSameValue ||
+        op == MCCFIInstruction::OpDefCfaRegister ||
+        op == MCCFIInstruction::OpRelOffset ||
+        op == MCCFIInstruction::OpRegister)
+      if (Inst.getRegister() != OtherInst.getRegister()) return false;
+    if (op == MCCFIInstruction::OpRegister)
+      if (Inst.getRegister2() != OtherInst.getRegister2()) return false;
+    if (op == MCCFIInstruction::OpDefCfa || op == MCCFIInstruction::OpOffset ||
+        op == MCCFIInstruction::OpRelOffset ||
+        op == MCCFIInstruction::OpDefCfaOffset ||
+        op == MCCFIInstruction::OpAdjustCfaOffset ||
+        op == MCCFIInstruction::OpGnuArgsSize)
+      if (Inst.getOffset() != OtherInst.getOffset()) return false;
+    return true;
+  }
   case MachineOperand::MO_Metadata:
     return getMetadata() == Other.getMetadata();
   case MachineOperand::MO_IntrinsicID:
     return getIntrinsicID() == Other.getIntrinsicID();
   case MachineOperand::MO_Predicate:
     return getPredicate() == Other.getPredicate();
   }
   llvm_unreachable("Invalid machine operand type");
 }
 
 // Note: this must stay exactly in sync with isIdenticalTo above.
 hash_code llvm::hash_value(const MachineOperand &MO) {
   switch (MO.getType()) {
   case MachineOperand::MO_Register:
return hash_combine(MO.getType(), MO.getTargetFlags(),
                         MO.getBlockAddress(), MO.getOffset());
   case MachineOperand::MO_RegisterMask:
   case MachineOperand::MO_RegisterLiveOut:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getRegMask());
   case MachineOperand::MO_Metadata:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getMetadata());
   case MachineOperand::MO_MCSymbol:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getMCSymbol());
-  case MachineOperand::MO_CFIIndex:
-    return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getCFIIndex());
+  case MachineOperand::MO_CFIIndex: {
+    const MachineFunction *MF = MO.getParent()->getParent()->getParent();
+    MCCFIInstruction Inst = MF->getFrameInstructions()[MO.getCFIIndex()];
+    return hash_combine(MO.getType(), MO.getTargetFlags(), Inst.getOperation(),
+                        Inst.getRegister(), Inst.getRegister2(),
+                        Inst.getOffset());
+  }
   case MachineOperand::MO_IntrinsicID:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getIntrinsicID());
   case MachineOperand::MO_Predicate:
     return hash_combine(MO.getType(), MO.getTargetFlags(), MO.getPredicate());
   }
   llvm_unreachable("Invalid machine operand type");
 }
 

lib/CodeGen/TailDuplicator.cpp

if (PreRegAlloc && MI.isReturn())
       return false;
 
     // Avoid duplicating calls before register allocation. Calls presents a
     // barrier to register allocation so duplicating them may end up increasing
     // spills.
     if (PreRegAlloc && MI.isCall())
       return false;
 
-    if (!MI.isPHI() && !MI.isDebugValue())
+    if (!MI.isPHI() && !MI.isDirective())
         InstrCount += 1;
 
     if (InstrCount > MaxDuplicateCount)
       return false;
   }
 
   // Check if any of the successors of TailBB has a PHI node in which the
   // value corresponding to TailBB uses a subregister.
   // If a phi node uses a register paired with a subregister, the actual

lib/CodeGen/TargetFrameLoweringImpl.cpp

unsigned TargetFrameLowering::getStackAlignmentSkew(
     const MachineFunction &MF) const {
   // When HHVM function is called, the stack is skewed as the return address
   // is removed from the stack before we enter the function.
   if (LLVM_UNLIKELY(MF.getFunction()->getCallingConv() == CallingConv::HHVM))
     return MF.getTarget().getPointerSize();
 
   return 0;
 }
+
+int TargetFrameLowering::getInitialCFAOffset(MachineFunction &MF) const {
+  assert(!MF.getTarget().maintainsCFAInfo() &&
+         "Must implement getInitialCFAOffset() if target maintains CFA "
+         "information!");
+  return -1;
+}
+
+unsigned TargetFrameLowering::getInitialCFARegister(MachineFunction &MF) const {
+  assert(!MF.getTarget().maintainsCFAInfo() &&
+         "Must implement getInitialCFARegister() if target maintains CFA "
+         "information!");
+  return 0;
+}
 No newline at end of file

lib/CodeGen/TargetPassConfig.cpp

if (addGCPasses()) {
     if (PrintGCInfo)
       addPass(createGCInfoPrinter(dbgs()), false, false);
   }
 
   // Basic block placement.
   if (getOptLevel() != CodeGenOpt::None)
     addBlockPlacement();
 
   addPreEmitPass();
 
+  // Verify basic block incoming and outgoing cfa offset and register values and
+  // correct CFA calculation rule where needed by inserting appropriate CFI
+  // instructions.
+  if (TM->maintainsCFAInfo())
+    addPass(createCFIInstrInserter(), false);
+
   if (TM->Options.EnableIPRA)
     // Collect register usage information and produce a register mask of
     // clobbered registers, to be used to optimize call sites.
     addPass(createRegUsageInfoCollector());
 
   addPass(&FuncletLayoutID, false);
 
   addPass(&StackMapLivenessID, false);

lib/Target/X86/X86FrameLowering.h

public:
 
   /// Sets up EBP and optionally ESI based on the incoming EBP value.  Only
   /// needed for 32-bit. Used in funclet prologues and at catchret destinations.
   MachineBasicBlock::iterator
   restoreWin32EHStackPointers(MachineBasicBlock &MBB,
                               MachineBasicBlock::iterator MBBI,
                               const DebugLoc &DL, bool RestoreSP = false) const;
 
+  int getInitialCFAOffset(MachineFunction &MF) const override;
+
+  unsigned getInitialCFARegister(MachineFunction &MF) const override;
+
 private:
   uint64_t calculateMaxStackAlign(const MachineFunction &MF) const;
 
   /// Emit target stack probe as a call to a helper function
   void emitStackProbeCall(MachineFunction &MF, MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MBBI, const DebugLoc &DL,
                           bool InProlog) const;
 

lib/Target/X86/X86FrameLowering.cpp

void X86FrameLowering::emitEpilogue(MachineFunction &MF,
   MachineBasicBlock *TargetMBB = nullptr;
 
   // Get the number of bytes to allocate from the FrameInfo.
   uint64_t StackSize = MFI.getStackSize();
   uint64_t MaxAlign = calculateMaxStackAlign(MF);
   unsigned CSSize = X86FI->getCalleeSavedFrameSize();
   uint64_t NumBytes = 0;
 
+  bool NeedsDwarfCFI =
+      MF.getTarget().maintainsCFAInfo() &&
+      (MF.getMMI().hasDebugInfo() || MF.getFunction()->needsUnwindTableEntry());
+
   if (RetOpcode && *RetOpcode == X86::CATCHRET) {
     // SEH shouldn't use catchret.
     assert(!isAsynchronousEHPersonality(
                classifyEHPersonality(MF.getFunction()->getPersonalityFn())) &&
            "SEH should not use CATCHRET");
 
     NumBytes = getWinEHFuncletFrameSize(MF);
     assert(hasFP(MF) && "EH funclets without FP not yet implemented");
if (RetOpcode && *RetOpcode == X86::CATCHRET) {
     // realigned.
     if (TRI->needsStackRealignment(MF) && !IsWin64Prologue)
       NumBytes = alignTo(FrameSize, MaxAlign);
 
     // Pop EBP.
     BuildMI(MBB, MBBI, DL,
             TII.get(Is64Bit ? X86::POP64r : X86::POP32r), MachineFramePtr)
         .setMIFlag(MachineInstr::FrameDestroy);
+    if (NeedsDwarfCFI) {
+      unsigned DwarfStackPtr =
+          TRI->getDwarfRegNum(Is64Bit ? X86::RSP : X86::ESP, true);
+      BuildCFI(MBB, MBBI, DL, MCCFIInstruction::createDefCfa(
+                                  nullptr, DwarfStackPtr, -SlotSize));
+      --MBBI;
+    }
   } else {
     NumBytes = StackSize - CSSize;
   }
   uint64_t SEHStackAllocAmt = NumBytes;
 
   MachineBasicBlock::iterator FirstCSPop = MBBI;
   // Skip the callee-saved pop instructions.
   while (MBBI != MBB.begin()) {
if (LEAAmount != 0) {
       unsigned Opc = (Uses64BitFramePtr ? X86::MOV64rr : X86::MOV32rr);
       BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
         .addReg(FramePtr);
       --MBBI;
     }
   } else if (NumBytes) {
     // Adjust stack pointer back: ESP += numbytes.
     emitSPUpdate(MBB, MBBI, NumBytes, /*InEpilogue=*/true);
+    if (!hasFP(MF) && NeedsDwarfCFI) {
+      // Define the current CFA rule to use the provided offset.
+      BuildCFI(MBB, MBBI, DL, MCCFIInstruction::createDefCfaOffset(
+                                  nullptr, -CSSize - SlotSize));
+    }
     --MBBI;
   }
 
   // Windows unwinder will not invoke function's exception handler if IP is
   // either in prologue or in epilogue.  This behavior causes a problem when a
   // call immediately precedes an epilogue, because the return address points
   // into the epilogue.  To cope with that, we insert an epilogue marker here,
   // then replace it with a 'nop' if it ends up immediately after a CALL in the
   // final emitted code.
   if (NeedsWinCFI && MF.hasWinCFI())
     BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_Epilogue));
 
+  if (!hasFP(MF) && NeedsDwarfCFI) {
+    MBBI = FirstCSPop;
+    int64_t Offset = -CSSize - SlotSize;
+    // Mark callee-saved pop instruction.
+    // Define the current CFA rule to use the provided offset.
+    while (MBBI != MBB.end()) {
+      MachineBasicBlock::iterator PI = MBBI;
+      unsigned Opc = PI->getOpcode();
+      ++MBBI;
+      if (Opc == X86::POP32r || Opc == X86::POP64r) {
+        Offset += SlotSize;
+        BuildCFI(MBB, MBBI, DL,
+                 MCCFIInstruction::createDefCfaOffset(nullptr, Offset));
+      }
+    }
+  }
+
   if (!RetOpcode || !isTailCallOpcode(*RetOpcode)) {
     // Add the return addr area delta back since we are not tail calling.
     int Offset = -1 * X86FI->getTCReturnAddrDelta();
     assert(Offset >= 0 && "TCDelta should never be positive");
     if (Offset) {
       MBBI = MBB.getFirstTerminator();
 
       // Check for possible merge with preceding ADD instruction.
addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV32rm), FramePtr),
                  UsedReg, true, Offset)
         .setMIFlag(MachineInstr::FrameSetup);
   } else {
     llvm_unreachable("32-bit frames with WinEH must use FramePtr or BasePtr");
   }
   return MBBI;
 }
 
+int X86FrameLowering::getInitialCFAOffset(MachineFunction &MF) const {
+  return TRI->getSlotSize();
+}
+
+unsigned X86FrameLowering::getInitialCFARegister(MachineFunction &MF) const {
+  return TRI->getDwarfRegNum(StackPtr, true);
+}
+
 namespace {
 // Struct used by orderFrameObjects to help sort the stack objects.
 struct X86FrameSortingObject {
   bool IsValid = false;         // true if we care about this Object.
   unsigned ObjectIndex = 0;     // Index of Object into MFI list.
   unsigned ObjectSize = 0;      // Size of Object in bytes.
   unsigned ObjectAlignment = 1; // Alignment of Object in bytes.
   unsigned ObjectNumUses = 0;   // Object static number of uses.

lib/Target/X86/X86TargetMachine.h

public:
 
   TargetLoweringObjectFile *getObjFileLowering() const override {
     return TLOF.get();
   }
 
   bool isMachineVerifierClean() const override {
     return false;
   }
+
+  bool maintainsCFAInfo() const override;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_X86_X86TARGETMACHINE_H

lib/Target/X86/X86TargetMachine.cpp

if (!I) {
     // function that reside in TargetOptions.
     resetTargetOptions(F);
     I = llvm::make_unique<X86Subtarget>(TargetTriple, CPU, FS, *this,
                                         Options.StackAlignmentOverride);
   }
   return I.get();
 }
 
+bool X86TargetMachine::maintainsCFAInfo() const {
+  return !getTargetTriple().isOSDarwin() && !getTargetTriple().isOSWindows();
+}
+
 //===----------------------------------------------------------------------===//
 // Command line options for x86
 //===----------------------------------------------------------------------===//
 static cl::opt<bool>
 UseVZeroUpper("x86-use-vzeroupper", cl::Hidden,
   cl::desc("Minimize AVX to SSE transition penalty"),
   cl::init(true));
 

test/CodeGen/X86/2009-03-16-PHIElimInLPad.ll

 lpad:   ; preds = %cont, %entry
   %v = phi i32 [ %x, %entry ], [ %a, %cont ]    ; <i32> [#uses=1]
         %exn = landingpad {i8*, i32}
                  cleanup
   ret i32 %v
 }
 
 ; CHECK: lpad
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset 16
 ; CHECK-NEXT: Ltmp
 
 declare i32 @__gxx_personality_v0(...)

test/CodeGen/X86/2011-10-19-widen_vselect.ll

 ; X32-NEXT:    movaps %xmm1, %xmm0
 ; X32-NEXT:    blendvps %xmm0, %xmm2, %xmm4
 ; X32-NEXT:    movss %xmm4, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movshdup {{.*#+}} xmm0 = xmm4[1,1,3,3]
 ; X32-NEXT:    movss %xmm0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movss %xmm4, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movss %xmm0, {{[0-9]+}}(%esp)
 ; X32-NEXT:    addl $60, %esp
+; X32-NEXT:  .Lcfi1:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: full_test:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    movsd {{.*#+}} xmm2 = mem[0],zero
 ; X64-NEXT:    cvttps2dq %xmm2, %xmm0
 ; X64-NEXT:    cvtdq2ps %xmm0, %xmm1
 ; X64-NEXT:    xorps %xmm0, %xmm0

test/CodeGen/X86/GlobalISel/add-scalar.ll

 ; X32-NEXT:    movl %esp, %ebp
 ; X32-NEXT:  .Lcfi2:
 ; X32-NEXT:    .cfi_def_cfa_register %ebp
 ; X32-NEXT:    movl 16(%ebp), %eax
 ; X32-NEXT:    movl 20(%ebp), %edx
 ; X32-NEXT:    addl 8(%ebp), %eax
 ; X32-NEXT:    adcl 12(%ebp), %edx
 ; X32-NEXT:    popl %ebp
+; X32-NEXT:  .Lcfi3:
+; X32-NEXT:    .cfi_def_cfa %esp, 4
 ; X32-NEXT:    retl
   %ret = add i64 %arg1, %arg2
   ret i64 %ret
 }
 
 define i32 @test_add_i32(i32 %arg1, i32 %arg2) {
 ; X64-LABEL: test_add_i32:
 ; X64:       # BB#0:

test/CodeGen/X86/GlobalISel/brcond.ll

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    jmp .LBB0_3
 ; X32-NEXT:  .LBB0_2: # %if.else
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:  .LBB0_3: # %return
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    movl (%esp), %eax
 ; X32-NEXT:    popl %ecx
+; X32-NEXT:  .Lcfi1:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 entry:
   %retval = alloca i32, align 4
   %cmp = icmp slt i32 %a, %b
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:
   store i32 %tValue, i32* %retval, align 4

test/CodeGen/X86/GlobalISel/callingconv.ll

 ; X32-LABEL: test_v8i32_args:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
 ; X32-NEXT:  .Lcfi0:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    movups 16(%esp), %xmm1
 ; X32-NEXT:    movaps %xmm2, %xmm0
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi1:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_v8i32_args:
 ; X64:       # BB#0:
 ; X64-NEXT:    movaps %xmm2, %xmm0
 ; X64-NEXT:    movaps %xmm3, %xmm1
 ; X64-NEXT:    retq
   ret <8 x i32> %arg2
 }
 
 declare void @trivial_callee()
 define void @test_trivial_call() {
 ; X32-LABEL: test_trivial_call:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi1:
+; X32-NEXT:  .Lcfi2:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    calll trivial_callee
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi3:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_trivial_call:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
 ; X64-NEXT:  .Lcfi0:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    callq trivial_callee
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi1:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   call void @trivial_callee()
   ret void
 }
 
 declare void @simple_arg_callee(i32 %in0, i32 %in1)
 define void @test_simple_arg_call(i32 %in0, i32 %in1) {
 ; X32-LABEL: test_simple_arg_call:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi2:
+; X32-NEXT:  .Lcfi4:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    movl 16(%esp), %eax
 ; X32-NEXT:    movl 20(%esp), %ecx
 ; X32-NEXT:    movl %ecx, (%esp)
 ; X32-NEXT:    movl %eax, 4(%esp)
 ; X32-NEXT:    calll simple_arg_callee
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi5:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_simple_arg_call:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
-; X64-NEXT:  .Lcfi1:
+; X64-NEXT:  .Lcfi2:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    movl %edi, %eax
 ; X64-NEXT:    movl %esi, %edi
 ; X64-NEXT:    movl %eax, %esi
 ; X64-NEXT:    callq simple_arg_callee
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi3:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   call void @simple_arg_callee(i32 %in1, i32 %in0)
   ret void
 }
 
 declare void @simple_arg8_callee(i32 %arg1, i32 %arg2, i32 %arg3, i32 %arg4, i32 %arg5, i32 %arg6, i32 %arg7, i32 %arg8)
 define void @test_simple_arg8_call(i32 %in0) {
 ; X32-LABEL: test_simple_arg8_call:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $44, %esp
-; X32-NEXT:  .Lcfi3:
+; X32-NEXT:  .Lcfi6:
 ; X32-NEXT:    .cfi_def_cfa_offset 48
 ; X32-NEXT:    movl 48(%esp), %eax
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    movl %eax, 4(%esp)
 ; X32-NEXT:    movl %eax, 8(%esp)
 ; X32-NEXT:    movl %eax, 12(%esp)
 ; X32-NEXT:    movl %eax, 16(%esp)
 ; X32-NEXT:    movl %eax, 20(%esp)
 ; X32-NEXT:    movl %eax, 24(%esp)
 ; X32-NEXT:    movl %eax, 28(%esp)
 ; X32-NEXT:    calll simple_arg8_callee
 ; X32-NEXT:    addl $44, %esp
+; X32-NEXT:  .Lcfi7:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_simple_arg8_call:
 ; X64:       # BB#0:
 ; X64-NEXT:    subq $24, %rsp
-; X64-NEXT:  .Lcfi2:
+; X64-NEXT:  .Lcfi4:
 ; X64-NEXT:    .cfi_def_cfa_offset 32
 ; X64-NEXT:    movl %edi, (%rsp)
 ; X64-NEXT:    movl %edi, 8(%rsp)
 ; X64-NEXT:    movl %edi, %esi
 ; X64-NEXT:    movl %edi, %edx
 ; X64-NEXT:    movl %edi, %ecx
 ; X64-NEXT:    movl %edi, %r8d
 ; X64-NEXT:    movl %edi, %r9d
 ; X64-NEXT:    callq simple_arg8_callee
 ; X64-NEXT:    addq $24, %rsp
+; X64-NEXT:  .Lcfi5:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   call void @simple_arg8_callee(i32 %in0, i32 %in0, i32 %in0, i32 %in0,i32 %in0, i32 %in0, i32 %in0, i32 %in0)
   ret void
 }
 
 declare i32 @simple_return_callee(i32 %in0)
 define i32 @test_simple_return_callee() {
 ; X32-LABEL: test_simple_return_callee:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi4:
+; X32-NEXT:  .Lcfi8:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    movl $5, %eax
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    calll simple_return_callee
 ; X32-NEXT:    addl %eax, %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi9:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_simple_return_callee:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
-; X64-NEXT:  .Lcfi3:
+; X64-NEXT:  .Lcfi6:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    movl $5, %edi
 ; X64-NEXT:    callq simple_return_callee
 ; X64-NEXT:    addl %eax, %eax
 ; X64-NEXT:    popq %rcx
+; X64-NEXT:  .Lcfi7:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   %call = call i32 @simple_return_callee(i32 5)
   %r = add i32 %call, %call
   ret i32 %r
 }
 
 declare <8 x i32> @split_return_callee(<8 x i32> %in0)
 define <8 x i32> @test_split_return_callee(<8 x i32> %arg1, <8 x i32> %arg2) {
 ; X32-LABEL: test_split_return_callee:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $44, %esp
-; X32-NEXT:  .Lcfi5:
+; X32-NEXT:  .Lcfi10:
 ; X32-NEXT:    .cfi_def_cfa_offset 48
 ; X32-NEXT:    movaps %xmm0, (%esp) # 16-byte Spill
 ; X32-NEXT:    movaps %xmm1, 16(%esp) # 16-byte Spill
 ; X32-NEXT:    movdqu 48(%esp), %xmm1
 ; X32-NEXT:    movdqa %xmm2, %xmm0
 ; X32-NEXT:    calll split_return_callee
 ; X32-NEXT:    paddd (%esp), %xmm0 # 16-byte Folded Reload
 ; X32-NEXT:    paddd 16(%esp), %xmm1 # 16-byte Folded Reload
 ; X32-NEXT:    addl $44, %esp
+; X32-NEXT:  .Lcfi11:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_split_return_callee:
 ; X64:       # BB#0:
 ; X64-NEXT:    subq $40, %rsp
-; X64-NEXT:  .Lcfi4:
+; X64-NEXT:  .Lcfi8:
 ; X64-NEXT:    .cfi_def_cfa_offset 48
 ; X64-NEXT:    movaps %xmm0, (%rsp) # 16-byte Spill
 ; X64-NEXT:    movaps %xmm1, 16(%rsp) # 16-byte Spill
 ; X64-NEXT:    movdqa %xmm2, %xmm0
 ; X64-NEXT:    movdqa %xmm3, %xmm1
 ; X64-NEXT:    callq split_return_callee
 ; X64-NEXT:    paddd (%rsp), %xmm0 # 16-byte Folded Reload
 ; X64-NEXT:    paddd 16(%rsp), %xmm1 # 16-byte Folded Reload
 ; X64-NEXT:    addq $40, %rsp
+; X64-NEXT:  .Lcfi9:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   %call = call <8 x i32> @split_return_callee(<8 x i32> %arg2)
   %r = add <8 x i32> %arg1, %call
   ret  <8 x i32> %r
 }
 
 define void @test_indirect_call(void()* %func) {
 ; X32-LABEL: test_indirect_call:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi6:
+; X32-NEXT:  .Lcfi12:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    calll *16(%esp)
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi13:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_indirect_call:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
-; X64-NEXT:  .Lcfi5:
+; X64-NEXT:  .Lcfi10:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    callq *%rdi
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi11:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   call void %func()
   ret void
 }
 
 declare void @take_char(i8)
 define void @test_abi_exts_call(i8* %addr) {
 ; X32-LABEL: test_abi_exts_call:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %ebx
-; X32-NEXT:  .Lcfi7:
+; X32-NEXT:  .Lcfi14:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    subl $8, %esp
-; X32-NEXT:  .Lcfi8:
+; X32-NEXT:  .Lcfi15:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
-; X32-NEXT:  .Lcfi9:
+; X32-NEXT:  .Lcfi16:
 ; X32-NEXT:    .cfi_offset %ebx, -8
 ; X32-NEXT:    movl 16(%esp), %eax
 ; X32-NEXT:    movb (%eax), %bl
 ; X32-NEXT:    movb %bl, (%esp)
 ; X32-NEXT:    calll take_char
 ; X32-NEXT:    movsbl %bl, %eax
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    calll take_char
 ; X32-NEXT:    movzbl %bl, %eax
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    calll take_char
 ; X32-NEXT:    addl $8, %esp
+; X32-NEXT:  .Lcfi17:
+; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    popl %ebx
+; X32-NEXT:  .Lcfi18:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_abi_exts_call:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rbx
-; X64-NEXT:  .Lcfi6:
+; X64-NEXT:  .Lcfi12:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
-; X64-NEXT:  .Lcfi7:
+; X64-NEXT:  .Lcfi13:
 ; X64-NEXT:    .cfi_offset %rbx, -16
 ; X64-NEXT:    movb (%rdi), %bl
 ; X64-NEXT:    movl %ebx, %edi
 ; X64-NEXT:    callq take_char
 ; X64-NEXT:    movsbl %bl, %ebx
 ; X64-NEXT:    movl %ebx, %edi
 ; X64-NEXT:    callq take_char
 ; X64-NEXT:    movzbl %bl, %edi
 ; X64-NEXT:    callq take_char
 ; X64-NEXT:    popq %rbx
+; X64-NEXT:  .Lcfi14:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
   %val = load i8, i8* %addr
   call void @take_char(i8 %val)
   call void @take_char(i8 signext %val)
   call void @take_char(i8 zeroext %val)
  ret void
 }
 
 declare void @variadic_callee(i8*, ...)
 define void @test_variadic_call_1(i8** %addr_ptr, i32* %val_ptr) {
 ; X32-LABEL: test_variadic_call_1:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi10:
+; X32-NEXT:  .Lcfi19:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    movl 16(%esp), %eax
 ; X32-NEXT:    movl 20(%esp), %ecx
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    movl (%ecx), %ecx
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    movl %ecx, 4(%esp)
 ; X32-NEXT:    calll variadic_callee
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi20:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_variadic_call_1:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
-; X64-NEXT:  .Lcfi8:
+; X64-NEXT:  .Lcfi15:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    movq (%rdi), %rdi
 ; X64-NEXT:    movl (%rsi), %esi
 ; X64-NEXT:    movb $0, %al
 ; X64-NEXT:    callq variadic_callee
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi16:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 
   %addr = load i8*, i8** %addr_ptr
   %val = load i32, i32* %val_ptr
   call void (i8*, ...) @variadic_callee(i8* %addr, i32 %val)
   ret void
 }
 
 define void @test_variadic_call_2(i8** %addr_ptr, double* %val_ptr) {
 ; X32-LABEL: test_variadic_call_2:
 ; X32:       # BB#0:
 ; X32-NEXT:    subl $12, %esp
-; X32-NEXT:  .Lcfi11:
+; X32-NEXT:  .Lcfi21:
 ; X32-NEXT:    .cfi_def_cfa_offset 16
 ; X32-NEXT:    movl 16(%esp), %eax
 ; X32-NEXT:    movl 20(%esp), %ecx
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    movl (%ecx), %edx
 ; X32-NEXT:    movl 4(%ecx), %ecx
 ; X32-NEXT:    movl %eax, (%esp)
 ; X32-NEXT:    movl $4, %eax
 ; X32-NEXT:    leal (%esp,%eax), %eax
 ; X32-NEXT:    movl %edx, 4(%esp)
 ; X32-NEXT:    movl %ecx, 4(%eax)
 ; X32-NEXT:    calll variadic_callee
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:  .Lcfi22:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_variadic_call_2:
 ; X64:       # BB#0:
 ; X64-NEXT:    pushq %rax
-; X64-NEXT:  .Lcfi9:
+; X64-NEXT:  .Lcfi17:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    movq (%rdi), %rdi
 ; X64-NEXT:    movq (%rsi), %rcx
 ; X64-NEXT:    movb $1, %al
 ; X64-NEXT:    movq %rcx, %xmm0
 ; X64-NEXT:    callq variadic_callee
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi18:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 
   %addr = load i8*, i8** %addr_ptr
   %val = load double, double* %val_ptr
   call void (i8*, ...) @variadic_callee(i8* %addr, double %val)
   ret void
 }

test/CodeGen/X86/GlobalISel/frameIndex.ll

 ;
 ; X32-LABEL: allocai32:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
 ; X32-NEXT:  .Lcfi0:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movl %esp, %eax
 ; X32-NEXT:    popl %ecx
+; X32-NEXT:  .Lcfi1:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X32ABI-LABEL: allocai32:
 ; X32ABI:       # BB#0:
 ; X32ABI-NEXT:    leal -4(%rsp), %eax
 ; X32ABI-NEXT:    retq
   %ptr1 = alloca i32
   ret i32* %ptr1
 }

test/CodeGen/X86/TruncAssertZext.ll

 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:    movabsq $-4294967041, %rcx # imm = 0xFFFFFFFF000000FF
 ; CHECK-NEXT:    andq %rax, %rcx
 ; CHECK-NEXT:    movl %ecx, %ecx
 ; CHECK-NEXT:    leaq (,%rcx,8), %rax
 ; CHECK-NEXT:    subq %rcx, %rax
 ; CHECK-NEXT:    shrq $32, %rax
 ; CHECK-NEXT:    popq %rcx
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %b = call i64 @foo()
   %or = and i64 %b, 18446744069414584575 ; this is 0xffffffff000000ff
   %trunc = trunc i64 %or to i32
   br label %l
 l:
   %ext = zext i32 %trunc to i64
   %mul = mul i64 %ext, 7
   br label %m
 m: ; keeps dag combine from seeing the multiply and the shift together
   %shr = lshr i64 %mul, 32
   trunc i64 %or to i32 ; keeps the and alive so it doesn't simplify
   ret i64 %shr
 }

test/CodeGen/X86/avx512-schedule.ll

 ; CHECK-NEXT:  .Lcfi0:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    vaddss %xmm0, %xmm0, %xmm0 # sched: [4:0.50]
 ; CHECK-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill sched: [1:1.00]
 ; CHECK-NEXT:    # sched: [1:1.00]
 ; CHECK-NEXT:    callq func_f32
 ; CHECK-NEXT:    vbroadcastss (%rsp), %zmm0 # 16-byte Folded Reload sched: [5:?]
 ; CHECK-NEXT:    addq $24, %rsp # sched: [1:0.25]
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq # sched: [2:1.00]
   %a  = fadd float %x, %x
   call void @func_f32(float %a)
   %b = insertelement <16 x float> undef, float %a, i32 0
   %c = shufflevector <16 x float> %b, <16 x float> undef, <16 x i32> zeroinitializer
   ret <16 x float> %c
 }
 
 declare void @func_f64(double)
 define <8 x double> @broadcast_sd_spill(double %x) {
 ; CHECK-LABEL: broadcast_sd_spill:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    subq $24, %rsp # sched: [1:0.25]
-; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:  .Lcfi2:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    vaddsd %xmm0, %xmm0, %xmm0 # sched: [4:0.50]
 ; CHECK-NEXT:    vmovapd %xmm0, (%rsp) # 16-byte Spill sched: [1:1.00]
 ; CHECK-NEXT:    # sched: [1:1.00]
 ; CHECK-NEXT:    callq func_f64
 ; CHECK-NEXT:    vbroadcastsd (%rsp), %zmm0 # 16-byte Folded Reload sched: [5:?]
 ; CHECK-NEXT:    addq $24, %rsp # sched: [1:0.25]
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq # sched: [2:1.00]
   %a  = fadd double %x, %x
   call void @func_f64(double %a)
   %b = insertelement <8 x double> undef, double %a, i32 0
   %c = shufflevector <8 x double> %b, <8 x double> undef, <8 x i32> zeroinitializer
   ret <8 x double> %c
 }

test/CodeGen/X86/avx512-select.ll

 ; X86-NEXT:    movl %esp, %ebp
 ; X86-NEXT:  .Lcfi2:
 ; X86-NEXT:    .cfi_def_cfa_register %ebp
 ; X86-NEXT:    andl $-64, %esp
 ; X86-NEXT:    subl $64, %esp
 ; X86-NEXT:    vmovaps 8(%ebp), %zmm1
 ; X86-NEXT:    movl %ebp, %esp
 ; X86-NEXT:    popl %ebp
+; X86-NEXT:  .Lcfi3:
+; X86-NEXT:     .cfi_def_cfa %esp, 4
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: select04:
 ; X64:       # BB#0:
 ; X64-NEXT:    vmovaps %zmm3, %zmm1
 ; X64-NEXT:    retq
   %sel = select <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 false, i1 false, i1 false, i1 false, i1 false, i1 false, i1 false, i1 false>, <16 x double> %a, <16 x double> %b
   ret <16 x double> %sel

test/CodeGen/X86/avx512-vbroadcast.ll

 ; ALL-NEXT:    subq $24, %rsp
 ; ALL-NEXT:  .Lcfi0:
 ; ALL-NEXT:    .cfi_def_cfa_offset 32
 ; ALL-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; ALL-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
 ; ALL-NEXT:    callq func_f32
 ; ALL-NEXT:    vbroadcastss (%rsp), %zmm0 # 16-byte Folded Reload
 ; ALL-NEXT:    addq $24, %rsp
+; ALL-NEXT:  .Lcfi1:
+; ALL-NEXT:    .cfi_def_cfa_offset 8
 ; ALL-NEXT:    retq
   %a  = fadd float %x, %x
   call void @func_f32(float %a)
   %b = insertelement <16 x float> undef, float %a, i32 0
   %c = shufflevector <16 x float> %b, <16 x float> undef, <16 x i32> zeroinitializer
   ret <16 x float> %c
 }
 
 declare void @func_f64(double)
 define <8 x double> @broadcast_sd_spill(double %x) {
 ; ALL-LABEL: broadcast_sd_spill:
 ; ALL:       # BB#0:
 ; ALL-NEXT:    subq $24, %rsp
-; ALL-NEXT:  .Lcfi1:
+; ALL-NEXT:  .Lcfi2:
 ; ALL-NEXT:    .cfi_def_cfa_offset 32
 ; ALL-NEXT:    vaddsd %xmm0, %xmm0, %xmm0
 ; ALL-NEXT:    vmovapd %xmm0, (%rsp) # 16-byte Spill
 ; ALL-NEXT:    callq func_f64
 ; ALL-NEXT:    vbroadcastsd (%rsp), %zmm0 # 16-byte Folded Reload
 ; ALL-NEXT:    addq $24, %rsp
+; ALL-NEXT:  .Lcfi3:
+; ALL-NEXT:    .cfi_def_cfa_offset 8
 ; ALL-NEXT:    retq
   %a  = fadd double %x, %x
   call void @func_f64(double %a)
   %b = insertelement <8 x double> undef, double %a, i32 0
   %c = shufflevector <8 x double> %b, <8 x double> undef, <8 x i32> zeroinitializer
   ret <8 x double> %c
 }

test/CodeGen/X86/avx512bw-intrinsics-fast-isel.ll

 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    kshiftlq $1, %k0, %k0
 ; X32-NEXT:    kshiftrq $1, %k0, %k0
 ; X32-NEXT:    kshiftlq $63, %k1, %k1
 ; X32-NEXT:    korq %k1, %k0, %k1
 ; X32-NEXT:    vpbroadcastb %eax, %zmm3 {%k1}
 ; X32-NEXT:    vmovdqa64 %zmm3, %zmm0
 ; X32-NEXT:    popl %ebx
+; X32-NEXT:  .Lcfi2:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovq %rdi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %zmm0 {%k1}
 ; X64-NEXT:    retq
   entry:
   %vecinit.i.i = insertelement <64 x i8> undef, i8 %__A, i32 0
   %vecinit63.i.i = shufflevector <64 x i8> %vecinit.i.i, <64 x i8> undef, <64 x i32> zeroinitializer
   %0 = bitcast <8 x i64> %__O to <64 x i8>
   %1 = bitcast i64 %__M to <64 x i1>
   %2 = select <64 x i1> %1, <64 x i8> %vecinit63.i.i, <64 x i8> %0
   %3 = bitcast <64 x i8> %2 to <8 x i64>
   ret <8 x i64> %3
 }
 
 define <8 x i64> @test_mm512_maskz_set1_epi8(i64 %__M, i8 signext %__A)  {
 ; X32-LABEL: test_mm512_maskz_set1_epi8:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    pushl %ebx
-; X32-NEXT:  .Lcfi2:
-; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:  .Lcfi3:
+; X32-NEXT:    .cfi_def_cfa_offset 8
+; X32-NEXT:  .Lcfi4:
 ; X32-NEXT:    .cfi_offset %ebx, -8
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, %ecx
 ; X32-NEXT:    andb $2, %cl
 ; X32-NEXT:    shrb %cl
 ; X32-NEXT:    kmovd %ecx, %k0
 ; X32-NEXT:    vpmovm2b %k0, %zmm0
 ; X32-NEXT:    vpsllw $8, %xmm0, %xmm0

 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    vpmovb2m %zmm0, %k1
 ; X32-NEXT:    kshiftlq $1, %k1, %k1
 ; X32-NEXT:    kshiftrq $1, %k1, %k1
 ; X32-NEXT:    kshiftlq $63, %k0, %k0
 ; X32-NEXT:    korq %k0, %k1, %k1
 ; X32-NEXT:    vpbroadcastb %eax, %zmm0 {%k1} {z}
 ; X32-NEXT:    popl %ebx
+; X32-NEXT:  .Lcfi5:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovq %rdi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq
   entry:

test/CodeGen/X86/avx512bw-intrinsics-upgrade.ll

 ; AVX512F-32-NEXT:    subl $12, %esp
 ; AVX512F-32-NEXT:  .Lcfi0:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi1:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.mask.pcmpeq.b.512(<64 x i8> %a, <64 x i8> %b, i64 -1)
   ret i64 %res
 }
 
 define i64 @test_mask_pcmpeq_b(<64 x i8> %a, <64 x i8> %b, i64 %mask) {
 ; AVX512BW-LABEL: test_mask_pcmpeq_b:
 ; AVX512BW:       ## BB#0:
 ; AVX512BW-NEXT:    kmovq %rdi, %k1
 ; AVX512BW-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_mask_pcmpeq_b:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $12, %esp
-; AVX512F-32-NEXT:  .Lcfi1:
+; AVX512F-32-NEXT:  .Lcfi2:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    kmovq {{[0-9]+}}(%esp), %k1
 ; AVX512F-32-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi3:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.mask.pcmpeq.b.512(<64 x i8> %a, <64 x i8> %b, i64 %mask)
   ret i64 %res
 }
 
 declare i64 @llvm.x86.avx512.mask.pcmpeq.b.512(<64 x i8>, <64 x i8>, i64)
 

 ; AVX512BW-NEXT:    vpcmpgtb %zmm1, %zmm0, %k0
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_pcmpgt_b:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $12, %esp
-; AVX512F-32-NEXT:  .Lcfi2:
+; AVX512F-32-NEXT:  .Lcfi4:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    vpcmpgtb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi5:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.mask.pcmpgt.b.512(<64 x i8> %a, <64 x i8> %b, i64 -1)
   ret i64 %res
 }
 
 define i64 @test_mask_pcmpgt_b(<64 x i8> %a, <64 x i8> %b, i64 %mask) {
 ; AVX512BW-LABEL: test_mask_pcmpgt_b:
 ; AVX512BW:       ## BB#0:
 ; AVX512BW-NEXT:    kmovq %rdi, %k1
 ; AVX512BW-NEXT:    vpcmpgtb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_mask_pcmpgt_b:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $12, %esp
-; AVX512F-32-NEXT:  .Lcfi3:
+; AVX512F-32-NEXT:  .Lcfi6:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    kmovq {{[0-9]+}}(%esp), %k1
 ; AVX512F-32-NEXT:    vpcmpgtb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi7:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.mask.pcmpgt.b.512(<64 x i8> %a, <64 x i8> %b, i64 %mask)
   ret i64 %res
 }
 
 declare i64 @llvm.x86.avx512.mask.pcmpgt.b.512(<64 x i8>, <64 x i8>, i64)
 

 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_cmp_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $60, %esp
-; AVX512F-32-NEXT:  .Lcfi4:
+; AVX512F-32-NEXT:  .Lcfi8:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 64
 ; AVX512F-32-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    vpcmpgtb %zmm0, %zmm1, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax

 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    addl (%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    kxnorq %k0, %k0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $60, %esp
+; AVX512F-32-NEXT:  .Lcfi9:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res0 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 0, i64 -1)
   %res1 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 1, i64 -1)
   %ret1 = add i64 %res0, %res1
   %res2 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 2, i64 -1)
   %ret2 = add i64 %ret1, %res2
   %res3 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 3, i64 -1)

 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    addq %rdi, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_mask_cmp_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    pushl %ebx
-; AVX512F-32-NEXT:  .Lcfi5:
+; AVX512F-32-NEXT:  .Lcfi10:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 8
 ; AVX512F-32-NEXT:    pushl %esi
-; AVX512F-32-NEXT:  .Lcfi6:
+; AVX512F-32-NEXT:  .Lcfi11:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 12
 ; AVX512F-32-NEXT:    subl $60, %esp
-; AVX512F-32-NEXT:  .Lcfi7:
+; AVX512F-32-NEXT:  .Lcfi12:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 72
-; AVX512F-32-NEXT:  .Lcfi8:
+; AVX512F-32-NEXT:  .Lcfi13:
 ; AVX512F-32-NEXT:    .cfi_offset %esi, -12
-; AVX512F-32-NEXT:  .Lcfi9:
+; AVX512F-32-NEXT:  .Lcfi14:
 ; AVX512F-32-NEXT:    .cfi_offset %ebx, -8
 ; AVX512F-32-NEXT:    vmovdqa64 %zmm1, %zmm6
 ; AVX512F-32-NEXT:    vmovdqa64 %zmm0, %zmm5
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; AVX512F-32-NEXT:    movl %ecx, %eax
 ; AVX512F-32-NEXT:    shrb $5, %al
 ; AVX512F-32-NEXT:    andb $1, %al
 ; AVX512F-32-NEXT:    movl %ecx, %ebx

 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    kmovq %k1, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl %esi, %eax
 ; AVX512F-32-NEXT:    adcl %ecx, %edx
 ; AVX512F-32-NEXT:    addl $60, %esp
+; AVX512F-32-NEXT:  .Lcfi15:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 12
 ; AVX512F-32-NEXT:    popl %esi
+; AVX512F-32-NEXT:  .Lcfi16:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 8
 ; AVX512F-32-NEXT:    popl %ebx
+; AVX512F-32-NEXT:  .Lcfi17:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res0 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 0, i64 %mask)
   %res1 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 1, i64 %mask)
   %ret1 = add i64 %res0, %res1
   %res2 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 2, i64 %mask)
   %ret2 = add i64 %ret1, %res2
   %res3 = call i64 @llvm.x86.avx512.mask.cmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 3, i64 %mask)

 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_ucmp_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $60, %esp
-; AVX512F-32-NEXT:  .Lcfi10:
+; AVX512F-32-NEXT:  .Lcfi18:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 64
 ; AVX512F-32-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    vpcmpltub %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax

 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    addl (%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    kxnorq %k0, %k0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $60, %esp
+; AVX512F-32-NEXT:  .Lcfi19:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res0 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 0, i64 -1)
   %res1 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 1, i64 -1)
   %ret1 = add i64 %res0, %res1
   %res2 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 2, i64 -1)
   %ret2 = add i64 %ret1, %res2
   %res3 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 3, i64 -1)

 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    addq %rdi, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_mask_x86_avx512_ucmp_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    pushl %ebx
-; AVX512F-32-NEXT:  .Lcfi11:
+; AVX512F-32-NEXT:  .Lcfi20:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 8
 ; AVX512F-32-NEXT:    pushl %esi
-; AVX512F-32-NEXT:  .Lcfi12:
+; AVX512F-32-NEXT:  .Lcfi21:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 12
 ; AVX512F-32-NEXT:    subl $60, %esp
-; AVX512F-32-NEXT:  .Lcfi13:
+; AVX512F-32-NEXT:  .Lcfi22:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 72
-; AVX512F-32-NEXT:  .Lcfi14:
+; AVX512F-32-NEXT:  .Lcfi23:
 ; AVX512F-32-NEXT:    .cfi_offset %esi, -12
-; AVX512F-32-NEXT:  .Lcfi15:
+; AVX512F-32-NEXT:  .Lcfi24:
 ; AVX512F-32-NEXT:    .cfi_offset %ebx, -8
 ; AVX512F-32-NEXT:    vmovdqa64 %zmm1, %zmm6
 ; AVX512F-32-NEXT:    vmovdqa64 %zmm0, %zmm5
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; AVX512F-32-NEXT:    movl %ecx, %eax
 ; AVX512F-32-NEXT:    shrb $5, %al
 ; AVX512F-32-NEXT:    andb $1, %al
 ; AVX512F-32-NEXT:    movl %ecx, %ebx

 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    kmovq %k1, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl %esi, %eax
 ; AVX512F-32-NEXT:    adcl %ecx, %edx
 ; AVX512F-32-NEXT:    addl $60, %esp
+; AVX512F-32-NEXT:  .Lcfi25:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 12
 ; AVX512F-32-NEXT:    popl %esi
+; AVX512F-32-NEXT:  .Lcfi26:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 8
 ; AVX512F-32-NEXT:    popl %ebx
+; AVX512F-32-NEXT:  .Lcfi27:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    vzeroupper
 ; AVX512F-32-NEXT:    retl
   %res0 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 0, i64 %mask)
   %res1 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 1, i64 %mask)
   %ret1 = add i64 %res0, %res1
   %res2 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 2, i64 %mask)
   %ret2 = add i64 %ret1, %res2
   %res3 = call i64 @llvm.x86.avx512.mask.ucmp.b.512(<64 x i8> %a0, <64 x i8> %a1, i32 3, i64 %mask)

test/CodeGen/X86/avx512bw-intrinsics.ll

 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k0
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; AVX512F-32-NEXT:    kunpckdq %k0, %k1, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi1:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.kunpck.dq(i64 %x0, i64 %x1)
   ret i64 %res
 }
 
 declare i64 @llvm.x86.avx512.cvtb2mask.512(<64 x i8>)
 
 define i64@test_int_x86_avx512_cvtb2mask_512(<64 x i8> %x0) {
 ; AVX512BW-LABEL: test_int_x86_avx512_cvtb2mask_512:
 ; AVX512BW:       ## BB#0:
 ; AVX512BW-NEXT:    vpmovb2m %zmm0, %k0
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_int_x86_avx512_cvtb2mask_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $12, %esp
-; AVX512F-32-NEXT:  .Lcfi1:
+; AVX512F-32-NEXT:  .Lcfi2:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-32-NEXT:    vpmovb2m %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $12, %esp
+; AVX512F-32-NEXT:  .Lcfi3:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    retl
     %res = call i64 @llvm.x86.avx512.cvtb2mask.512(<64 x i8> %x0)
     ret i64 %res
 }
 
 declare i32 @llvm.x86.avx512.cvtw2mask.512(<32 x i16>)
 
 define i32@test_int_x86_avx512_cvtw2mask_512(<32 x i16> %x0) {

 ; AVX512BW-NEXT:    vptestmb %zmm1, %zmm0, %k0
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_int_x86_avx512_ptestm_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $20, %esp
-; AVX512F-32-NEXT:  .Lcfi2:
+; AVX512F-32-NEXT:  .Lcfi4:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 24
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k0
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; AVX512F-32-NEXT:    kunpckdq %k0, %k1, %k1
 ; AVX512F-32-NEXT:    vptestmb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    vptestmb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcxl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $20, %esp
+; AVX512F-32-NEXT:  .Lcfi5:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.ptestm.b.512(<64 x i8> %x0, <64 x i8> %x1, i64 %x2)
   %res1 = call i64 @llvm.x86.avx512.ptestm.b.512(<64 x i8> %x0, <64 x i8> %x1, i64-1)
   %res2 = add i64 %res, %res1
   ret i64 %res2
 }
 
 declare i32 @llvm.x86.avx512.ptestm.w.512(<32 x i16>, <32 x i16>, i32)

 ; AVX512BW-NEXT:    vptestnmb %zmm1, %zmm0, %k0
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    addq %rcx, %rax
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512F-32-LABEL: test_int_x86_avx512_ptestnm_b_512:
 ; AVX512F-32:       # BB#0:
 ; AVX512F-32-NEXT:    subl $20, %esp
-; AVX512F-32-NEXT:  .Lcfi3:
+; AVX512F-32-NEXT:  .Lcfi6:
 ; AVX512F-32-NEXT:    .cfi_def_cfa_offset 24
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k0
 ; AVX512F-32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; AVX512F-32-NEXT:    kunpckdq %k0, %k1, %k1
 ; AVX512F-32-NEXT:    vptestnmb %zmm1, %zmm0, %k0 {%k1}
 ; AVX512F-32-NEXT:    kmovq %k0, (%esp)
 ; AVX512F-32-NEXT:    vptestnmb %zmm1, %zmm0, %k0
 ; AVX512F-32-NEXT:    kmovq %k0, {{[0-9]+}}(%esp)
 ; AVX512F-32-NEXT:    movl (%esp), %eax
 ; AVX512F-32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; AVX512F-32-NEXT:    adcxl {{[0-9]+}}(%esp), %edx
 ; AVX512F-32-NEXT:    addl $20, %esp
+; AVX512F-32-NEXT:  .Lcfi7:
+; AVX512F-32-NEXT:    .cfi_def_cfa_offset 4
 ; AVX512F-32-NEXT:    retl
   %res = call i64 @llvm.x86.avx512.ptestnm.b.512(<64 x i8> %x0, <64 x i8> %x1, i64 %x2)
   %res1 = call i64 @llvm.x86.avx512.ptestnm.b.512(<64 x i8> %x0, <64 x i8> %x1, i64-1)
   %res2 = add i64 %res, %res1
   ret i64 %res2
 }
 
 declare i32 @llvm.x86.avx512.ptestnm.w.512(<32 x i16>, <32 x i16>, i32 %x2)

test/CodeGen/X86/avx512vl-intrinsics-fast-isel.ll

 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastd %xmm1, %xmm0 {%k1}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi1:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_broadcastd_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vpbroadcastd %xmm1, %xmm0 {%k1}
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg0 = bitcast <2 x i64> %a0 to <4 x i32>
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <4 x i32>
   %res0 = shufflevector <4 x i32> %arg2, <4 x i32> undef, <4 x i32> zeroinitializer
   %res1 = select <4 x i1> %arg1, <4 x i32> %res0, <4 x i32> %arg0
   %res2 = bitcast <4 x i32> %res1 to <2 x i64>
   ret <2 x i64> %res2
 }
 
 define <2 x i64> @test_mm_maskz_broadcastd_epi32(i8 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm_maskz_broadcastd_epi32:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi1:
+; X32-NEXT:  .Lcfi2:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastd %xmm0, %xmm0 {%k1} {z}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi3:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_broadcastd_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x i64> %a0, <2 x i64> undef, <2 x i32> zeroinitializer
   ret <2 x i64> %res
 }
 
 define <2 x i64> @test_mm_mask_broadcastq_epi64(<2 x i64> %a0, i8 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm_mask_broadcastq_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi2:
+; X32-NEXT:  .Lcfi4:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastq %xmm1, %xmm0 {%k1}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi5:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_broadcastq_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vpbroadcastq %xmm1, %xmm0 {%k1}
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i2
   %arg1 = bitcast i2 %trn1 to <2 x i1>
   %res0 = shufflevector <2 x i64> %a2, <2 x i64> undef, <2 x i32> zeroinitializer
   %res1 = select <2 x i1> %arg1, <2 x i64> %res0, <2 x i64> %a0
   ret <2 x i64> %res1
 }
 
 define <2 x i64> @test_mm_maskz_broadcastq_epi64(i8 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm_maskz_broadcastq_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi3:
+; X32-NEXT:  .Lcfi6:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastq %xmm0, %xmm0 {%k1} {z}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi7:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_broadcastq_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x i64> %a0, <2 x i64> undef, <4 x i32> zeroinitializer
   ret <4 x i64> %res
 }
 
 define <4 x i64> @test_mm256_mask_broadcastq_epi64(<4 x i64> %a0, i8 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm256_mask_broadcastq_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi4:
+; X32-NEXT:  .Lcfi8:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastq %xmm1, %ymm0 {%k1}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi9:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_broadcastq_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vpbroadcastq %xmm1, %ymm0 {%k1}
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <2 x i64> %a2, <2 x i64> undef, <4 x i32> zeroinitializer
   %res1 = select <4 x i1> %arg1, <4 x i64> %res0, <4 x i64> %a0
   ret <4 x i64> %res1
 }
 
 define <4 x i64> @test_mm256_maskz_broadcastq_epi64(i8 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm256_maskz_broadcastq_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi5:
+; X32-NEXT:  .Lcfi10:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpbroadcastq %xmm0, %ymm0 {%k1} {z}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi11:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_broadcastq_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x double> %a0, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %res
 }
 
 define <2 x double> @test_mm_mask_broadcastsd_pd(<2 x double> %a0, i8 %a1, <2 x double> %a2) {
 ; X32-LABEL: test_mm_mask_broadcastsd_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi6:
+; X32-NEXT:  .Lcfi12:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} = xmm1[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi13:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_broadcastsd_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} = xmm1[0,0]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i2
   %arg1 = bitcast i2 %trn1 to <2 x i1>
   %res0 = shufflevector <2 x double> %a2, <2 x double> undef, <2 x i32> zeroinitializer
   %res1 = select <2 x i1> %arg1, <2 x double> %res0, <2 x double> %a0
   ret <2 x double> %res1
 }
 
 define <2 x double> @test_mm_maskz_broadcastsd_pd(i8 %a0, <2 x double> %a1) {
 ; X32-LABEL: test_mm_maskz_broadcastsd_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi7:
+; X32-NEXT:  .Lcfi14:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} {z} = xmm0[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi15:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_broadcastsd_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x double> %a0, <2 x double> undef, <4 x i32> zeroinitializer
   ret <4 x double> %res
 }
 
 define <4 x double> @test_mm256_mask_broadcastsd_pd(<4 x double> %a0, i8 %a1, <2 x double> %a2) {
 ; X32-LABEL: test_mm256_mask_broadcastsd_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi8:
+; X32-NEXT:  .Lcfi16:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vbroadcastsd %xmm1, %ymm0 {%k1}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi17:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_broadcastsd_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vbroadcastsd %xmm1, %ymm0 {%k1}
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <2 x double> %a2, <2 x double> undef, <4 x i32> zeroinitializer
   %res1 = select <4 x i1> %arg1, <4 x double> %res0, <4 x double> %a0
   ret <4 x double> %res1
 }
 
 define <4 x double> @test_mm256_maskz_broadcastsd_pd(i8 %a0, <2 x double> %a1) {
 ; X32-LABEL: test_mm256_maskz_broadcastsd_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi9:
+; X32-NEXT:  .Lcfi18:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vbroadcastsd %xmm0, %ymm0 {%k1} {z}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi19:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_broadcastsd_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x float> %a0, <4 x float> undef, <4 x i32> zeroinitializer
   ret <4 x float> %res
 }
 
 define <4 x float> @test_mm_mask_broadcastss_ps(<4 x float> %a0, i8 %a1, <4 x float> %a2) {
 ; X32-LABEL: test_mm_mask_broadcastss_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi10:
+; X32-NEXT:  .Lcfi20:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vbroadcastss %xmm1, %xmm0 {%k1}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi21:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_broadcastss_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vbroadcastss %xmm1, %xmm0 {%k1}
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x float> %a2, <4 x float> undef, <4 x i32> zeroinitializer
   %res1 = select <4 x i1> %arg1, <4 x float> %res0, <4 x float> %a0
   ret <4 x float> %res1
 }
 
 define <4 x float> @test_mm_maskz_broadcastss_ps(i8 %a0, <4 x float> %a1) {
 ; X32-LABEL: test_mm_maskz_broadcastss_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi11:
+; X32-NEXT:  .Lcfi22:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vbroadcastss %xmm0, %xmm0 {%k1} {z}
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi23:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_broadcastss_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x double> %a0, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %res
 }
 
 define <2 x double> @test_mm_mask_movddup_pd(<2 x double> %a0, i8 %a1, <2 x double> %a2) {
 ; X32-LABEL: test_mm_mask_movddup_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi12:
+; X32-NEXT:  .Lcfi24:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} = xmm1[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi25:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_movddup_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} = xmm1[0,0]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i2
   %arg1 = bitcast i2 %trn1 to <2 x i1>
   %res0 = shufflevector <2 x double> %a2, <2 x double> undef, <2 x i32> zeroinitializer
   %res1 = select <2 x i1> %arg1, <2 x double> %res0, <2 x double> %a0
   ret <2 x double> %res1
 }
 
 define <2 x double> @test_mm_maskz_movddup_pd(i8 %a0, <2 x double> %a1) {
 ; X32-LABEL: test_mm_maskz_movddup_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi13:
+; X32-NEXT:  .Lcfi26:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} xmm0 {%k1} {z} = xmm0[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi27:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_movddup_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x double> %a0, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
   ret <4 x double> %res
 }
 
 define <4 x double> @test_mm256_mask_movddup_pd(<4 x double> %a0, i8 %a1, <4 x double> %a2) {
 ; X32-LABEL: test_mm256_mask_movddup_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi14:
+; X32-NEXT:  .Lcfi28:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} ymm0 {%k1} = ymm1[0,0,2,2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi29:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_movddup_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vmovddup {{.*#+}} ymm0 {%k1} = ymm1[0,0,2,2]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x double> %a2, <4 x double> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
   %res1 = select <4 x i1> %arg1, <4 x double> %res0, <4 x double> %a0
   ret <4 x double> %res1
 }
 
 define <4 x double> @test_mm256_maskz_movddup_pd(i8 %a0, <4 x double> %a1) {
 ; X32-LABEL: test_mm256_maskz_movddup_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi15:
+; X32-NEXT:  .Lcfi30:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovddup {{.*#+}} ymm0 {%k1} {z} = ymm0[0,0,2,2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi31:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_movddup_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x float> %a0, <4 x float> undef, <4 x i32> <i32 1, i32 1, i32 3, i32 3>
   ret <4 x float> %res
 }
 
 define <4 x float> @test_mm_mask_movehdup_ps(<4 x float> %a0, i8 %a1, <4 x float> %a2) {
 ; X32-LABEL: test_mm_mask_movehdup_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi16:
+; X32-NEXT:  .Lcfi32:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovshdup {{.*#+}} xmm0 {%k1} = xmm1[1,1,3,3]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi33:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_movehdup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vmovshdup {{.*#+}} xmm0 {%k1} = xmm1[1,1,3,3]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x float> %a2, <4 x float> undef, <4 x i32> <i32 1, i32 1, i32 3, i32 3>
   %res1 = select <4 x i1> %arg1, <4 x float> %res0, <4 x float> %a0
   ret <4 x float> %res1
 }
 
 define <4 x float> @test_mm_maskz_movehdup_ps(i8 %a0, <4 x float> %a1) {
 ; X32-LABEL: test_mm_maskz_movehdup_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi17:
+; X32-NEXT:  .Lcfi34:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovshdup {{.*#+}} xmm0 {%k1} {z} = xmm0[1,1,3,3]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi35:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_movehdup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x float> %a0, <4 x float> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
   ret <4 x float> %res
 }
 
 define <4 x float> @test_mm_mask_moveldup_ps(<4 x float> %a0, i8 %a1, <4 x float> %a2) {
 ; X32-LABEL: test_mm_mask_moveldup_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi18:
+; X32-NEXT:  .Lcfi36:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovsldup {{.*#+}} xmm0 {%k1} = xmm1[0,0,2,2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi37:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_moveldup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vmovsldup {{.*#+}} xmm0 {%k1} = xmm1[0,0,2,2]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x float> %a2, <4 x float> undef, <4 x i32> <i32 0, i32 0, i32 2, i32 2>
   %res1 = select <4 x i1> %arg1, <4 x float> %res0, <4 x float> %a0
   ret <4 x float> %res1
 }
 
 define <4 x float> @test_mm_maskz_moveldup_ps(i8 %a0, <4 x float> %a1) {
 ; X32-LABEL: test_mm_maskz_moveldup_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi19:
+; X32-NEXT:  .Lcfi38:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vmovsldup {{.*#+}} xmm0 {%k1} {z} = xmm0[0,0,2,2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi39:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_moveldup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x i64> %a0, <4 x i64> undef, <4 x i32> <i32 3, i32 0, i32 0, i32 0>
   ret <4 x i64> %res
 }
 
 define <4 x i64> @test_mm256_mask_permutex_epi64(<4 x i64> %a0, i8 %a1, <4 x i64> %a2) {
 ; X32-LABEL: test_mm256_mask_permutex_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi20:
+; X32-NEXT:  .Lcfi40:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpermq {{.*#+}} ymm0 {%k1} = ymm1[1,0,0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi41:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_permutex_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vpermq {{.*#+}} ymm0 {%k1} = ymm1[1,0,0,0]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x i64> %a2, <4 x i64> undef, <4 x i32> <i32 1, i32 0, i32 0, i32 0>
   %res1 = select <4 x i1> %arg1, <4 x i64> %res0, <4 x i64> %a0
   ret <4 x i64> %res1
 }
 
 define <4 x i64> @test_mm256_maskz_permutex_epi64(i8 %a0, <4 x i64> %a1) {
 ; X32-LABEL: test_mm256_maskz_permutex_epi64:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi21:
+; X32-NEXT:  .Lcfi42:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpermq {{.*#+}} ymm0 {%k1} {z} = ymm0[1,0,0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi43:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_permutex_epi64:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x double> %a0, <4 x double> undef, <4 x i32> <i32 3, i32 0, i32 0, i32 0>
   ret <4 x double> %res
 }
 
 define <4 x double> @test_mm256_mask_permutex_pd(<4 x double> %a0, i8 %a1, <4 x double> %a2) {
 ; X32-LABEL: test_mm256_mask_permutex_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi22:
+; X32-NEXT:  .Lcfi44:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpermpd {{.*#+}} ymm0 {%k1} = ymm1[1,0,0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi45:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_permutex_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vpermpd {{.*#+}} ymm0 {%k1} = ymm1[1,0,0,0]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x double> %a2, <4 x double> undef, <4 x i32> <i32 1, i32 0, i32 0, i32 0>
   %res1 = select <4 x i1> %arg1, <4 x double> %res0, <4 x double> %a0
   ret <4 x double> %res1
 }
 
 define <4 x double> @test_mm256_maskz_permutex_pd(i8 %a0, <4 x double> %a1) {
 ; X32-LABEL: test_mm256_maskz_permutex_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi23:
+; X32-NEXT:  .Lcfi46:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vpermpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1,0,0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi47:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_permutex_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <2 x double> %a0, <2 x double> %a1, <2 x i32> <i32 1, i32 3>
   ret <2 x double> %res
 }
 
 define <2 x double> @test_mm_mask_shuffle_pd(<2 x double> %a0, i8 %a1, <2 x double> %a2, <2 x double> %a3) {
 ; X32-LABEL: test_mm_mask_shuffle_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi24:
+; X32-NEXT:  .Lcfi48:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} = xmm1[1],xmm2[1]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi49:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_shuffle_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} = xmm1[1],xmm2[1]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i2
   %arg1 = bitcast i2 %trn1 to <2 x i1>
   %res0 = shufflevector <2 x double> %a2, <2 x double> %a3, <2 x i32> <i32 1, i32 3>
   %res1 = select <2 x i1> %arg1, <2 x double> %res0, <2 x double> %a0
   ret <2 x double> %res1
 }
 
 define <2 x double> @test_mm_maskz_shuffle_pd(i8 %a0, <2 x double> %a1, <2 x double> %a2) {
 ; X32-LABEL: test_mm_maskz_shuffle_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi25:
+; X32-NEXT:  .Lcfi50:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $3, %al
 ; X32-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; X32-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vunpckhpd {{.*#+}} xmm0 {%k1} {z} = xmm0[1],xmm1[1]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi51:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_shuffle_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $3, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x double> %a0, <4 x double> %a1, <4 x i32> <i32 1, i32 5, i32 2, i32 6>
   ret <4 x double> %res
 }
 
 define <4 x double> @test_mm256_mask_shuffle_pd(<4 x double> %a0, i8 %a1, <4 x double> %a2, <4 x double> %a3) {
 ; X32-LABEL: test_mm256_mask_shuffle_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi26:
+; X32-NEXT:  .Lcfi52:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vshufpd {{.*#+}} ymm0 {%k1} = ymm1[1],ymm2[1],ymm1[2],ymm2[2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi53:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_shuffle_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vshufpd {{.*#+}} ymm0 {%k1} = ymm1[1],ymm2[1],ymm1[2],ymm2[2]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x double> %a2, <4 x double> %a3, <4 x i32> <i32 1, i32 5, i32 2, i32 6>
   %res1 = select <4 x i1> %arg1, <4 x double> %res0, <4 x double> %a0
   ret <4 x double> %res1
 }
 
 define <4 x double> @test_mm256_maskz_shuffle_pd(i8 %a0, <4 x double> %a1, <4 x double> %a2) {
 ; X32-LABEL: test_mm256_maskz_shuffle_pd:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi27:
+; X32-NEXT:  .Lcfi54:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vshufpd {{.*#+}} ymm0 {%k1} {z} = ymm0[1],ymm1[1],ymm0[2],ymm1[2]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi55:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_shuffle_pd:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
; X64-NEXT:    retq
   %res = shufflevector <4 x float> %a0, <4 x float> %a1, <4 x i32> <i32 0, i32 1, i32 4, i32 4>
   ret <4 x float> %res
 }
 
 define <4 x float> @test_mm_mask_shuffle_ps(<4 x float> %a0, i8 %a1, <4 x float> %a2, <4 x float> %a3) {
 ; X32-LABEL: test_mm_mask_shuffle_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi28:
+; X32-NEXT:  .Lcfi56:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vshufps {{.*#+}} xmm0 {%k1} = xmm1[0,1],xmm2[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi57:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_shuffle_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1
 ; X64-NEXT:    vshufps {{.*#+}} xmm0 {%k1} = xmm1[0,1],xmm2[0,0]
 ; X64-NEXT:    retq
   %trn1 = trunc i8 %a1 to i4
   %arg1 = bitcast i4 %trn1 to <4 x i1>
   %res0 = shufflevector <4 x float> %a2, <4 x float> %a3, <4 x i32> <i32 0, i32 1, i32 4, i32 4>
   %res1 = select <4 x i1> %arg1, <4 x float> %res0, <4 x float> %a0
   ret <4 x float> %res1
 }
 
 define <4 x float> @test_mm_maskz_shuffle_ps(i8 %a0, <4 x float> %a1, <4 x float> %a2) {
 ; X32-LABEL: test_mm_maskz_shuffle_ps:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %eax
-; X32-NEXT:  .Lcfi29:
+; X32-NEXT:  .Lcfi58:
 ; X32-NEXT:    .cfi_def_cfa_offset 8
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X32-NEXT:    andb $15, %al
 ; X32-NEXT:    movb %al, (%esp)
 ; X32-NEXT:    movzbl (%esp), %eax
 ; X32-NEXT:    kmovw %eax, %k1
 ; X32-NEXT:    vshufps {{.*#+}} xmm0 {%k1} {z} = xmm0[0,1],xmm1[0,0]
 ; X32-NEXT:    popl %eax
+; X32-NEXT:  .Lcfi59:
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_shuffle_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    andb $15, %dil
 ; X64-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
 ; X64-NEXT:    movzbl -{{[0-9]+}}(%rsp), %eax
 ; X64-NEXT:    kmovw %eax, %k1

test/CodeGen/X86/avx512vl-vbroadcast.ll

 ; NOTE: Assertions have been autogenerated by update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512f -mattr=+avx512vl| FileCheck %s
 
 declare void @func_f32(float)
 define <8 x float> @_256_broadcast_ss_spill(float %x) {
 ; CHECK-LABEL: _256_broadcast_ss_spill:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    subq $24, %rsp
 ; CHECK-NEXT:  .Lcfi0:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
 ; CHECK-NEXT:    callq func_f32
 ; CHECK-NEXT:    vbroadcastss (%rsp), %ymm0 # 16-byte Folded Reload
 ; CHECK-NEXT:    addq $24, %rsp
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %a  = fadd float %x, %x
   call void @func_f32(float %a)
   %b = insertelement <8 x float> undef, float %a, i32 0
   %c = shufflevector <8 x float> %b, <8 x float> undef, <8 x i32> zeroinitializer
   ret <8 x float> %c
 }
 
 define <4 x float> @_128_broadcast_ss_spill(float %x) {
 ; CHECK-LABEL: _128_broadcast_ss_spill:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    subq $24, %rsp
-; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:  .Lcfi2:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    vaddss %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vmovaps %xmm0, (%rsp) # 16-byte Spill
 ; CHECK-NEXT:    callq func_f32
 ; CHECK-NEXT:    vbroadcastss (%rsp), %xmm0 # 16-byte Folded Reload
 ; CHECK-NEXT:    addq $24, %rsp
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %a  = fadd float %x, %x
   call void @func_f32(float %a)
   %b = insertelement <4 x float> undef, float %a, i32 0
   %c = shufflevector <4 x float> %b, <4 x float> undef, <4 x i32> zeroinitializer
   ret <4 x float> %c
 }
 
 declare void @func_f64(double)
 define <4 x double> @_256_broadcast_sd_spill(double %x) {
 ; CHECK-LABEL: _256_broadcast_sd_spill:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    subq $24, %rsp
-; CHECK-NEXT:  .Lcfi2:
+; CHECK-NEXT:  .Lcfi4:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 32
 ; CHECK-NEXT:    vaddsd %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    vmovapd %xmm0, (%rsp) # 16-byte Spill
 ; CHECK-NEXT:    callq func_f64
 ; CHECK-NEXT:    vbroadcastsd (%rsp), %ymm0 # 16-byte Folded Reload
 ; CHECK-NEXT:    addq $24, %rsp
+; CHECK-NEXT:  .Lcfi5:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %a  = fadd double %x, %x
   call void @func_f64(double %a)
   %b = insertelement <4 x double> undef, double %a, i32 0
   %c = shufflevector <4 x double> %b, <4 x double> undef, <4 x i32> zeroinitializer
   ret <4 x double> %c
 }
 

test/CodeGen/X86/bitcast-and-setcc-512.ll

 ; AVX512F-NEXT:    kmovw %k0, {{[0-9]+}}(%rsp)
 ; AVX512F-NEXT:    vpmovsxbd %xmm0, %zmm0
 ; AVX512F-NEXT:    vpslld $31, %zmm0, %zmm0
 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; AVX512F-NEXT:    kmovw %k0, (%rsp)
 ; AVX512F-NEXT:    movl (%rsp), %eax
 ; AVX512F-NEXT:    movq %rbp, %rsp
 ; AVX512F-NEXT:    popq %rbp
+; AVX512F-NEXT:  .Lcfi3:
+; AVX512F-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512BW-LABEL: v32i16:
 ; AVX512BW:       # BB#0:
 ; AVX512BW-NEXT:    vpcmpgtw %zmm1, %zmm0, %k1
 ; AVX512BW-NEXT:    vpcmpgtw %zmm3, %zmm2, %k0 {%k1}
 ; AVX512BW-NEXT:    kmovd %k0, %eax

 ; AVX1-NEXT:    andb $1, %al
 ; AVX1-NEXT:    movb %al, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    movl (%rsp), %ecx
 ; AVX1-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX1-NEXT:    shlq $32, %rax
 ; AVX1-NEXT:    orq %rcx, %rax
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
+; AVX1-NEXT:  .Lcfi3:
+; AVX1-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: v64i8:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:  .Lcfi0:
 ; AVX2-NEXT:    .cfi_def_cfa_offset 16

 ; AVX2-NEXT:    andb $1, %al
 ; AVX2-NEXT:    movb %al, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    movl (%rsp), %ecx
 ; AVX2-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX2-NEXT:    shlq $32, %rax
 ; AVX2-NEXT:    orq %rcx, %rax
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
+; AVX2-NEXT:  .Lcfi3:
+; AVX2-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: v64i8:
 ; AVX512F:       # BB#0:
 ; AVX512F-NEXT:    pushq %rbp
-; AVX512F-NEXT:  .Lcfi3:
-; AVX512F-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-NEXT:  .Lcfi4:
+; AVX512F-NEXT:    .cfi_def_cfa_offset 16
+; AVX512F-NEXT:  .Lcfi5:
 ; AVX512F-NEXT:    .cfi_offset %rbp, -16
 ; AVX512F-NEXT:    movq %rsp, %rbp
-; AVX512F-NEXT:  .Lcfi5:
+; AVX512F-NEXT:  .Lcfi6:
 ; AVX512F-NEXT:    .cfi_def_cfa_register %rbp
 ; AVX512F-NEXT:    andq $-32, %rsp
 ; AVX512F-NEXT:    subq $64, %rsp
 ; AVX512F-NEXT:    vpcmpgtb %ymm3, %ymm1, %ymm1
 ; AVX512F-NEXT:    vpcmpgtb %ymm2, %ymm0, %ymm0
 ; AVX512F-NEXT:    vpcmpgtb %ymm7, %ymm5, %ymm2
 ; AVX512F-NEXT:    vpand %ymm2, %ymm1, %ymm1
 ; AVX512F-NEXT:    vpcmpgtb %ymm6, %ymm4, %ymm2

 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; AVX512F-NEXT:    kmovw %k0, {{[0-9]+}}(%rsp)
 ; AVX512F-NEXT:    movl (%rsp), %ecx
 ; AVX512F-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX512F-NEXT:    shlq $32, %rax
 ; AVX512F-NEXT:    orq %rcx, %rax
 ; AVX512F-NEXT:    movq %rbp, %rsp
 ; AVX512F-NEXT:    popq %rbp
+; AVX512F-NEXT:  .Lcfi7:
+; AVX512F-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512BW-LABEL: v64i8:
 ; AVX512BW:       # BB#0:
 ; AVX512BW-NEXT:    vpcmpgtb %zmm1, %zmm0, %k1
 ; AVX512BW-NEXT:    vpcmpgtb %zmm3, %zmm2, %k0 {%k1}
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
   %x0 = icmp sgt <64 x i8> %a, %b
   %x1 = icmp sgt <64 x i8> %c, %d
   %y = and <64 x i1> %x0, %x1
   %res = bitcast <64 x i1> %y to i64
   ret i64 %res
 }

test/CodeGen/X86/bitcast-int-to-vector-bool-zext.ll

 ; AVX512-NEXT:    vpinsrb $10, %ebp, %xmm0, %xmm0
 ; AVX512-NEXT:    vpinsrb $11, %edi, %xmm0, %xmm0
 ; AVX512-NEXT:    vpinsrb $12, %eax, %xmm0, %xmm0
 ; AVX512-NEXT:    vpinsrb $13, %ecx, %xmm0, %xmm0
 ; AVX512-NEXT:    vpinsrb $14, %edx, %xmm0, %xmm0
 ; AVX512-NEXT:    vpinsrb $15, %r9d, %xmm0, %xmm0
 ; AVX512-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
 ; AVX512-NEXT:    popq %rbx
+; AVX512-NEXT:  .Lcfi12:
+; AVX512-NEXT:    .cfi_def_cfa_offset 48
 ; AVX512-NEXT:    popq %r12
+; AVX512-NEXT:  .Lcfi13:
+; AVX512-NEXT:    .cfi_def_cfa_offset 40
 ; AVX512-NEXT:    popq %r13
+; AVX512-NEXT:  .Lcfi14:
+; AVX512-NEXT:    .cfi_def_cfa_offset 32
 ; AVX512-NEXT:    popq %r14
+; AVX512-NEXT:  .Lcfi15:
+; AVX512-NEXT:    .cfi_def_cfa_offset 24
 ; AVX512-NEXT:    popq %r15
+; AVX512-NEXT:  .Lcfi16:
+; AVX512-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512-NEXT:    popq %rbp
+; AVX512-NEXT:  .Lcfi17:
+; AVX512-NEXT:    .cfi_def_cfa_offset 8
 ; AVX512-NEXT:    retq
   %1 = bitcast i16 %a0 to <16 x i1>
   %2 = zext <16 x i1> %1 to <16 x i8>
   ret <16 x i8> %2
 }
 
 ;
 ; 256-bit vectors

test/CodeGen/X86/bitcast-setcc-512.ll

 ; AVX512F-NEXT:    vpinsrb $15, %eax, %xmm0, %xmm0
 ; AVX512F-NEXT:    vpmovsxbd %xmm0, %zmm0
 ; AVX512F-NEXT:    vpslld $31, %zmm0, %zmm0
 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; AVX512F-NEXT:    kmovw %k0, (%rsp)
 ; AVX512F-NEXT:    movl (%rsp), %eax
 ; AVX512F-NEXT:    movq %rbp, %rsp
 ; AVX512F-NEXT:    popq %rbp
+; AVX512F-NEXT:  .Lcfi3:
+; AVX512F-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512BW-LABEL: v32i16:
 ; AVX512BW:       # BB#0:
 ; AVX512BW-NEXT:    vpcmpgtw %zmm1, %zmm0, %k0
 ; AVX512BW-NEXT:    kmovd %k0, %eax
 ; AVX512BW-NEXT:    vzeroupper

 ; AVX1-NEXT:    andb $1, %al
 ; AVX1-NEXT:    movb %al, {{[0-9]+}}(%rsp)
 ; AVX1-NEXT:    movl (%rsp), %ecx
 ; AVX1-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX1-NEXT:    shlq $32, %rax
 ; AVX1-NEXT:    orq %rcx, %rax
 ; AVX1-NEXT:    movq %rbp, %rsp
 ; AVX1-NEXT:    popq %rbp
+; AVX1-NEXT:  .Lcfi3:
+; AVX1-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: v64i8:
 ; AVX2:       # BB#0:
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:  .Lcfi0:
 ; AVX2-NEXT:    .cfi_def_cfa_offset 16

 ; AVX2-NEXT:    andb $1, %al
 ; AVX2-NEXT:    movb %al, {{[0-9]+}}(%rsp)
 ; AVX2-NEXT:    movl (%rsp), %ecx
 ; AVX2-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX2-NEXT:    shlq $32, %rax
 ; AVX2-NEXT:    orq %rcx, %rax
 ; AVX2-NEXT:    movq %rbp, %rsp
 ; AVX2-NEXT:    popq %rbp
+; AVX2-NEXT:  .Lcfi3:
+; AVX2-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: v64i8:
 ; AVX512F:       # BB#0:
 ; AVX512F-NEXT:    pushq %rbp
-; AVX512F-NEXT:  .Lcfi3:
-; AVX512F-NEXT:    .cfi_def_cfa_offset 16
 ; AVX512F-NEXT:  .Lcfi4:
+; AVX512F-NEXT:    .cfi_def_cfa_offset 16
+; AVX512F-NEXT:  .Lcfi5:
 ; AVX512F-NEXT:    .cfi_offset %rbp, -16
 ; AVX512F-NEXT:    movq %rsp, %rbp
-; AVX512F-NEXT:  .Lcfi5:
+; AVX512F-NEXT:  .Lcfi6:
 ; AVX512F-NEXT:    .cfi_def_cfa_register %rbp
 ; AVX512F-NEXT:    andq $-32, %rsp
 ; AVX512F-NEXT:    subq $64, %rsp
 ; AVX512F-NEXT:    vpcmpgtb %ymm2, %ymm0, %ymm0
 ; AVX512F-NEXT:    vextracti128 $1, %ymm0, %xmm2
 ; AVX512F-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; AVX512F-NEXT:    vpslld $31, %zmm2, %zmm2
 ; AVX512F-NEXT:    vptestmd %zmm2, %zmm2, %k0

 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; AVX512F-NEXT:    kmovw %k0, {{[0-9]+}}(%rsp)
 ; AVX512F-NEXT:    movl (%rsp), %ecx
 ; AVX512F-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; AVX512F-NEXT:    shlq $32, %rax
 ; AVX512F-NEXT:    orq %rcx, %rax
 ; AVX512F-NEXT:    movq %rbp, %rsp
 ; AVX512F-NEXT:    popq %rbp
+; AVX512F-NEXT:  .Lcfi7:
+; AVX512F-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; AVX512BW-LABEL: v64i8:
 ; AVX512BW:       # BB#0:
 ; AVX512BW-NEXT:    vpcmpgtb %zmm1, %zmm0, %k0
 ; AVX512BW-NEXT:    kmovq %k0, %rax
 ; AVX512BW-NEXT:    vzeroupper

test/CodeGen/X86/emutls-pie.ll

 
 define i32 @my_get_xyz() {
 ; X32-LABEL: my_get_xyz:
 ; X32:      movl my_emutls_v_xyz@GOT(%ebx), %eax
 ; X32-NEXT: movl %eax, (%esp)
 ; X32-NEXT: calll my_emutls_get_address@PLT
 ; X32-NEXT: movl (%eax), %eax
 ; X32-NEXT: addl $8, %esp
+; X32-NEXT: :
+; X32-NEXT: .cfi_def_cfa_offset 8
 ; X32-NEXT: popl %ebx
+; X32-NEXT: :
+; X32-NEXT: .cfi_def_cfa_offset 4
 ; X32-NEXT: retl
 ; X64-LABEL: my_get_xyz:
 ; X64:      movq my_emutls_v_xyz@GOTPCREL(%rip), %rdi
 ; X64-NEXT: callq my_emutls_get_address@PLT
 ; X64-NEXT: movl (%rax), %eax
 ; X64-NEXT: popq %rcx
+; X64-NEXT: :
+; X64-NEXT: .cfi_def_cfa_offset 8
 ; X64-NEXT: retq
 
 entry:
   %call = call i8* @my_emutls_get_address(i8* bitcast (i8** @my_emutls_v_xyz to i8*))
   %0 = bitcast i8* %call to i32*
   %1 = load i32, i32* %0, align 4
   ret i32 %1
 }
 
 @i = thread_local global i32 15
 @i2 = external thread_local global i32
 
 define i32 @f1() {
 ; X32-LABEL: f1:
 ; X32:      leal __emutls_v.i@GOTOFF(%ebx), %eax
 ; X32-NEXT: movl %eax, (%esp)
 ; X32-NEXT: calll __emutls_get_address@PLT
 ; X32-NEXT: movl (%eax), %eax
 ; X32-NEXT: addl $8, %esp
+; X32-NEXT: :
+; X32-NEXT: .cfi_def_cfa_offset 8
 ; X32-NEXT: popl %ebx
+; X32-NEXT: :
+; X32-NEXT: .cfi_def_cfa_offset 4
 ; X32-NEXT: retl
 ; X64-LABEL: f1:
 ; X64:      leaq __emutls_v.i(%rip), %rdi
 ; X64-NEXT: callq __emutls_get_address@PLT
 ; X64-NEXT: movl (%rax), %eax
 ; X64-NEXT: popq %rcx
+; X64-NEXT: :
+; X64-NEXT: .cfi_def_cfa_offset 8
 ; X64-NEXT: retq
 
 entry:
   %tmp1 = load i32, i32* @i
   ret i32 %tmp1
 }
 
 define i32* @f2() {

test/CodeGen/X86/emutls.ll

 declare i8* @my_emutls_get_address(i8*)
 
 define i32 @my_get_xyz() {
 ; X32-LABEL: my_get_xyz:
 ; X32:         movl $my_emutls_v_xyz, (%esp)
 ; X32-NEXT:    calll my_emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ; X64-LABEL: my_get_xyz:
 ; X64:         movl $my_emutls_v_xyz, %edi
 ; X64-NEXT:    callq my_emutls_get_address
 ; X64-NEXT:    movl (%rax), %eax
 ; X64-NEXT:    popq %rcx
+; X64-NEXT:    :
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 
 entry:
   %call = call i8* @my_emutls_get_address(i8* bitcast (i8** @my_emutls_v_xyz to i8*))
   %0 = bitcast i8* %call to i32*
   %1 = load i32, i32* %0, align 4
   ret i32 %1
 }
 
 @i1 = thread_local global i32 15
 @i2 = external thread_local global i32
 @i3 = internal thread_local global i32 15
 @i4 = hidden thread_local global i32 15
 @i5 = external hidden thread_local global i32
 @s1 = thread_local global i16 15
 @b1 = thread_local global i8 0
 
 define i32 @f1() {
 ; X32-LABEL: f1:
 ; X32:         movl $__emutls_v.i1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ; X64-LABEL: f1:
 ; X64:         movl $__emutls_v.i1, %edi
 ; X64-NEXT:    callq __emutls_get_address
 ; X64-NEXT:    movl (%rax), %eax
 ; X64-NEXT:    popq %rcx
+; X64-NEXT:    :
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 
 entry:
   %tmp1 = load i32, i32* @i1
   ret i32 %tmp1
 }
 
 define i32* @f2() {
 ; X32-LABEL: f2:
 ; X32:         movl $__emutls_v.i1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 ; X64-LABEL: f2:
 ; X64:         movl $__emutls_v.i1, %edi
 ; X64-NEXT:    callq __emutls_get_address
 ; X64-NEXT:    popq %rcx
+; X64-NEXT:    :
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 
 entry:
   ret i32* @i1
 }
 
 define i32 @f3() nounwind {
 ; X32-LABEL: f3:
 ; X32:         movl $__emutls_v.i2, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i32, i32* @i2
   ret i32 %tmp1
 }
 
 define i32* @f4() {
 ; X32-LABEL: f4:
 ; X32:         movl $__emutls_v.i2, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   ret i32* @i2
 }
 
 define i32 @f5() nounwind {
 ; X32-LABEL: f5:
 ; X32:         movl $__emutls_v.i3, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i32, i32* @i3
   ret i32 %tmp1
 }
 
 define i32* @f6() {
 ; X32-LABEL: f6:
 ; X32:         movl $__emutls_v.i3, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   ret i32* @i3
 }
 
 define i32 @f7() {
 ; X32-LABEL: f7:
 ; X32:         movl $__emutls_v.i4, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i32, i32* @i4
   ret i32 %tmp1
 }
 
 define i32* @f8() {
 ; X32-LABEL: f8:
 ; X32:         movl $__emutls_v.i4, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   ret i32* @i4
 }
 
 define i32 @f9() {
 ; X32-LABEL: f9:
 ; X32:         movl $__emutls_v.i5, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i32, i32* @i5
   ret i32 %tmp1
 }
 
 define i32* @f10() {
 ; X32-LABEL: f10:
 ; X32:         movl $__emutls_v.i5, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   ret i32* @i5
 }
 
 define i16 @f11() {
 ; X32-LABEL: f11:
 ; X32:         movl $__emutls_v.s1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movzwl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i16, i16* @s1
   ret i16 %tmp1
 }
 
 define i32 @f12() {
 ; X32-LABEL: f12:
 ; X32:         movl $__emutls_v.s1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movswl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i16, i16* @s1
   %tmp2 = sext i16 %tmp1 to i32
   ret i32 %tmp2
 }
 
 define i8 @f13() {
 ; X32-LABEL: f13:
 ; X32:         movl $__emutls_v.b1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movb (%eax), %al
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i8, i8* @b1
   ret i8 %tmp1
 }
 
 define i32 @f14() {
 ; X32-LABEL: f14:
 ; X32:         movl $__emutls_v.b1, (%esp)
 ; X32-NEXT:    calll __emutls_get_address
 ; X32-NEXT:    movsbl (%eax), %eax
 ; X32-NEXT:    addl $12, %esp
+; X32-NEXT:    :
+; X32-NEXT:    .cfi_def_cfa_offset 4
 ; X32-NEXT:    retl
 
 entry:
   %tmp1 = load i8, i8* @b1
   %tmp2 = sext i8 %tmp1 to i32
   ret i32 %tmp2
 }
 

test/CodeGen/X86/epilogue-cfi-fp.ll

+; RUN: llc -O0 %s -o - | FileCheck %s
+
+; ModuleID = 'epilogue-cfi-fp.c'
+source_filename = "epilogue-cfi-fp.c"
+target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
+target triple = "i686-pc-linux"
+
+; Function Attrs: noinline nounwind
+define i32 @foo(i32 %i, i32 %j, i32 %k, i32 %l, i32 %m) #0 {
+
+; CHECK-LABEL:   foo:
+; CHECK:         popl %ebp
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa %esp, 4
+; CHECK-NEXT:    retl
+
+entry:
+  %i.addr = alloca i32, align 4
+  %j.addr = alloca i32, align 4
+  %k.addr = alloca i32, align 4
+  %l.addr = alloca i32, align 4
+  %m.addr = alloca i32, align 4
+  store i32 %i, i32* %i.addr, align 4
+  store i32 %j, i32* %j.addr, align 4
+  store i32 %k, i32* %k.addr, align 4
+  store i32 %l, i32* %l.addr, align 4
+  store i32 %m, i32* %m.addr, align 4
+  ret i32 0
+}
+
+attributes #0 = { "no-frame-pointer-elim"="true" }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6, !7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 5.0.0 (http://llvm.org/git/clang.git 3f8116e6a2815b1d5f3491493938d0c63c9f42c9) (http://llvm.org/git/llvm.git 4fde77f8f1a8e4482e69b6a7484bc7d1b99b3c0a)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
+!1 = !DIFile(filename: "epilogue-cfi-fp.c", directory: "epilogue-dwarf/test")
+!2 = !{}
+!3 = !{i32 1, !"NumRegisterParameters", i32 0}
+!4 = !{i32 2, !"Dwarf Version", i32 4}
+!5 = !{i32 2, !"Debug Info Version", i32 3}
+!6 = !{i32 1, !"wchar_size", i32 4}
+!7 = !{i32 7, !"PIC Level", i32 2}
+

test/CodeGen/X86/epilogue-cfi-no-fp.ll

+; RUN: llc -O0 < %s | FileCheck %s
+
+; ModuleID = 'epilogue-cfi-no-fp.c'
+source_filename = "epilogue-cfi-no-fp.c"
+target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
+target triple = "i686-pc-linux"
+
+; Function Attrs: noinline nounwind
+define i32 @foo(i32 %i, i32 %j, i32 %k, i32 %l, i32 %m) {
+; CHECK-LABEL:   foo:
+; CHECK:         addl $20, %esp
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 12
+; CHECK-NEXT:    popl %edi
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    popl %ebx
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
+; CHECK-NEXT:    retl
+entry:
+  %i.addr = alloca i32, align 4
+  %j.addr = alloca i32, align 4
+  %k.addr = alloca i32, align 4
+  %l.addr = alloca i32, align 4
+  %m.addr = alloca i32, align 4
+  store i32 %i, i32* %i.addr, align 4
+  store i32 %j, i32* %j.addr, align 4
+  store i32 %k, i32* %k.addr, align 4
+  store i32 %l, i32* %l.addr, align 4
+  store i32 %m, i32* %m.addr, align 4
+  ret i32 0
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6, !7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 5.0.0 (http://llvm.org/git/clang.git 3f8116e6a2815b1d5f3491493938d0c63c9f42c9) (http://llvm.org/git/llvm.git 4fde77f8f1a8e4482e69b6a7484bc7d1b99b3c0a)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
+!1 = !DIFile(filename: "epilogue-cfi-no-fp.c", directory: "epilogue-dwarf/test")
+!2 = !{}
+!3 = !{i32 1, !"NumRegisterParameters", i32 0}
+!4 = !{i32 2, !"Dwarf Version", i32 4}
+!5 = !{i32 2, !"Debug Info Version", i32 3}
+!6 = !{i32 1, !"wchar_size", i32 4}
+!7 = !{i32 7, !"PIC Level", i32 2}
+
+

test/CodeGen/X86/fast-isel-store.ll

 ; SSE64-NEXT:  .Lcfi0:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm1
 ; SSE64-NEXT:    addpd %xmm2, %xmm0
 ; SSE64-NEXT:    movupd %xmm0, (%eax)
 ; SSE64-NEXT:    movupd %xmm1, 16(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi1:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVX32-LABEL: test_store_4xf64:
 ; AVX32:       # BB#0:
 ; AVX32-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; AVX32-NEXT:    vmovupd %ymm0, (%rdi)
 ; AVX32-NEXT:    retq
 ;

 ; SSE32-NEXT:    addpd %xmm2, %xmm0
 ; SSE32-NEXT:    movapd %xmm0, (%rdi)
 ; SSE32-NEXT:    movapd %xmm1, 16(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_4xf64_aligned:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi1:
+; SSE64-NEXT:  .Lcfi2:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm1
 ; SSE64-NEXT:    addpd %xmm2, %xmm0
 ; SSE64-NEXT:    movapd %xmm0, (%eax)
 ; SSE64-NEXT:    movapd %xmm1, 16(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi3:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVX32-LABEL: test_store_4xf64_aligned:
 ; AVX32:       # BB#0:
 ; AVX32-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; AVX32-NEXT:    vmovapd %ymm0, (%rdi)
 ; AVX32-NEXT:    retq
 ;

 ; SSE32-NEXT:    movups %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movups %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movups %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_16xi32:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi2:
+; SSE64-NEXT:  .Lcfi4:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    movups %xmm0, (%eax)
 ; SSE64-NEXT:    movups %xmm1, 16(%eax)
 ; SSE64-NEXT:    movups %xmm2, 32(%eax)
 ; SSE64-NEXT:    movups %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi5:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_16xi32:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vmovups %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovups %ymm1, 32(%rdi)
 ; AVXONLY32-NEXT:    retq
 ;

 ; SSE32-NEXT:    movaps %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movaps %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movaps %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_16xi32_aligned:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi3:
+; SSE64-NEXT:  .Lcfi6:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    movaps %xmm0, (%eax)
 ; SSE64-NEXT:    movaps %xmm1, 16(%eax)
 ; SSE64-NEXT:    movaps %xmm2, 32(%eax)
 ; SSE64-NEXT:    movaps %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi7:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_16xi32_aligned:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vmovaps %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovaps %ymm1, 32(%rdi)
 ; AVXONLY32-NEXT:    retq
 ;

 ; SSE32-NEXT:    movups %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movups %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movups %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_16xf32:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi4:
+; SSE64-NEXT:  .Lcfi8:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    movups %xmm0, (%eax)
 ; SSE64-NEXT:    movups %xmm1, 16(%eax)
 ; SSE64-NEXT:    movups %xmm2, 32(%eax)
 ; SSE64-NEXT:    movups %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi9:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_16xf32:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vmovups %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovups %ymm1, 32(%rdi)
 ; AVXONLY32-NEXT:    retq
 ;

 ; SSE32-NEXT:    movaps %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movaps %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movaps %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_16xf32_aligned:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi5:
+; SSE64-NEXT:  .Lcfi10:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    movaps %xmm0, (%eax)
 ; SSE64-NEXT:    movaps %xmm1, 16(%eax)
 ; SSE64-NEXT:    movaps %xmm2, 32(%eax)
 ; SSE64-NEXT:    movaps %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi11:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_16xf32_aligned:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vmovaps %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovaps %ymm1, 32(%rdi)
 ; AVXONLY32-NEXT:    retq
 ;

 ; SSE32-NEXT:    movupd %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movupd %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movupd %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_8xf64:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi6:
+; SSE64-NEXT:  .Lcfi12:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movapd {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm2
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm1
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm0
 ; SSE64-NEXT:    movupd %xmm0, (%eax)
 ; SSE64-NEXT:    movupd %xmm1, 16(%eax)
 ; SSE64-NEXT:    movupd %xmm2, 32(%eax)
 ; SSE64-NEXT:    movupd %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi13:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_8xf64:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vaddpd %ymm3, %ymm1, %ymm1
 ; AVXONLY32-NEXT:    vaddpd %ymm2, %ymm0, %ymm0
 ; AVXONLY32-NEXT:    vmovupd %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovupd %ymm1, 32(%rdi)

 ; AVXONLY64-NEXT:    subl $32, %esp
 ; AVXONLY64-NEXT:    movl 8(%ebp), %eax
 ; AVXONLY64-NEXT:    vaddpd 40(%ebp), %ymm1, %ymm1
 ; AVXONLY64-NEXT:    vaddpd %ymm2, %ymm0, %ymm0
 ; AVXONLY64-NEXT:    vmovupd %ymm0, (%eax)
 ; AVXONLY64-NEXT:    vmovupd %ymm1, 32(%eax)
 ; AVXONLY64-NEXT:    movl %ebp, %esp
 ; AVXONLY64-NEXT:    popl %ebp
+; AVXONLY64-NEXT:  .Lcfi3:
+; AVXONLY64-NEXT:    .cfi_def_cfa %esp, 4
 ; AVXONLY64-NEXT:    retl
 ;
 ; AVX51232-LABEL: test_store_8xf64:
 ; AVX51232:       # BB#0:
 ; AVX51232-NEXT:    vaddpd %zmm1, %zmm0, %zmm0
 ; AVX51232-NEXT:    vmovupd %zmm0, (%rdi)
 ; AVX51232-NEXT:    retq
 ;

 ; SSE32-NEXT:    movapd %xmm1, 16(%rdi)
 ; SSE32-NEXT:    movapd %xmm2, 32(%rdi)
 ; SSE32-NEXT:    movapd %xmm3, 48(%rdi)
 ; SSE32-NEXT:    retq
 ;
 ; SSE64-LABEL: test_store_8xf64_aligned:
 ; SSE64:       # BB#0:
 ; SSE64-NEXT:    subl $12, %esp
-; SSE64-NEXT:  .Lcfi7:
+; SSE64-NEXT:  .Lcfi14:
 ; SSE64-NEXT:    .cfi_def_cfa_offset 16
 ; SSE64-NEXT:    movapd {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm3
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm2
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm1
 ; SSE64-NEXT:    addpd {{[0-9]+}}(%esp), %xmm0
 ; SSE64-NEXT:    movapd %xmm0, (%eax)
 ; SSE64-NEXT:    movapd %xmm1, 16(%eax)
 ; SSE64-NEXT:    movapd %xmm2, 32(%eax)
 ; SSE64-NEXT:    movapd %xmm3, 48(%eax)
 ; SSE64-NEXT:    addl $12, %esp
+; SSE64-NEXT:  .Lcfi15:
+; SSE64-NEXT:    .cfi_def_cfa_offset 4
 ; SSE64-NEXT:    retl
 ;
 ; AVXONLY32-LABEL: test_store_8xf64_aligned:
 ; AVXONLY32:       # BB#0:
 ; AVXONLY32-NEXT:    vaddpd %ymm3, %ymm1, %ymm1
 ; AVXONLY32-NEXT:    vaddpd %ymm2, %ymm0, %ymm0
 ; AVXONLY32-NEXT:    vmovapd %ymm0, (%rdi)
 ; AVXONLY32-NEXT:    vmovapd %ymm1, 32(%rdi)
 ; AVXONLY32-NEXT:    retq
 ;
 ; AVXONLY64-LABEL: test_store_8xf64_aligned:
 ; AVXONLY64:       # BB#0:
 ; AVXONLY64-NEXT:    pushl %ebp
-; AVXONLY64-NEXT:  .Lcfi3:
-; AVXONLY64-NEXT:    .cfi_def_cfa_offset 8
 ; AVXONLY64-NEXT:  .Lcfi4:
+; AVXONLY64-NEXT:    .cfi_def_cfa_offset 8
+; AVXONLY64-NEXT:  .Lcfi5:
 ; AVXONLY64-NEXT:    .cfi_offset %ebp, -8
 ; AVXONLY64-NEXT:    movl %esp, %ebp
-; AVXONLY64-NEXT:  .Lcfi5:
+; AVXONLY64-NEXT:  .Lcfi6:
 ; AVXONLY64-NEXT:    .cfi_def_cfa_register %ebp
 ; AVXONLY64-NEXT:    andl $-32, %esp
 ; AVXONLY64-NEXT:    subl $32, %esp
 ; AVXONLY64-NEXT:    movl 8(%ebp), %eax
 ; AVXONLY64-NEXT:    vaddpd 40(%ebp), %ymm1, %ymm1
 ; AVXONLY64-NEXT:    vaddpd %ymm2, %ymm0, %ymm0
 ; AVXONLY64-NEXT:    vmovapd %ymm0, (%eax)
 ; AVXONLY64-NEXT:    vmovapd %ymm1, 32(%eax)
 ; AVXONLY64-NEXT:    movl %ebp, %esp
 ; AVXONLY64-NEXT:    popl %ebp
+; AVXONLY64-NEXT:  .Lcfi7:
+; AVXONLY64-NEXT:    .cfi_def_cfa %esp, 4
 ; AVXONLY64-NEXT:    retl
 ;
 ; AVX51232-LABEL: test_store_8xf64_aligned:
 ; AVX51232:       # BB#0:
 ; AVX51232-NEXT:    vaddpd %zmm1, %zmm0, %zmm0
 ; AVX51232-NEXT:    vmovapd %zmm0, (%rdi)
 ; AVX51232-NEXT:    retq
 ;

test/CodeGen/X86/frame-lowering-debug-intrinsic.ll

 ; Test ensuring debug intrinsics do not affect generated function prologue.
 ;
 ; RUN: llc -O1 -mtriple=x86_64-unknown-unknown -o - %s | FileCheck %s
 
 define i64 @fn1NoDebug(i64 %a) {
   %call = call i64 @fn(i64 %a, i64 0)
   ret i64 %call
 }
 
 ; CHECK-LABEL: fn1NoDebug
 ; CHECK: popq %rcx
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-NEXT: ret
 
 define i64 @fn1WithDebug(i64 %a) !dbg !4 {
   %call = call i64 @fn(i64 %a, i64 0)
   tail call void @llvm.dbg.value(metadata i64 %call, i64 0, metadata !5, metadata !6), !dbg !7
   ret i64 %call
 }
 
 ; CHECK-LABEL: fn1WithDebug
 ; CHECK: popq %rcx
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-NEXT: ret
 
 %struct.Buffer = type { i8, [63 x i8] }
 
 define void @fn2NoDebug(%struct.Buffer* byval align 64 %p1) {
   ret void
 }
 
 ; CHECK-LABEL: fn2NoDebug
 ; CHECK: and
 ; CHECK-NOT: add
 ; CHECK-NOT: sub
 ; CHECK: mov
 ; CHECK-NEXT: pop
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT: ret
 
 define void @fn2WithDebug(%struct.Buffer* byval align 64 %p1) !dbg !8 {
   call void @llvm.dbg.declare(metadata %struct.Buffer* %p1, metadata !9, metadata !6), !dbg !10
   ret void
 }
 
 ; CHECK-LABEL: fn2WithDebug
 ; CHECK: and
 ; CHECK-NOT: add
 ; CHECK-NOT: sub
 ; CHECK: mov
 ; CHECK-NEXT: pop
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT: ret
 
 declare i64 @fn(i64, i64)
 
 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !llvm.dbg.cu = !{!0}

test/CodeGen/X86/haddsub-2.ll

 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm6 = xmm6[0],xmm12[0],xmm6[1],xmm12[1],xmm6[2],xmm12[2],xmm6[3],xmm12[3]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm5 = xmm5[0],xmm13[0],xmm5[1],xmm13[1],xmm5[2],xmm13[2],xmm5[3],xmm13[3]
 ; SSE3-NEXT:    punpckldq {{.*#+}} xmm5 = xmm5[0],xmm6[0],xmm5[1],xmm6[1]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm2 = xmm2[0],xmm14[0],xmm2[1],xmm14[1],xmm2[2],xmm14[2],xmm2[3],xmm14[3]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm15[0],xmm1[1],xmm15[1],xmm1[2],xmm15[2],xmm1[3],xmm15[3]
 ; SSE3-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
 ; SSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm5[0]
 ; SSE3-NEXT:    popq %rbx
+; SSE3-NEXT:  .Lcfi12:
+; SSE3-NEXT:    .cfi_def_cfa_offset 48
 ; SSE3-NEXT:    popq %r12
+; SSE3-NEXT:  .Lcfi13:
+; SSE3-NEXT:    .cfi_def_cfa_offset 40
 ; SSE3-NEXT:    popq %r13
+; SSE3-NEXT:  .Lcfi14:
+; SSE3-NEXT:    .cfi_def_cfa_offset 32
 ; SSE3-NEXT:    popq %r14
+; SSE3-NEXT:  .Lcfi15:
+; SSE3-NEXT:    .cfi_def_cfa_offset 24
 ; SSE3-NEXT:    popq %r15
+; SSE3-NEXT:  .Lcfi16:
+; SSE3-NEXT:    .cfi_def_cfa_offset 16
 ; SSE3-NEXT:    popq %rbp
+; SSE3-NEXT:  .Lcfi17:
+; SSE3-NEXT:    .cfi_def_cfa_offset 8
 ; SSE3-NEXT:    retq
 ;
 ; SSSE3-LABEL: avx2_vphadd_w_test:
 ; SSSE3:       # BB#0:
 ; SSSE3-NEXT:    phaddw %xmm1, %xmm0
 ; SSSE3-NEXT:    phaddw %xmm3, %xmm2
 ; SSSE3-NEXT:    movdqa %xmm2, %xmm1
 ; SSSE3-NEXT:    retq
; AVX2-NEXT:    retq
   %vecinit29 = insertelement <8 x i32> %vecinit25, i32 %add28, i32 7
   ret <8 x i32> %vecinit29
 }
 
 define <16 x i16> @avx2_hadd_w(<16 x i16> %a, <16 x i16> %b) {
 ; SSE3-LABEL: avx2_hadd_w:
 ; SSE3:       # BB#0:
 ; SSE3-NEXT:    pushq %rbp
-; SSE3-NEXT:  .Lcfi12:
+; SSE3-NEXT:  .Lcfi18:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 16
 ; SSE3-NEXT:    pushq %r15
-; SSE3-NEXT:  .Lcfi13:
+; SSE3-NEXT:  .Lcfi19:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 24
 ; SSE3-NEXT:    pushq %r14
-; SSE3-NEXT:  .Lcfi14:
+; SSE3-NEXT:  .Lcfi20:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 32
 ; SSE3-NEXT:    pushq %r13
-; SSE3-NEXT:  .Lcfi15:
+; SSE3-NEXT:  .Lcfi21:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 40
 ; SSE3-NEXT:    pushq %r12
-; SSE3-NEXT:  .Lcfi16:
+; SSE3-NEXT:  .Lcfi22:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 48
 ; SSE3-NEXT:    pushq %rbx
-; SSE3-NEXT:  .Lcfi17:
+; SSE3-NEXT:  .Lcfi23:
 ; SSE3-NEXT:    .cfi_def_cfa_offset 56
-; SSE3-NEXT:  .Lcfi18:
+; SSE3-NEXT:  .Lcfi24:
 ; SSE3-NEXT:    .cfi_offset %rbx, -56
-; SSE3-NEXT:  .Lcfi19:
+; SSE3-NEXT:  .Lcfi25:
 ; SSE3-NEXT:    .cfi_offset %r12, -48
-; SSE3-NEXT:  .Lcfi20:
+; SSE3-NEXT:  .Lcfi26:
 ; SSE3-NEXT:    .cfi_offset %r13, -40
-; SSE3-NEXT:  .Lcfi21:
+; SSE3-NEXT:  .Lcfi27:
 ; SSE3-NEXT:    .cfi_offset %r14, -32
-; SSE3-NEXT:  .Lcfi22:
+; SSE3-NEXT:  .Lcfi28:
 ; SSE3-NEXT:    .cfi_offset %r15, -24
-; SSE3-NEXT:  .Lcfi23:
+; SSE3-NEXT:  .Lcfi29:
 ; SSE3-NEXT:    .cfi_offset %rbp, -16
 ; SSE3-NEXT:    movd %xmm0, %eax
 ; SSE3-NEXT:    pextrw $1, %xmm0, %r10d
 ; SSE3-NEXT:    addl %eax, %r10d
 ; SSE3-NEXT:    pextrw $2, %xmm0, %eax
 ; SSE3-NEXT:    pextrw $3, %xmm0, %r11d
 ; SSE3-NEXT:    addl %eax, %r11d
 ; SSE3-NEXT:    pextrw $4, %xmm0, %eax

 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm6 = xmm6[0],xmm12[0],xmm6[1],xmm12[1],xmm6[2],xmm12[2],xmm6[3],xmm12[3]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm5 = xmm5[0],xmm13[0],xmm5[1],xmm13[1],xmm5[2],xmm13[2],xmm5[3],xmm13[3]
 ; SSE3-NEXT:    punpckldq {{.*#+}} xmm5 = xmm5[0],xmm6[0],xmm5[1],xmm6[1]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm2 = xmm2[0],xmm14[0],xmm2[1],xmm14[1],xmm2[2],xmm14[2],xmm2[3],xmm14[3]
 ; SSE3-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm15[0],xmm1[1],xmm15[1],xmm1[2],xmm15[2],xmm1[3],xmm15[3]
 ; SSE3-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
 ; SSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm5[0]
 ; SSE3-NEXT:    popq %rbx
+; SSE3-NEXT:  .Lcfi30:
+; SSE3-NEXT:    .cfi_def_cfa_offset 48
 ; SSE3-NEXT:    popq %r12
+; SSE3-NEXT:  .Lcfi31:
+; SSE3-NEXT:    .cfi_def_cfa_offset 40
 ; SSE3-NEXT:    popq %r13
+; SSE3-NEXT:  .Lcfi32:
+; SSE3-NEXT:    .cfi_def_cfa_offset 32
 ; SSE3-NEXT:    popq %r14
+; SSE3-NEXT:  .Lcfi33:
+; SSE3-NEXT:    .cfi_def_cfa_offset 24
 ; SSE3-NEXT:    popq %r15
+; SSE3-NEXT:  .Lcfi34:
+; SSE3-NEXT:    .cfi_def_cfa_offset 16
 ; SSE3-NEXT:    popq %rbp
+; SSE3-NEXT:  .Lcfi35:
+; SSE3-NEXT:    .cfi_def_cfa_offset 8
 ; SSE3-NEXT:    retq
 ;
 ; SSSE3-LABEL: avx2_hadd_w:
 ; SSSE3:       # BB#0:
 ; SSSE3-NEXT:    phaddw %xmm2, %xmm0
 ; SSSE3-NEXT:    phaddw %xmm3, %xmm1
 ; SSSE3-NEXT:    retq
 ;

test/CodeGen/X86/hipe-cc64.ll

 ; Sanity-check the tail call sequence. Number of arguments was chosen as to
 ; expose a bug where the tail call sequence clobbered the stack.
 define cc 11 { i64, i64, i64 } @tailcaller(i64 %hp, i64 %p) #0 {
   ; CHECK:      movl  $15, %esi
   ; CHECK-NEXT: movl  $31, %edx
   ; CHECK-NEXT: movl  $47, %ecx
   ; CHECK-NEXT: movl  $63, %r8d
   ; CHECK-NEXT: popq  %rax
+  ; CHECK-NEXT: :
+  ; CHECK-NEXT: .cfi_def_cfa_offset 16
   ; CHECK-NEXT: jmp tailcallee
   %ret = tail call cc11 { i64, i64, i64 } @tailcallee(i64 %hp, i64 %p, i64 15,
      i64 31, i64 47, i64 63, i64 79) #1
   ret { i64, i64, i64 } %ret
 }
 
 !hipe.literals = !{ !0, !1, !2 }
 !0 = !{ !"P_NSP_LIMIT", i32 160 }
 !1 = !{ !"X86_LEAF_WORDS", i32 24 }
 !2 = !{ !"AMD64_LEAF_WORDS", i32 24 }
 @clos = external constant i64
 declare cc 11 void @bar(i64, i64, i64, i64, i64, i64)
 declare cc 11 { i64, i64, i64 } @tailcallee(i64, i64, i64, i64, i64, i64, i64)

test/CodeGen/X86/illegal-bitfield-loadstore.ll

 ; X86-NEXT:    movb %al, 2(%ecx)
 ; X86-NEXT:    shll $16, %eax
 ; X86-NEXT:    orl %esi, %eax
 ; X86-NEXT:    shll $13, %edx
 ; X86-NEXT:    andl $16769023, %eax # imm = 0xFFDFFF
 ; X86-NEXT:    orl %edx, %eax
 ; X86-NEXT:    movw %ax, (%ecx)
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi2:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i24_insert_bit:
 ; X64:       # BB#0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rdi), %ecx
 ; X64-NEXT:    movb %cl, 2(%rdi)
 ; X64-NEXT:    shll $16, %ecx

test/CodeGen/X86/imul.ll

 ; X86-NEXT:    shll $5, %esi
 ; X86-NEXT:    subl %eax, %esi
 ; X86-NEXT:    movl $-31, %edx
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:    mull %edx
 ; X86-NEXT:    subl %ecx, %edx
 ; X86-NEXT:    subl %esi, %edx
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi2:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 entry:
   %tmp3 = mul i64 %a, -31
   ret i64 %tmp3
 }
 
 
 define i64 @test6(i64 %a) {

 ; X64-NEXT:    shlq $5, %rax
 ; X64-NEXT:    leaq (%rax,%rdi), %rax
 ; X64-NEXT:    negq %rax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: test7:
 ; X86:       # BB#0: # %entry
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:  .Lcfi2:
-; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:  .Lcfi3:
+; X86-NEXT:    .cfi_def_cfa_offset 8
+; X86-NEXT:  .Lcfi4:
 ; X86-NEXT:    .cfi_offset %esi, -8
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, %esi
 ; X86-NEXT:    shll $5, %esi
 ; X86-NEXT:    addl %eax, %esi
 ; X86-NEXT:    movl $-33, %edx
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:    mull %edx
 ; X86-NEXT:    subl %ecx, %edx
 ; X86-NEXT:    subl %esi, %edx
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi5:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 entry:
   %tmp3 = mul i64 %a, -33
   ret i64 %tmp3
 }
 
 define i64 @testOverflow(i64 %a) {
 ; X64-LABEL: testOverflow:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    movabsq $9223372036854775807, %rax # imm = 0x7FFFFFFFFFFFFFFF
 ; X64-NEXT:    imulq %rdi, %rax
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: testOverflow:
 ; X86:       # BB#0: # %entry
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:  .Lcfi4:
+; X86-NEXT:  .Lcfi6:
 ; X86-NEXT:    .cfi_def_cfa_offset 8
-; X86-NEXT:  .Lcfi5:
+; X86-NEXT:  .Lcfi7:
 ; X86-NEXT:    .cfi_offset %esi, -8
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl $-1, %edx
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:    mull %edx
 ; X86-NEXT:    movl %ecx, %esi
 ; X86-NEXT:    shll $31, %esi
 ; X86-NEXT:    subl %ecx, %esi
 ; X86-NEXT:    addl %esi, %edx
 ; X86-NEXT:    subl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi8:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 entry:
   %tmp3 = mul i64 %a, 9223372036854775807
   ret i64 %tmp3
 }

test/CodeGen/X86/lea-opt-cse1.ll

 ; X86-NEXT:    movl (%eax), %ecx
 ; X86-NEXT:    movl 16(%eax), %edx
 ; X86-NEXT:    leal 1(%ecx,%edx), %esi
 ; X86-NEXT:    addl %edx, %ecx
 ; X86-NEXT:    movl %esi, 12(%eax)
 ; X86-NEXT:    leal 1(%edx,%ecx), %ecx
 ; X86-NEXT:    movl %ecx, 16(%eax)
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi2:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
  entry:
    %h0 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 0
    %0 = load i32, i32* %h0, align 8
    %h3 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 3
    %h4 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 4
    %1 = load i32, i32* %h4, align 8
    %add = add i32 %0, 1
    %add4 = add i32 %add, %1
    store i32 %add4, i32* %h3, align 4
    %add29 = add i32 %add4 , %1
    store i32 %add29, i32* %h4, align 8
    ret void
 }

test/CodeGen/X86/lea-opt-cse2.ll

 ; X86-NEXT:    movl %edi, 12(%eax)
 ; X86-NEXT:    decl %ecx
 ; X86-NEXT:    jne .LBB0_1
 ; X86-NEXT:  # BB#2: # %exit
 ; X86-NEXT:    addl %esi, %edx
 ; X86-NEXT:    leal 1(%esi,%edx), %ecx
 ; X86-NEXT:    movl %ecx, 16(%eax)
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi4:
+; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:    popl %edi
+; X86-NEXT:  .Lcfi5:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
  entry:
    br label %loop
 
  loop:
    %iter = phi i32 [%n ,%entry ] ,[ %iter.ctr ,%loop]
    %h0 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 0
    %0 = load i32, i32* %h0, align 8

test/CodeGen/X86/lea-opt-cse3.ll

 ; X86-NEXT:    cmpl $10, %ecx
 ; X86-NEXT:    je .LBB2_2
 ; X86-NEXT:  # BB#1: # %mid
 ; X86-NEXT:    leal 4(%esi,%edx,8), %eax
 ; X86-NEXT:    imull %eax, %ecx
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:  .LBB2_2: # %exit
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi2:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 entry:
   %mul = shl i32 %b, 2
   %add = add i32 %a, 4
   %add1 = add i32 %add, %mul
   %cmp = icmp ne i32 %add1 , 10
   br i1 %cmp , label %mid , label %exit
 mid:

 ; X64-NEXT:    imull %eax, %ecx
 ; X64-NEXT:    movl %ecx, %eax
 ; X64-NEXT:  .LBB3_2: # %exit
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: foo1_mult_basic_blocks_illegal_scale:
 ; X86:       # BB#0: # %entry
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:  .Lcfi2:
-; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:  .Lcfi3:
+; X86-NEXT:    .cfi_def_cfa_offset 8
+; X86-NEXT:  .Lcfi4:
 ; X86-NEXT:    .cfi_offset %esi, -8
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    leal 4(%esi,%edx,2), %ecx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl $10, %ecx
 ; X86-NEXT:    je .LBB3_2
 ; X86-NEXT:  # BB#1: # %mid
 ; X86-NEXT:    leal 4(%esi,%edx,8), %eax
 ; X86-NEXT:    imull %eax, %ecx
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:  .LBB3_2: # %exit
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi5:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 entry:
   %mul = shl i32 %b, 1
   %add = add i32 %a, 4
   %add1 = add i32 %add, %mul
   %cmp = icmp ne i32 %add1 , 10
   br i1 %cmp, label %mid , label %exit
 mid:

test/CodeGen/X86/lea-opt-cse4.ll

 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    leal 1(%ecx,%edx), %esi
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    movl %esi, 12(%eax)
 ; X86-NEXT:    leal 1(%ecx,%edx), %ecx
 ; X86-NEXT:    movl %ecx, 16(%eax)
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi2:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
  entry:
    %h0 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 0
    %0 = load i32, i32* %h0, align 8
    %h3 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 3
    %h4 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 4
    %1 = load i32, i32* %h4, align 8
    %add  = add i32 %0  , 1

 ; X64-NEXT:    addl %eax, %ecx
 ; X64-NEXT:    addl %eax, %ecx
 ; X64-NEXT:    movl %ecx, 16(%rdi)
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: foo_loop:
 ; X86:       # BB#0: # %entry
 ; X86-NEXT:    pushl %edi
-; X86-NEXT:  .Lcfi2:
+; X86-NEXT:  .Lcfi3:
 ; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:  .Lcfi3:
-; X86-NEXT:    .cfi_def_cfa_offset 12
 ; X86-NEXT:  .Lcfi4:
-; X86-NEXT:    .cfi_offset %esi, -12
+; X86-NEXT:    .cfi_def_cfa_offset 12
 ; X86-NEXT:  .Lcfi5:
+; X86-NEXT:    .cfi_offset %esi, -12
+; X86-NEXT:  .Lcfi6:
 ; X86-NEXT:    .cfi_offset %edi, -8
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    .p2align 4, 0x90
 ; X86-NEXT:  .LBB1_1: # %loop
 ; X86-NEXT:    # =>This Inner Loop Header: Depth=1
 ; X86-NEXT:    movl (%eax), %esi
 ; X86-NEXT:    movl 16(%eax), %ecx
 ; X86-NEXT:    leal 1(%esi,%ecx), %edi
 ; X86-NEXT:    movl %edi, 12(%eax)
 ; X86-NEXT:    decl %edx
 ; X86-NEXT:    jne .LBB1_1
 ; X86-NEXT:  # BB#2: # %exit
 ; X86-NEXT:    addl %ecx, %esi
 ; X86-NEXT:    leal 1(%ecx,%esi), %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    addl %ecx, %edx
 ; X86-NEXT:    movl %edx, 16(%eax)
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi7:
+; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:    popl %edi
+; X86-NEXT:  .Lcfi8:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
  entry:
    br label %loop
 
  loop:
    %iter = phi i32 [%n ,%entry ] ,[ %iter.ctr ,%loop]
    %h0 = getelementptr inbounds %struct.SA, %struct.SA* %ctx, i64 0, i32 0
    %0 = load i32, i32* %h0, align 8

test/CodeGen/X86/legalize-shift-64.ll

 ; CHECK-NEXT:    movl %ebx, %ebp
 ; CHECK-NEXT:    xorl %ebx, %ebx
 ; CHECK-NEXT:  .LBB4_4:
 ; CHECK-NEXT:    movl %ebp, 12(%eax)
 ; CHECK-NEXT:    movl %ebx, 8(%eax)
 ; CHECK-NEXT:    movl %esi, 4(%eax)
 ; CHECK-NEXT:    movl %edi, (%eax)
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi8:
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    popl %edi
+; CHECK-NEXT:  .Lcfi9:
+; CHECK-NEXT:    .cfi_def_cfa_offset 12
 ; CHECK-NEXT:    popl %ebx
+; CHECK-NEXT:  .Lcfi10:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:  .Lcfi11:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl $4
   %shl = shl <2 x i64> %A, %B
   ret <2 x i64> %shl
 }
 
 ; PR16108
 define i32 @test6() {
 ; CHECK-LABEL: test6:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %ebp
-; CHECK-NEXT:  .Lcfi8:
+; CHECK-NEXT:  .Lcfi12:
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
-; CHECK-NEXT:  .Lcfi9:
+; CHECK-NEXT:  .Lcfi13:
 ; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
-; CHECK-NEXT:  .Lcfi10:
+; CHECK-NEXT:  .Lcfi14:
 ; CHECK-NEXT:    .cfi_def_cfa_register %ebp
 ; CHECK-NEXT:    andl $-8, %esp
 ; CHECK-NEXT:    subl $16, %esp
 ; CHECK-NEXT:    movl $1, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    xorl %ecx, %ecx
 ; CHECK-NEXT:    shldl $32, %eax, %ecx

 ; CHECK-NEXT:  # BB#3: # %if.then
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    jmp .LBB5_4
 ; CHECK-NEXT:  .LBB5_5: # %if.end
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:  .LBB5_4: # %if.then
 ; CHECK-NEXT:    movl %ebp, %esp
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:  .Lcfi15:
+; CHECK-NEXT:    .cfi_def_cfa %esp, 4
 ; CHECK-NEXT:    retl
   %x = alloca i32, align 4
   %t = alloca i64, align 8
   store volatile i32 1, i32* %x, align 4
   %load = load volatile i32, i32* %x, align 4
   %shl = shl i32 %load, 8
   %add = add i32 %shl, -224
   %sh_prom = zext i32 %add to i64

test/CodeGen/X86/live-out-reg-info.ll

 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    shrl $23, %edi
 ; CHECK-NEXT:    btl $8, %edi
 ; CHECK-NEXT:    jb .LBB0_2
 ; CHECK-NEXT:  # BB#1: # %true
 ; CHECK-NEXT:    callq qux
 ; CHECK-NEXT:  .LBB0_2: # %false
 ; CHECK-NEXT:    popq %rax
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %t0 = lshr i32 %a, 23
   br label %next
 next:
   %t1 = and i32 %t0, 256
   %t2 = icmp eq i32 %t1, 0
   br i1 %t2, label %true, label %false
 true:
   call void @qux()
   ret void
 false:
   ret void
 }
 

test/CodeGen/X86/load-combine.ll

 ; CHECK-NEXT:    orl %ecx, %esi
 ; CHECK-NEXT:    movzbl 2(%eax), %ecx
 ; CHECK-NEXT:    shll $8, %ecx
 ; CHECK-NEXT:    orl %esi, %ecx
 ; CHECK-NEXT:    movzbl 3(%eax), %eax
 ; CHECK-NEXT:    orl %ecx, %eax
 ; CHECK-NEXT:    orl %edx, %eax
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi2:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 ;
 ; CHECK64-LABEL: load_i32_by_i8_bswap_uses:
 ; CHECK64:       # BB#0:
 ; CHECK64-NEXT:    movzbl (%rdi), %eax
 ; CHECK64-NEXT:    shll $24, %eax
 ; CHECK64-NEXT:    movzbl 1(%rdi), %ecx
 ; CHECK64-NEXT:    movl %ecx, %edx

 ; res1 = ((i32) p[0] << 24) | ((i32) p[1] << 16)
 ; *q = 0;
 ; res2 = ((i32) p[2] << 8) | (i32) p[3]
 ; res1 | res2
 define i32 @load_i32_by_i8_bswap_store_in_between(i32* %arg, i32* %arg1) {
 ; CHECK-LABEL: load_i32_by_i8_bswap_store_in_between:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %esi
-; CHECK-NEXT:  .Lcfi2:
-; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:  .Lcfi4:
 ; CHECK-NEXT:    .cfi_offset %esi, -8
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movzbl (%ecx), %edx
 ; CHECK-NEXT:    shll $24, %edx
 ; CHECK-NEXT:    movzbl 1(%ecx), %esi
 ; CHECK-NEXT:    movl $0, (%eax)
 ; CHECK-NEXT:    shll $16, %esi
 ; CHECK-NEXT:    orl %edx, %esi
 ; CHECK-NEXT:    movzbl 2(%ecx), %edx
 ; CHECK-NEXT:    shll $8, %edx
 ; CHECK-NEXT:    orl %esi, %edx
 ; CHECK-NEXT:    movzbl 3(%ecx), %eax
 ; CHECK-NEXT:    orl %edx, %eax
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi5:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 ;
 ; CHECK64-LABEL: load_i32_by_i8_bswap_store_in_between:
 ; CHECK64:       # BB#0:
 ; CHECK64-NEXT:    movzbl (%rdi), %eax
 ; CHECK64-NEXT:    shll $24, %eax
 ; CHECK64-NEXT:    movzbl 1(%rdi), %ecx
 ; CHECK64-NEXT:    movl $0, (%rsi)

test/CodeGen/X86/masked_gather_scatter.ll

 ; KNL_32-NEXT:    vmovdqa64 8(%ebp), %zmm1
 ; KNL_32-NEXT:    kshiftrw $8, %k1, %k2
 ; KNL_32-NEXT:    vpgatherdq (,%ymm0), %zmm2 {%k1}
 ; KNL_32-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; KNL_32-NEXT:    vpgatherdq (,%ymm0), %zmm1 {%k2}
 ; KNL_32-NEXT:    vmovdqa64 %zmm2, %zmm0
 ; KNL_32-NEXT:    movl %ebp, %esp
 ; KNL_32-NEXT:    popl %ebp
+; KNL_32-NEXT:  .Lcfi3:
+; KNL_32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test_gather_16i64:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; SKX-NEXT:    vpslld $31, %zmm2, %zmm2
 ; SKX-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; SKX-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX-NEXT:    vpgatherqq (,%zmm0), %zmm3 {%k1}
 ; SKX-NEXT:    vpgatherqq (,%zmm1), %zmm4 {%k2}
 ; SKX-NEXT:    vmovdqa64 %zmm3, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm4, %zmm1
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test_gather_16i64:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    pushl %ebp
-; SKX_32-NEXT:  .Lcfi1:
-; SKX_32-NEXT:    .cfi_def_cfa_offset 8
 ; SKX_32-NEXT:  .Lcfi2:
+; SKX_32-NEXT:    .cfi_def_cfa_offset 8
+; SKX_32-NEXT:  .Lcfi3:
 ; SKX_32-NEXT:    .cfi_offset %ebp, -8
 ; SKX_32-NEXT:    movl %esp, %ebp
-; SKX_32-NEXT:  .Lcfi3:
+; SKX_32-NEXT:  .Lcfi4:
 ; SKX_32-NEXT:    .cfi_def_cfa_register %ebp
 ; SKX_32-NEXT:    andl $-64, %esp
 ; SKX_32-NEXT:    subl $64, %esp
 ; SKX_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; SKX_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; SKX_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; SKX_32-NEXT:    vmovdqa64 8(%ebp), %zmm1
 ; SKX_32-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX_32-NEXT:    vpgatherdq (,%ymm0), %zmm2 {%k1}
 ; SKX_32-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; SKX_32-NEXT:    vpgatherdq (,%ymm0), %zmm1 {%k2}
 ; SKX_32-NEXT:    vmovdqa64 %zmm2, %zmm0
 ; SKX_32-NEXT:    movl %ebp, %esp
 ; SKX_32-NEXT:    popl %ebp
+; SKX_32-NEXT:  .Lcfi5:
+; SKX_32-NEXT:    .cfi_def_cfa %esp, 4
 ; SKX_32-NEXT:    retl
   %res = call <16 x i64> @llvm.masked.gather.v16i64.v16p0i64(<16 x i64*> %ptrs, i32 4, <16 x i1> %mask, <16 x i64> %src0)
   ret <16 x i64> %res
 }
 declare <16 x i64> @llvm.masked.gather.v16i64.v16p0i64(<16 x i64*> %ptrs, i32, <16 x i1> %mask, <16 x i64> %src0)
 define <16 x float> @test_gather_16f32(<16 x float*> %ptrs, <16 x i1> %mask, <16 x float> %src0)  {
 ; KNL_64-LABEL: test_gather_16f32:
 ; KNL_64:       # BB#0:

 ; KNL_64-NEXT:    vgatherqpd (,%zmm1), %zmm4 {%k2}
 ; KNL_64-NEXT:    vmovapd %zmm3, %zmm0
 ; KNL_64-NEXT:    vmovapd %zmm4, %zmm1
 ; KNL_64-NEXT:    retq
 ;
 ; KNL_32-LABEL: test_gather_16f64:
 ; KNL_32:       # BB#0:
 ; KNL_32-NEXT:    pushl %ebp
-; KNL_32-NEXT:  .Lcfi3:
-; KNL_32-NEXT:    .cfi_def_cfa_offset 8
 ; KNL_32-NEXT:  .Lcfi4:
+; KNL_32-NEXT:    .cfi_def_cfa_offset 8
+; KNL_32-NEXT:  .Lcfi5:
 ; KNL_32-NEXT:    .cfi_offset %ebp, -8
 ; KNL_32-NEXT:    movl %esp, %ebp
-; KNL_32-NEXT:  .Lcfi5:
+; KNL_32-NEXT:  .Lcfi6:
 ; KNL_32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL_32-NEXT:    andl $-64, %esp
 ; KNL_32-NEXT:    subl $64, %esp
 ; KNL_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; KNL_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; KNL_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; KNL_32-NEXT:    vmovapd 8(%ebp), %zmm1
 ; KNL_32-NEXT:    kshiftrw $8, %k1, %k2
 ; KNL_32-NEXT:    vgatherdpd (,%ymm0), %zmm2 {%k1}
 ; KNL_32-NEXT:    vextractf64x4 $1, %zmm0, %ymm0
 ; KNL_32-NEXT:    vgatherdpd (,%ymm0), %zmm1 {%k2}
 ; KNL_32-NEXT:    vmovapd %zmm2, %zmm0
 ; KNL_32-NEXT:    movl %ebp, %esp
 ; KNL_32-NEXT:    popl %ebp
+; KNL_32-NEXT:  .Lcfi7:
+; KNL_32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test_gather_16f64:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; SKX-NEXT:    vpslld $31, %zmm2, %zmm2
 ; SKX-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; SKX-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX-NEXT:    vgatherqpd (,%zmm0), %zmm3 {%k1}
 ; SKX-NEXT:    vgatherqpd (,%zmm1), %zmm4 {%k2}
 ; SKX-NEXT:    vmovapd %zmm3, %zmm0
 ; SKX-NEXT:    vmovapd %zmm4, %zmm1
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test_gather_16f64:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    pushl %ebp
-; SKX_32-NEXT:  .Lcfi4:
+; SKX_32-NEXT:  .Lcfi6:
 ; SKX_32-NEXT:    .cfi_def_cfa_offset 8
-; SKX_32-NEXT:  .Lcfi5:
+; SKX_32-NEXT:  .Lcfi7:
 ; SKX_32-NEXT:    .cfi_offset %ebp, -8
 ; SKX_32-NEXT:    movl %esp, %ebp
-; SKX_32-NEXT:  .Lcfi6:
+; SKX_32-NEXT:  .Lcfi8:
 ; SKX_32-NEXT:    .cfi_def_cfa_register %ebp
 ; SKX_32-NEXT:    andl $-64, %esp
 ; SKX_32-NEXT:    subl $64, %esp
 ; SKX_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; SKX_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; SKX_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; SKX_32-NEXT:    vmovapd 8(%ebp), %zmm1
 ; SKX_32-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX_32-NEXT:    vgatherdpd (,%ymm0), %zmm2 {%k1}
 ; SKX_32-NEXT:    vextractf64x4 $1, %zmm0, %ymm0
 ; SKX_32-NEXT:    vgatherdpd (,%ymm0), %zmm1 {%k2}
 ; SKX_32-NEXT:    vmovapd %zmm2, %zmm0
 ; SKX_32-NEXT:    movl %ebp, %esp
 ; SKX_32-NEXT:    popl %ebp
+; SKX_32-NEXT:  .Lcfi9:
+; SKX_32-NEXT:    .cfi_def_cfa %esp, 4
 ; SKX_32-NEXT:    retl
   %res = call <16 x double> @llvm.masked.gather.v16f64.v16p0f64(<16 x double*> %ptrs, i32 4, <16 x i1> %mask, <16 x double> %src0)
   ret <16 x double> %res
 }
 declare <16 x double> @llvm.masked.gather.v16f64.v16p0f64(<16 x double*> %ptrs, i32, <16 x i1> %mask, <16 x double> %src0)
 define void @test_scatter_16i32(<16 x i32*> %ptrs, <16 x i1> %mask, <16 x i32> %src0)  {
 ; KNL_64-LABEL: test_scatter_16i32:
 ; KNL_64:       # BB#0:

 ; KNL_64-NEXT:    vpscatterqq %zmm3, (,%zmm0) {%k1}
 ; KNL_64-NEXT:    vpscatterqq %zmm4, (,%zmm1) {%k2}
 ; KNL_64-NEXT:    vzeroupper
 ; KNL_64-NEXT:    retq
 ;
 ; KNL_32-LABEL: test_scatter_16i64:
 ; KNL_32:       # BB#0:
 ; KNL_32-NEXT:    pushl %ebp
-; KNL_32-NEXT:  .Lcfi6:
+; KNL_32-NEXT:  .Lcfi8:
 ; KNL_32-NEXT:    .cfi_def_cfa_offset 8
-; KNL_32-NEXT:  .Lcfi7:
+; KNL_32-NEXT:  .Lcfi9:
 ; KNL_32-NEXT:    .cfi_offset %ebp, -8
 ; KNL_32-NEXT:    movl %esp, %ebp
-; KNL_32-NEXT:  .Lcfi8:
+; KNL_32-NEXT:  .Lcfi10:
 ; KNL_32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL_32-NEXT:    andl $-64, %esp
 ; KNL_32-NEXT:    subl $64, %esp
 ; KNL_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; KNL_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; KNL_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; KNL_32-NEXT:    vmovdqa64 8(%ebp), %zmm1
 ; KNL_32-NEXT:    kshiftrw $8, %k1, %k2
 ; KNL_32-NEXT:    vpscatterdq %zmm2, (,%ymm0) {%k1}
 ; KNL_32-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; KNL_32-NEXT:    vpscatterdq %zmm1, (,%ymm0) {%k2}
 ; KNL_32-NEXT:    movl %ebp, %esp
 ; KNL_32-NEXT:    popl %ebp
+; KNL_32-NEXT:  .Lcfi11:
+; KNL_32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL_32-NEXT:    vzeroupper
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test_scatter_16i64:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; SKX-NEXT:    vpslld $31, %zmm2, %zmm2
 ; SKX-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; SKX-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX-NEXT:    vpscatterqq %zmm3, (,%zmm0) {%k1}
 ; SKX-NEXT:    vpscatterqq %zmm4, (,%zmm1) {%k2}
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test_scatter_16i64:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    pushl %ebp
-; SKX_32-NEXT:  .Lcfi7:
+; SKX_32-NEXT:  .Lcfi10:
 ; SKX_32-NEXT:    .cfi_def_cfa_offset 8
-; SKX_32-NEXT:  .Lcfi8:
+; SKX_32-NEXT:  .Lcfi11:
 ; SKX_32-NEXT:    .cfi_offset %ebp, -8
 ; SKX_32-NEXT:    movl %esp, %ebp
-; SKX_32-NEXT:  .Lcfi9:
+; SKX_32-NEXT:  .Lcfi12:
 ; SKX_32-NEXT:    .cfi_def_cfa_register %ebp
 ; SKX_32-NEXT:    andl $-64, %esp
 ; SKX_32-NEXT:    subl $64, %esp
 ; SKX_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; SKX_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; SKX_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; SKX_32-NEXT:    vmovdqa64 8(%ebp), %zmm1
 ; SKX_32-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX_32-NEXT:    vpscatterdq %zmm2, (,%ymm0) {%k1}
 ; SKX_32-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; SKX_32-NEXT:    vpscatterdq %zmm1, (,%ymm0) {%k2}
 ; SKX_32-NEXT:    movl %ebp, %esp
 ; SKX_32-NEXT:    popl %ebp
+; SKX_32-NEXT:  .Lcfi13:
+; SKX_32-NEXT:    .cfi_def_cfa %esp, 4
 ; SKX_32-NEXT:    vzeroupper
 ; SKX_32-NEXT:    retl
   call void @llvm.masked.scatter.v16i64.v16p0i64(<16 x i64> %src0, <16 x i64*> %ptrs, i32 4, <16 x i1> %mask)
   ret void
 }
 declare void @llvm.masked.scatter.v16i64.v16p0i64(<16 x i64> %src0, <16 x i64*> %ptrs, i32, <16 x i1> %mask)
 define void @test_scatter_16f32(<16 x float*> %ptrs, <16 x i1> %mask, <16 x float> %src0)  {
 ; KNL_64-LABEL: test_scatter_16f32:

 ; KNL_64-NEXT:    vscatterqpd %zmm3, (,%zmm0) {%k1}
 ; KNL_64-NEXT:    vscatterqpd %zmm4, (,%zmm1) {%k2}
 ; KNL_64-NEXT:    vzeroupper
 ; KNL_64-NEXT:    retq
 ;
 ; KNL_32-LABEL: test_scatter_16f64:
 ; KNL_32:       # BB#0:
 ; KNL_32-NEXT:    pushl %ebp
-; KNL_32-NEXT:  .Lcfi9:
+; KNL_32-NEXT:  .Lcfi12:
 ; KNL_32-NEXT:    .cfi_def_cfa_offset 8
-; KNL_32-NEXT:  .Lcfi10:
+; KNL_32-NEXT:  .Lcfi13:
 ; KNL_32-NEXT:    .cfi_offset %ebp, -8
 ; KNL_32-NEXT:    movl %esp, %ebp
-; KNL_32-NEXT:  .Lcfi11:
+; KNL_32-NEXT:  .Lcfi14:
 ; KNL_32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL_32-NEXT:    andl $-64, %esp
 ; KNL_32-NEXT:    subl $64, %esp
 ; KNL_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; KNL_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; KNL_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; KNL_32-NEXT:    vmovapd 8(%ebp), %zmm1
 ; KNL_32-NEXT:    kshiftrw $8, %k1, %k2
 ; KNL_32-NEXT:    vscatterdpd %zmm2, (,%ymm0) {%k1}
 ; KNL_32-NEXT:    vextractf64x4 $1, %zmm0, %ymm0
 ; KNL_32-NEXT:    vscatterdpd %zmm1, (,%ymm0) {%k2}
 ; KNL_32-NEXT:    movl %ebp, %esp
 ; KNL_32-NEXT:    popl %ebp
+; KNL_32-NEXT:  .Lcfi15:
+; KNL_32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL_32-NEXT:    vzeroupper
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test_scatter_16f64:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpmovsxbd %xmm2, %zmm2
 ; SKX-NEXT:    vpslld $31, %zmm2, %zmm2
 ; SKX-NEXT:    vptestmd %zmm2, %zmm2, %k1
 ; SKX-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX-NEXT:    vscatterqpd %zmm3, (,%zmm0) {%k1}
 ; SKX-NEXT:    vscatterqpd %zmm4, (,%zmm1) {%k2}
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test_scatter_16f64:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    pushl %ebp
-; SKX_32-NEXT:  .Lcfi10:
+; SKX_32-NEXT:  .Lcfi14:
 ; SKX_32-NEXT:    .cfi_def_cfa_offset 8
-; SKX_32-NEXT:  .Lcfi11:
+; SKX_32-NEXT:  .Lcfi15:
 ; SKX_32-NEXT:    .cfi_offset %ebp, -8
 ; SKX_32-NEXT:    movl %esp, %ebp
-; SKX_32-NEXT:  .Lcfi12:
+; SKX_32-NEXT:  .Lcfi16:
 ; SKX_32-NEXT:    .cfi_def_cfa_register %ebp
 ; SKX_32-NEXT:    andl $-64, %esp
 ; SKX_32-NEXT:    subl $64, %esp
 ; SKX_32-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; SKX_32-NEXT:    vpslld $31, %zmm1, %zmm1
 ; SKX_32-NEXT:    vptestmd %zmm1, %zmm1, %k1
 ; SKX_32-NEXT:    vmovapd 8(%ebp), %zmm1
 ; SKX_32-NEXT:    kshiftrw $8, %k1, %k2
 ; SKX_32-NEXT:    vscatterdpd %zmm2, (,%ymm0) {%k1}
 ; SKX_32-NEXT:    vextractf64x4 $1, %zmm0, %ymm0
 ; SKX_32-NEXT:    vscatterdpd %zmm1, (,%ymm0) {%k2}
 ; SKX_32-NEXT:    movl %ebp, %esp
 ; SKX_32-NEXT:    popl %ebp
+; SKX_32-NEXT:  .Lcfi17:
+; SKX_32-NEXT:    .cfi_def_cfa %esp, 4
 ; SKX_32-NEXT:    vzeroupper
 ; SKX_32-NEXT:    retl
   call void @llvm.masked.scatter.v16f64.v16p0f64(<16 x double> %src0, <16 x double*> %ptrs, i32 4, <16 x i1> %mask)
   ret void
 }
 declare void @llvm.masked.scatter.v16f64.v16p0f64(<16 x double> %src0, <16 x double*> %ptrs, i32, <16 x i1> %mask)
 
 define <4 x i64> @test_pr28312(<4 x i64*> %p1, <4 x i1> %k, <4 x i1> %k2,<4 x i64> %d) {

 ; KNL_64-NEXT:    vpgatherqq (,%zmm0), %zmm1 {%k1}
 ; KNL_64-NEXT:    vpaddq %ymm1, %ymm1, %ymm0
 ; KNL_64-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; KNL_64-NEXT:    retq
 ;
 ; KNL_32-LABEL: test_pr28312:
 ; KNL_32:       # BB#0:
 ; KNL_32-NEXT:    pushl %ebp
-; KNL_32-NEXT:  .Lcfi12:
+; KNL_32-NEXT:  .Lcfi16:
 ; KNL_32-NEXT:    .cfi_def_cfa_offset 8
-; KNL_32-NEXT:  .Lcfi13:
+; KNL_32-NEXT:  .Lcfi17:
 ; KNL_32-NEXT:    .cfi_offset %ebp, -8
 ; KNL_32-NEXT:    movl %esp, %ebp
-; KNL_32-NEXT:  .Lcfi14:
+; KNL_32-NEXT:  .Lcfi18:
 ; KNL_32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL_32-NEXT:    andl $-32, %esp
 ; KNL_32-NEXT:    subl $32, %esp
 ; KNL_32-NEXT:    # kill: %XMM0<def> %XMM0<kill> %YMM0<def>
 ; KNL_32-NEXT:    vpslld $31, %xmm1, %xmm1
 ; KNL_32-NEXT:    vpsrad $31, %xmm1, %xmm1
 ; KNL_32-NEXT:    vpmovsxdq %xmm1, %ymm1
 ; KNL_32-NEXT:    vmovdqa %ymm1, %ymm1
 ; KNL_32-NEXT:    vpmovsxdq %ymm0, %zmm0
 ; KNL_32-NEXT:    vpsllq $63, %zmm1, %zmm1
 ; KNL_32-NEXT:    vptestmq %zmm1, %zmm1, %k1
 ; KNL_32-NEXT:    vpgatherqq (,%zmm0), %zmm1 {%k1}
 ; KNL_32-NEXT:    vpaddq %ymm1, %ymm1, %ymm0
 ; KNL_32-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; KNL_32-NEXT:    movl %ebp, %esp
 ; KNL_32-NEXT:    popl %ebp
+; KNL_32-NEXT:  .Lcfi19:
+; KNL_32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test_pr28312:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpslld $31, %xmm1, %xmm1
 ; SKX-NEXT:    vptestmd %xmm1, %xmm1, %k1
 ; SKX-NEXT:    vpgatherqq (,%ymm0), %ymm1 {%k1}
 ; SKX-NEXT:    vpaddq %ymm1, %ymm1, %ymm0
 ; SKX-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test_pr28312:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    pushl %ebp
-; SKX_32-NEXT:  .Lcfi13:
+; SKX_32-NEXT:  .Lcfi18:
 ; SKX_32-NEXT:    .cfi_def_cfa_offset 8
-; SKX_32-NEXT:  .Lcfi14:
+; SKX_32-NEXT:  .Lcfi19:
 ; SKX_32-NEXT:    .cfi_offset %ebp, -8
 ; SKX_32-NEXT:    movl %esp, %ebp
-; SKX_32-NEXT:  .Lcfi15:
+; SKX_32-NEXT:  .Lcfi20:
 ; SKX_32-NEXT:    .cfi_def_cfa_register %ebp
 ; SKX_32-NEXT:    andl $-32, %esp
 ; SKX_32-NEXT:    subl $32, %esp
 ; SKX_32-NEXT:    vpslld $31, %xmm1, %xmm1
 ; SKX_32-NEXT:    vptestmd %xmm1, %xmm1, %k1
 ; SKX_32-NEXT:    vpgatherdq (,%xmm0), %ymm1 {%k1}
 ; SKX_32-NEXT:    vpaddq %ymm1, %ymm1, %ymm0
 ; SKX_32-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; SKX_32-NEXT:    movl %ebp, %esp
 ; SKX_32-NEXT:    popl %ebp
+; SKX_32-NEXT:  .Lcfi21:
+; SKX_32-NEXT:    .cfi_def_cfa %esp, 4
 ; SKX_32-NEXT:    retl
   %g1 = call <4 x i64> @llvm.masked.gather.v4i64.v4p0i64(<4 x i64*> %p1, i32 8, <4 x i1> %k, <4 x i64> undef)
   %g2 = call <4 x i64> @llvm.masked.gather.v4i64.v4p0i64(<4 x i64*> %p1, i32 8, <4 x i1> %k, <4 x i64> undef)
   %g3 = call <4 x i64> @llvm.masked.gather.v4i64.v4p0i64(<4 x i64*> %p1, i32 8, <4 x i1> %k, <4 x i64> undef)
   %a = add <4 x i64> %g1, %g2
   %b = add <4 x i64> %a, %g3
   ret <4 x i64> %b
 }
 declare <4 x i64> @llvm.masked.gather.v4i64.v4p0i64(<4 x i64*>, i32, <4 x i1>, <4 x i64>)

test/CodeGen/X86/memset-nonzero.ll

 ; SSE:       # BB#0:
 ; SSE-NEXT:    pushq %rax
 ; SSE-NEXT:  .Lcfi0:
 ; SSE-NEXT:    .cfi_def_cfa_offset 16
 ; SSE-NEXT:    movl $42, %esi
 ; SSE-NEXT:    movl $256, %edx # imm = 0x100
 ; SSE-NEXT:    callq memset
 ; SSE-NEXT:    popq %rax
+; SSE-NEXT:  .Lcfi1:
+; SSE-NEXT:    .cfi_def_cfa_offset 8
 ; SSE-NEXT:    retq
 ;
 ; SSE2FAST-LABEL: memset_256_nonzero_bytes:
 ; SSE2FAST:       # BB#0:
 ; SSE2FAST-NEXT:    movaps {{.*#+}} xmm0 = [42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42]
 ; SSE2FAST-NEXT:    movups %xmm0, 240(%rdi)
 ; SSE2FAST-NEXT:    movups %xmm0, 224(%rdi)
 ; SSE2FAST-NEXT:    movups %xmm0, 208(%rdi)

test/CodeGen/X86/merge-consecutive-loads-128.ll

 ; X32-SSE1-NEXT:    movl 12(%ecx), %esi
 ; X32-SSE1-NEXT:    movl 16(%ecx), %edi
 ; X32-SSE1-NEXT:    movl 20(%ecx), %ecx
 ; X32-SSE1-NEXT:    movl %ecx, 12(%eax)
 ; X32-SSE1-NEXT:    movl %edi, 8(%eax)
 ; X32-SSE1-NEXT:    movl %esi, 4(%eax)
 ; X32-SSE1-NEXT:    movl %edx, (%eax)
 ; X32-SSE1-NEXT:    popl %esi
+; X32-SSE1-NEXT:  .Lcfi4:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    popl %edi
+; X32-SSE1-NEXT:  .Lcfi5:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE1-NEXT:    retl $4
 ;
 ; X32-SSE41-LABEL: merge_2i64_i64_12:
 ; X32-SSE41:       # BB#0:
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movups 8(%eax), %xmm0
 ; X32-SSE41-NEXT:    retl
   %ptr0 = getelementptr inbounds i64, i64* %ptr, i64 1

 ; AVX-LABEL: merge_4i32_i32_23u5:
 ; AVX:       # BB#0:
 ; AVX-NEXT:    vmovups 8(%rdi), %xmm0
 ; AVX-NEXT:    retq
 ;
 ; X32-SSE1-LABEL: merge_4i32_i32_23u5:
 ; X32-SSE1:       # BB#0:
 ; X32-SSE1-NEXT:    pushl %esi
-; X32-SSE1-NEXT:  .Lcfi4:
+; X32-SSE1-NEXT:  .Lcfi6:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
-; X32-SSE1-NEXT:  .Lcfi5:
+; X32-SSE1-NEXT:  .Lcfi7:
 ; X32-SSE1-NEXT:    .cfi_offset %esi, -8
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-SSE1-NEXT:    movl 8(%ecx), %edx
 ; X32-SSE1-NEXT:    movl 12(%ecx), %esi
 ; X32-SSE1-NEXT:    movl 20(%ecx), %ecx
 ; X32-SSE1-NEXT:    movl %esi, 4(%eax)
 ; X32-SSE1-NEXT:    movl %edx, (%eax)
 ; X32-SSE1-NEXT:    movl %ecx, 12(%eax)
 ; X32-SSE1-NEXT:    popl %esi
+; X32-SSE1-NEXT:  .Lcfi8:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE1-NEXT:    retl $4
 ;
 ; X32-SSE41-LABEL: merge_4i32_i32_23u5:
 ; X32-SSE41:       # BB#0:
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movups 8(%eax), %xmm0
 ; X32-SSE41-NEXT:    retl
   %ptr0 = getelementptr inbounds i32, i32* %ptr, i64 2

 ; AVX-LABEL: merge_8i16_i16_23u567u9:
 ; AVX:       # BB#0:
 ; AVX-NEXT:    vmovups 4(%rdi), %xmm0
 ; AVX-NEXT:    retq
 ;
 ; X32-SSE1-LABEL: merge_8i16_i16_23u567u9:
 ; X32-SSE1:       # BB#0:
 ; X32-SSE1-NEXT:    pushl %edi
-; X32-SSE1-NEXT:  .Lcfi6:
+; X32-SSE1-NEXT:  .Lcfi9:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    pushl %esi
-; X32-SSE1-NEXT:  .Lcfi7:
+; X32-SSE1-NEXT:  .Lcfi10:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 12
-; X32-SSE1-NEXT:  .Lcfi8:
+; X32-SSE1-NEXT:  .Lcfi11:
 ; X32-SSE1-NEXT:    .cfi_offset %esi, -12
-; X32-SSE1-NEXT:  .Lcfi9:
+; X32-SSE1-NEXT:  .Lcfi12:
 ; X32-SSE1-NEXT:    .cfi_offset %edi, -8
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-SSE1-NEXT:    movl 4(%ecx), %edx
 ; X32-SSE1-NEXT:    movl 10(%ecx), %esi
 ; X32-SSE1-NEXT:    movzwl 14(%ecx), %edi
 ; X32-SSE1-NEXT:    movzwl 18(%ecx), %ecx
 ; X32-SSE1-NEXT:    movw %di, 10(%eax)
 ; X32-SSE1-NEXT:    movw %cx, 14(%eax)
 ; X32-SSE1-NEXT:    movl %esi, 6(%eax)
 ; X32-SSE1-NEXT:    movl %edx, (%eax)
 ; X32-SSE1-NEXT:    popl %esi
+; X32-SSE1-NEXT:  .Lcfi13:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    popl %edi
+; X32-SSE1-NEXT:  .Lcfi14:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE1-NEXT:    retl $4
 ;
 ; X32-SSE41-LABEL: merge_8i16_i16_23u567u9:
 ; X32-SSE41:       # BB#0:
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movups 4(%eax), %xmm0
 ; X32-SSE41-NEXT:    retl
   %ptr0 = getelementptr inbounds i16, i16* %ptr, i64 2

 ; AVX-LABEL: merge_16i8_i8_01u3456789ABCDuF:
 ; AVX:       # BB#0:
 ; AVX-NEXT:    vmovups (%rdi), %xmm0
 ; AVX-NEXT:    retq
 ;
 ; X32-SSE1-LABEL: merge_16i8_i8_01u3456789ABCDuF:
 ; X32-SSE1:       # BB#0:
 ; X32-SSE1-NEXT:    pushl %ebp
-; X32-SSE1-NEXT:  .Lcfi10:
+; X32-SSE1-NEXT:  .Lcfi15:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    pushl %ebx
-; X32-SSE1-NEXT:  .Lcfi11:
+; X32-SSE1-NEXT:  .Lcfi16:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 12
 ; X32-SSE1-NEXT:    pushl %edi
-; X32-SSE1-NEXT:  .Lcfi12:
+; X32-SSE1-NEXT:  .Lcfi17:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 16
 ; X32-SSE1-NEXT:    pushl %esi
-; X32-SSE1-NEXT:  .Lcfi13:
+; X32-SSE1-NEXT:  .Lcfi18:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 20
-; X32-SSE1-NEXT:  .Lcfi14:
+; X32-SSE1-NEXT:  .Lcfi19:
 ; X32-SSE1-NEXT:    .cfi_offset %esi, -20
-; X32-SSE1-NEXT:  .Lcfi15:
+; X32-SSE1-NEXT:  .Lcfi20:
 ; X32-SSE1-NEXT:    .cfi_offset %edi, -16
-; X32-SSE1-NEXT:  .Lcfi16:
+; X32-SSE1-NEXT:  .Lcfi21:
 ; X32-SSE1-NEXT:    .cfi_offset %ebx, -12
-; X32-SSE1-NEXT:  .Lcfi17:
+; X32-SSE1-NEXT:  .Lcfi22:
 ; X32-SSE1-NEXT:    .cfi_offset %ebp, -8
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-SSE1-NEXT:    movzwl (%ecx), %ebp
 ; X32-SSE1-NEXT:    movl 3(%ecx), %esi
 ; X32-SSE1-NEXT:    movl 7(%ecx), %edi
 ; X32-SSE1-NEXT:    movzwl 11(%ecx), %ebx
 ; X32-SSE1-NEXT:    movb 13(%ecx), %dl
 ; X32-SSE1-NEXT:    movb 15(%ecx), %cl
 ; X32-SSE1-NEXT:    movb %dl, 13(%eax)
 ; X32-SSE1-NEXT:    movb %cl, 15(%eax)
 ; X32-SSE1-NEXT:    movw %bx, 11(%eax)
 ; X32-SSE1-NEXT:    movl %edi, 7(%eax)
 ; X32-SSE1-NEXT:    movl %esi, 3(%eax)
 ; X32-SSE1-NEXT:    movw %bp, (%eax)
 ; X32-SSE1-NEXT:    popl %esi
+; X32-SSE1-NEXT:  .Lcfi23:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 16
 ; X32-SSE1-NEXT:    popl %edi
+; X32-SSE1-NEXT:  .Lcfi24:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 12
 ; X32-SSE1-NEXT:    popl %ebx
+; X32-SSE1-NEXT:  .Lcfi25:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    popl %ebp
+; X32-SSE1-NEXT:  .Lcfi26:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE1-NEXT:    retl $4
 ;
 ; X32-SSE41-LABEL: merge_16i8_i8_01u3456789ABCDuF:
 ; X32-SSE41:       # BB#0:
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movups (%eax), %xmm0
 ; X32-SSE41-NEXT:    retl
   %ptr0 = getelementptr inbounds i8, i8* %ptr, i64 0

 ; AVX-NEXT:    vmovsd {{.*#+}} xmm0 = mem[0],zero
 ; AVX-NEXT:    vmovsd {{.*#+}} xmm1 = mem[0],zero
 ; AVX-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
 ; AVX-NEXT:    retq
 ;
 ; X32-SSE1-LABEL: merge_2i64_i64_12_volatile:
 ; X32-SSE1:       # BB#0:
 ; X32-SSE1-NEXT:    pushl %edi
-; X32-SSE1-NEXT:  .Lcfi18:
+; X32-SSE1-NEXT:  .Lcfi27:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    pushl %esi
-; X32-SSE1-NEXT:  .Lcfi19:
+; X32-SSE1-NEXT:  .Lcfi28:
 ; X32-SSE1-NEXT:    .cfi_def_cfa_offset 12
-; X32-SSE1-NEXT:  .Lcfi20:
+; X32-SSE1-NEXT:  .Lcfi29:
 ; X32-SSE1-NEXT:    .cfi_offset %esi, -12
-; X32-SSE1-NEXT:  .Lcfi21:
+; X32-SSE1-NEXT:  .Lcfi30:
 ; X32-SSE1-NEXT:    .cfi_offset %edi, -8
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE1-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-SSE1-NEXT:    movl 8(%ecx), %edx
 ; X32-SSE1-NEXT:    movl 12(%ecx), %esi
 ; X32-SSE1-NEXT:    movl 16(%ecx), %edi
 ; X32-SSE1-NEXT:    movl 20(%ecx), %ecx
 ; X32-SSE1-NEXT:    movl %ecx, 12(%eax)
 ; X32-SSE1-NEXT:    movl %edi, 8(%eax)
 ; X32-SSE1-NEXT:    movl %esi, 4(%eax)
 ; X32-SSE1-NEXT:    movl %edx, (%eax)
 ; X32-SSE1-NEXT:    popl %esi
+; X32-SSE1-NEXT:  .Lcfi31:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE1-NEXT:    popl %edi
+; X32-SSE1-NEXT:  .Lcfi32:
+; X32-SSE1-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE1-NEXT:    retl $4
 ;
 ; X32-SSE41-LABEL: merge_2i64_i64_12_volatile:
 ; X32-SSE41:       # BB#0:
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X32-SSE41-NEXT:    pinsrd $1, 12(%eax), %xmm0
 ; X32-SSE41-NEXT:    pinsrd $2, 16(%eax), %xmm0

test/CodeGen/X86/movtopush.ll

 ; LINUX: .cfi_adjust_cfa_offset 4
 ; LINUX: pushl   $3
 ; LINUX: .cfi_adjust_cfa_offset 4
 ; LINUX: pushl   $2
 ; LINUX: .cfi_adjust_cfa_offset 4
 ; LINUX: pushl   $1
 ; LINUX: .cfi_adjust_cfa_offset 4
 ; LINUX: calll   good
-; LINUX: addl    $28, %esp
+; LINUX: addl    $16, %esp
 ; LINUX: .cfi_adjust_cfa_offset -16
+; LINUX: addl    $12, %esp
+; LINUX: .cfi_def_cfa_offset 4
 ; LINUX-NOT: add
 ; LINUX: retl
 define void @pr27140() optsize {
 entry:
   tail call void @good(i32 1, i32 2, i32 3, i32 4)
   ret void
 }
 

test/CodeGen/X86/mul-i256.ll

 ; X32-NEXT:    movl %ecx, 20(%ebx)
 ; X32-NEXT:    movl %edx, 24(%ebx)
 ; X32-NEXT:    movl %eax, 28(%ebx)
 ; X32-NEXT:    leal -12(%ebp), %esp
 ; X32-NEXT:    popl %esi
 ; X32-NEXT:    popl %edi
 ; X32-NEXT:    popl %ebx
 ; X32-NEXT:    popl %ebp
+; X32-NEXT:  .Lcfi6:
+; X32-NEXT:    .cfi_def_cfa %esp, 4
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    pushq %r15
 ; X64-NEXT:  .Lcfi0:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    pushq %r14

 ; X64-NEXT:    adcq %rcx, %rdx
 ; X64-NEXT:    addq %r10, %rax
 ; X64-NEXT:    adcq %rdi, %rdx
 ; X64-NEXT:    movq %r14, (%r9)
 ; X64-NEXT:    movq %r11, 8(%r9)
 ; X64-NEXT:    movq %rax, 16(%r9)
 ; X64-NEXT:    movq %rdx, 24(%r9)
 ; X64-NEXT:    popq %rbx
+; X64-NEXT:  .Lcfi6:
+; X64-NEXT:    .cfi_def_cfa_offset 24
 ; X64-NEXT:    popq %r14
+; X64-NEXT:  .Lcfi7:
+; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    popq %r15
+; X64-NEXT:  .Lcfi8:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 entry:
   %av = load i256, i256* %a
   %bv = load i256, i256* %b
   %r = mul i256 %av, %bv
   store i256 %r, i256* %out
   ret void
 }
 
 attributes #0 = { norecurse nounwind uwtable }

test/CodeGen/X86/pr21792.ll

 ; CHECK-NEXT:    leaq stuff(%r9), %rsi
 ; CHECK-NEXT:    andl $2032, %edx # imm = 0x7F0
 ; CHECK-NEXT:    leaq stuff(%rdx), %rdx
 ; CHECK-NEXT:    leaq stuff(%rcx), %rcx
 ; CHECK-NEXT:    leaq stuff+8(%rax), %r8
 ; CHECK-NEXT:    leaq stuff+8(%r9), %r9
 ; CHECK-NEXT:    callq toto
 ; CHECK-NEXT:    popq %rax
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
 entry:
   %tmp2 = bitcast <4 x float> %vx to <2 x i64>
   %and.i = and <2 x i64> %tmp2, <i64 8727373547504, i64 8727373547504>
   %tmp3 = bitcast <2 x i64> %and.i to <4 x i32>
   %index.sroa.0.0.vec.extract = extractelement <4 x i32> %tmp3, i32 0
   %idx.ext = sext i32 %index.sroa.0.0.vec.extract to i64
   %add.ptr = getelementptr inbounds i8, i8* bitcast ([256 x double]* @stuff to i8*), i64 %idx.ext

test/CodeGen/X86/pr29112.ll

 ; CHECK-NEXT:    vmovaps %xmm9, (%rsp)
 ; CHECK-NEXT:    vmovaps {{[0-9]+}}(%rsp), %xmm3 # 16-byte Reload
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    callq foo
 ; CHECK-NEXT:    vmovaps {{[0-9]+}}(%rsp), %xmm1 # 16-byte Reload
 ; CHECK-NEXT:    vaddps {{[0-9]+}}(%rsp), %xmm1, %xmm1 # 16-byte Folded Reload
 ; CHECK-NEXT:    vaddps %xmm0, %xmm1, %xmm0
 ; CHECK-NEXT:    addq $88, %rsp
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %a1 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 4, i32 20, i32 1, i32 17>
 
   %a2 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 4, i32 21, i32 1, i32 17>
   %a5 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 4, i32 20, i32 1, i32 27>
   %a6 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 3, i32 20, i32 1, i32 17>
   %a7 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 4, i32 21, i32 1, i32 17>
   %a8 = shufflevector <16 x float>%c1, <16 x float>%c2, <4 x i32> <i32 5, i32 20, i32 1, i32 19>

test/CodeGen/X86/pr30430.ll

 ; CHECK-NEXT:    vmovss %xmm9, {{[0-9]+}}(%rsp) # 4-byte Spill
 ; CHECK-NEXT:    vmovss %xmm10, {{[0-9]+}}(%rsp) # 4-byte Spill
 ; CHECK-NEXT:    vmovss %xmm11, {{[0-9]+}}(%rsp) # 4-byte Spill
 ; CHECK-NEXT:    vmovss %xmm12, {{[0-9]+}}(%rsp) # 4-byte Spill
 ; CHECK-NEXT:    vmovss %xmm13, {{[0-9]+}}(%rsp) # 4-byte Spill
 ; CHECK-NEXT:    vmovss %xmm14, (%rsp) # 4-byte Spill
 ; CHECK-NEXT:    movq %rbp, %rsp
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 entry:
   %__A.addr.i = alloca float, align 4
   %__B.addr.i = alloca float, align 4
   %__C.addr.i = alloca float, align 4
   %__D.addr.i = alloca float, align 4
   %__E.addr.i = alloca float, align 4
   %__F.addr.i = alloca float, align 4

test/CodeGen/X86/pr32241.ll

 ; CHECK-NEXT:  .LBB0_4: # %lor.end5
 ; CHECK-NEXT:    movb {{[0-9]+}}(%esp), %al # 1-byte Reload
 ; CHECK-NEXT:    andb $1, %al
 ; CHECK-NEXT:    movzbl %al, %ecx
 ; CHECK-NEXT:    movw %cx, %dx
 ; CHECK-NEXT:    movw %dx, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    addl $24, %esp
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi4:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 entry:
   %aa = alloca i16, align 2
   %bb = alloca i16, align 2
   %cc = alloca i16, align 2
   store i16 10959, i16* %aa, align 2
   store i16 -15498, i16* %bb, align 2
   store i16 19417, i16* %cc, align 2

test/CodeGen/X86/pr32256.ll

 ; CHECK-NEXT:    movb %al, %cl
 ; CHECK-NEXT:    movb %cl, (%esp) # 1-byte Spill
 ; CHECK-NEXT:    jmp .LBB0_2
 ; CHECK-NEXT:  .LBB0_2: # %land.end
 ; CHECK-NEXT:    movb (%esp), %al # 1-byte Reload
 ; CHECK-NEXT:    andb $1, %al
 ; CHECK-NEXT:    movb %al, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    addl $2, %esp
+; CHECK-NEXT:  .Lcfi1:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 entry:
   %b = alloca i8, align 1
   %0 = load i8, i8* @c, align 1
   %tobool = trunc i8 %0 to i1
   %lnot = xor i1 %tobool, true
   br i1 %lnot, label %land.rhs, label %land.end
 

test/CodeGen/X86/pr32282.ll

 ; X86-NEXT:    .cfi_adjust_cfa_offset 4
 ; X86-NEXT:    calll __divdi3
 ; X86-NEXT:    addl $16, %esp
 ; X86-NEXT:  .Lcfi5:
 ; X86-NEXT:    .cfi_adjust_cfa_offset -16
 ; X86-NEXT:    orl %eax, %edx
 ; X86-NEXT:    setne {{[0-9]+}}(%esp)
 ; X86-NEXT:    popl %eax
+; X86-NEXT:  .Lcfi6:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: foo:
 ; X64:       # BB#0:
 ; X64-NEXT:    movq {{.*}}(%rip), %rax
 ; X64-NEXT:    movabsq $3013716102212485120, %rcx # imm = 0x29D2DED3DE400000
 ; X64-NEXT:    andnq %rcx, %rax, %rcx
 ; X64-NEXT:    shrq $21, %rcx

test/CodeGen/X86/pr32329.ll

 ; X86-NEXT:    cmovnel %ecx, %esi
 ; X86-NEXT:    cmpl %edx, %edi
 ; X86-NEXT:    movl %ebp, var_50+4
 ; X86-NEXT:    movl %esi, var_50
 ; X86-NEXT:    setge var_205
 ; X86-NEXT:    imull %eax, %ebx
 ; X86-NEXT:    movb %bl, var_218
 ; X86-NEXT:    popl %esi
+; X86-NEXT:  .Lcfi8:
+; X86-NEXT:    .cfi_def_cfa_offset 16
 ; X86-NEXT:    popl %edi
+; X86-NEXT:  .Lcfi9:
+; X86-NEXT:    .cfi_def_cfa_offset 12
 ; X86-NEXT:    popl %ebx
+; X86-NEXT:  .Lcfi10:
+; X86-NEXT:    .cfi_def_cfa_offset 8
 ; X86-NEXT:    popl %ebp
+; X86-NEXT:  .Lcfi11:
+; X86-NEXT:    .cfi_def_cfa_offset 4
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: foo:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    movl {{.*}}(%rip), %eax
 ; X64-NEXT:    movsbl {{.*}}(%rip), %r9d
 ; X64-NEXT:    movzwl {{.*}}(%rip), %r8d
 ; X64-NEXT:    movl {{.*}}(%rip), %ecx

test/CodeGen/X86/pr32345.ll

 ; 6860-NEXT:    movb %al, %cl
 ; 6860-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; 6860-NEXT:    movb %cl, (%eax)
 ; 6860-NEXT:    leal -12(%ebp), %esp
 ; 6860-NEXT:    popl %esi
 ; 6860-NEXT:    popl %edi
 ; 6860-NEXT:    popl %ebx
 ; 6860-NEXT:    popl %ebp
+; 6860-NEXT:  .Lcfi6:
+; 6860-NEXT:    .cfi_def_cfa %esp, 4
 ; 6860-NEXT:    retl
 ;
 ; X64-LABEL: foo:
 ; X64:       # BB#0: # %bb
 ; X64-NEXT:    movzwl {{.*}}(%rip), %ecx
 ; X64-NEXT:    movzwl {{.*}}(%rip), %eax
 ; X64-NEXT:    xorw %cx, %ax
 ; X64-NEXT:    xorl %ecx, %eax

 ; 686-NEXT:    testb $32, %cl
 ; 686-NEXT:    jne .LBB0_2
 ; 686-NEXT:  # BB#1: # %bb
 ; 686-NEXT:    movl %eax, %edx
 ; 686-NEXT:  .LBB0_2: # %bb
 ; 686-NEXT:    movb %dl, (%eax)
 ; 686-NEXT:    movl %ebp, %esp
 ; 686-NEXT:    popl %ebp
+; 686-NEXT:  .Lcfi3:
+; 686-NEXT:    .cfi_def_cfa %esp, 4
 ; 686-NEXT:    retl
 bb:
   %tmp = alloca i64, align 8
   %tmp1 = load i16, i16* @var_22, align 2
   %tmp2 = zext i16 %tmp1 to i32
   %tmp3 = load i16, i16* @var_27, align 2
   %tmp4 = zext i16 %tmp3 to i32
   %tmp5 = xor i32 %tmp2, %tmp4

test/CodeGen/X86/pr32451.ll

 ; CHECK-NEXT:    andb $1, %bl
 ; CHECK-NEXT:    movzbl %bl, %edx
 ; CHECK-NEXT:    movl %edx, (%esp)
 ; CHECK-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; CHECK-NEXT:    calll jl_box_int32
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx # 4-byte Reload
 ; CHECK-NEXT:    movl %eax, (%ecx)
 ; CHECK-NEXT:    addl $16, %esp
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %ebx
+; CHECK-NEXT:  .Lcfi4:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 top:
   %3 = alloca i8***
   store volatile i8*** %1, i8**** %3
   %4 = call i8*** @julia.gc_root_decl()
   %5 = call i8**** @jl_get_ptls_states()
   %6 = bitcast i8**** %5 to i8***
   %7 = getelementptr i8**, i8*** %6, i64 3

test/CodeGen/X86/pr9743.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -disable-fp-elim -asm-verbose=0 | FileCheck %s
 
 define void @f() {
   ret void
 }
 
 ; CHECK:       .cfi_startproc
 ; CHECK-NEXT:  pushq
 ; CHECK-NEXT: :
 ; CHECK-NEXT:  .cfi_def_cfa_offset 16
 ; CHECK-NEXT: :
 ; CHECK-NEXT:  .cfi_offset %rbp, -16
 ; CHECK-NEXT:  movq    %rsp, %rbp
 ; CHECK-NEXT: :
 ; CHECK-NEXT:  .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:  popq    %rbp
+; CHECK-NEXT: :
+; CHECK-NEXT:  .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:  ret

test/CodeGen/X86/push-cfi-debug.ll

 ; CHECK: subl $8, %esp
 ; CHECK: .cfi_adjust_cfa_offset 8
 ; CHECK: pushl $4
 ; CHECK: .cfi_adjust_cfa_offset 4
 ; CHECK: pushl $3
 ; CHECK: .cfi_adjust_cfa_offset 4
 ; CHECK: calll stdfoo
 ; CHECK: .cfi_adjust_cfa_offset -8
-; CHECK: addl $20, %esp
+; CHECK: addl $8, %esp
 ; CHECK: .cfi_adjust_cfa_offset -8
+; CHECK: addl $12, %esp
+; CHECK: .cfi_def_cfa_offset 4
 define void @test1() #0 !dbg !4 {
 entry:
   tail call void @foo(i32 1, i32 2) #1, !dbg !10
   tail call x86_stdcallcc void @stdfoo(i32 3, i32 4) #1, !dbg !11
   ret void, !dbg !12
 }
 
 attributes #0 = { nounwind optsize }

test/CodeGen/X86/push-cfi-obj.ll

 ; RUN: llc < %s -mtriple=i686-pc-linux -filetype=obj | llvm-readobj -s -sr -sd | FileCheck %s -check-prefix=LINUX
 ; RUN: llc < %s -mtriple=i686-darwin-macosx10.7 -filetype=obj | llvm-readobj -sections | FileCheck -check-prefix=DARWIN %s
 
 ; On darwin, check that we manage to generate the compact unwind section
 ; DARWIN: Name: __compact_unwind
 ; DARWIN: Segment: __LD
 
 ; LINUX:         Name: .eh_frame
 ; LINUX-NEXT:    Type: SHT_PROGBITS (0x1)
 ; LINUX-NEXT:    Flags [ (0x2)
 ; LINUX-NEXT:      SHF_ALLOC (0x2)
 ; LINUX-NEXT:    ]
 ; LINUX-NEXT:    Address: 0x0
 ; LINUX-NEXT:    Offset: 0x68
-; LINUX-NEXT:    Size: 64
+; LINUX-NEXT:    Size: 72
 ; LINUX-NEXT:    Link: 0
 ; LINUX-NEXT:    Info: 0
 ; LINUX-NEXT:    AddressAlignment: 4
 ; LINUX-NEXT:    EntrySize: 0
 ; LINUX-NEXT:    Relocations [
 ; LINUX-NEXT:    ]
 ; LINUX-NEXT:    SectionData (
 ; LINUX-NEXT:      0000: 1C000000 00000000 017A504C 5200017C  |.........zPLR..||
 ; LINUX-NEXT:      0010: 08070000 00000000 1B0C0404 88010000  |................|
-; LINUX-NEXT:      0020: 1C000000 24000000 00000000 1D000000  |....$...........|
+; LINUX-NEXT:      0020: 24000000 24000000 00000000 1D000000  |$...$...........|
 ; LINUX-NEXT:      0030: 04000000 00410E08 8502420D 05432E10  |.....A....B..C..|
+; LINUX-NEXT:      0040: 540C0404 410C0508                    |T...A...|
 ; LINUX-NEXT:    )
 
 declare i32 @__gxx_personality_v0(...)
 declare void @good(i32 %a, i32 %b, i32 %c, i32 %d)
 
 define void @test() #0 personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 entry:
   invoke void @good(i32 1, i32 2, i32 3, i32 4)
           to label %continue unwind label %cleanup
 continue:
   ret void
-cleanup:  
+cleanup:
   landingpad { i8*, i32 }
      cleanup
   ret void
 }
 
 attributes #0 = { optsize "no-frame-pointer-elim"="true" }

test/CodeGen/X86/push-cfi.ll

 ; LINUX-NEXT: .cfi_adjust_cfa_offset 4
 ; LINUX-NEXT: pushl   $2
 ; LINUX-NEXT: Lcfi{{[0-9]+}}:
 ; LINUX-NEXT: .cfi_adjust_cfa_offset 4
 ; LINUX-NEXT: pushl   $1
 ; LINUX-NEXT: Lcfi{{[0-9]+}}:
 ; LINUX-NEXT: .cfi_adjust_cfa_offset 4
 ; LINUX-NEXT: call
-; LINUX-NEXT: addl $28, %esp
+; LINUX-NEXT: addl $16, %esp
 ; LINUX: .cfi_adjust_cfa_offset -16
+; LINUX: addl $12, %esp
 ; DARWIN-NOT: .cfi_escape
 ; DARWIN-NOT: pushl
 define void @test2_nofp() #0 personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 entry:
   call void @good(i32 1, i32 2, i32 3, i32 4)
   ret void
 }
 

test/CodeGen/X86/return-ext.ll

entry:
   %1 = zext i1 %0 to i32
   ret i32 %1
 
 ; The high 24 bits of %eax from a function returning i1 are undefined.
 ; CHECK-LABEL: use_i1:
 ; CHECK: call
 ; CHECK-NEXT: movzbl
 ; CHECK-NEXT: {{pop|add}}
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset {{4|8}}
 ; CHECK-NEXT: ret
 }
 
 define i32 @use_i8() {
 entry:
   %0 = call i8 @unsigned_i8();
   %1 = zext i8 %0 to i32
   ret i32 %1
 
 ; The high 24 bits of %eax from a function returning i8 are undefined.
 ; CHECK-LABEL: use_i8:
 ; CHECK: call
 ; CHECK-NEXT: movzbl
 ; CHECK-NEXT: {{pop|add}}
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset {{4|8}}
 ; CHECK-NEXT: ret
 }
 
 define i32 @use_i16() {
 entry:
   %0 = call i16 @unsigned_i16();
   %1 = zext i16 %0 to i32
   ret i32 %1
 
 ; The high 16 bits of %eax from a function returning i16 are undefined.
 ; CHECK-LABEL: use_i16:
 ; CHECK: call
 ; CHECK-NEXT: movzwl
 ; CHECK-NEXT: {{pop|add}}
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset {{4|8}}
 ; CHECK-NEXT: ret
 }

test/CodeGen/X86/rtm.ll

 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    pushq %rax
 ; X64-NEXT:  .Lcfi0:
 ; X64-NEXT:    .cfi_def_cfa_offset 16
 ; X64-NEXT:    movl %edi, {{[0-9]+}}(%rsp)
 ; X64-NEXT:    xabort $1
 ; X64-NEXT:    callq f1
 ; X64-NEXT:    popq %rax
+; X64-NEXT:  .Lcfi1:
+; X64-NEXT:    .cfi_def_cfa_offset 8
 ; X64-NEXT:    retq
 entry:
   %x.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4
   call void @llvm.x86.xabort(i8 1)
   call void @f1()
   ret void
 }
 

test/CodeGen/X86/setcc-lowering.ll

 ; KNL-32-NEXT:    cmovlw %dx, %si
 ; KNL-32-NEXT:    kmovw %esi, %k1
 ; KNL-32-NEXT:    kandw %k0, %k1, %k1
 ; KNL-32-NEXT:    kmovw %k1, %esi
 ; KNL-32-NEXT:    testw %si, %si
 ; KNL-32-NEXT:    jne .LBB1_1
 ; KNL-32-NEXT:  # BB#2: # %for_exit600
 ; KNL-32-NEXT:    popl %esi
+; KNL-32-NEXT:  .Lcfi2:
+; KNL-32-NEXT:    .cfi_def_cfa_offset 4
 ; KNL-32-NEXT:    retl
 allocas:
   br label %for_test11.preheader
 
 for_test11.preheader:                             ; preds = %for_test11.preheader, %allocas
   br i1 undef, label %for_loop599, label %for_test11.preheader
 
 for_loop599:                                      ; preds = %for_loop599, %for_test11.preheader

test/CodeGen/X86/statepoint-call-lowering.ll

 
 define i1 @test_relocate(i32 addrspace(1)* %a) gc "statepoint-example" {
 ; CHECK-LABEL: test_relocate
 ; Check that an ununsed relocate has no code-generation impact
 ; CHECK: pushq %rax
 ; CHECK: callq return_i1
 ; CHECK-NEXT: .Ltmp5:
 ; CHECK-NEXT: popq %rcx
+; CHECK-NEXT: .Lcfi11:
+; CHECK-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-NEXT: retq
 entry:
   %safepoint_token = tail call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 0, i32 0, i32 0, i32 addrspace(1)* %a)
   %call1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token,  i32 7, i32 7)
   %call2 = call zeroext i1 @llvm.experimental.gc.result.i1(token %safepoint_token)
   ret i1 %call2
 }
 

test/CodeGen/X86/statepoint-gctransition-call-lowering.ll

 
 define i1 @test_relocate(i32 addrspace(1)* %a) gc "statepoint-example" {
 ; CHECK-LABEL: test_relocate
 ; Check that an ununsed relocate has no code-generation impact
 ; CHECK: pushq %rax
 ; CHECK: callq return_i1
 ; CHECK-NEXT: .Ltmp4:
 ; CHECK-NEXT: popq %rcx
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-NEXT: retq
 entry:
   %safepoint_token = tail call token (i64, i32, i1 ()*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_i1f(i64 0, i32 0, i1 ()* @return_i1, i32 0, i32 1, i32 0, i32 0, i32 addrspace(1)* %a)
   %call1 = call i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %safepoint_token, i32 7, i32 7)
   %call2 = call zeroext i1 @llvm.experimental.gc.result.i1(token %safepoint_token)
   ret i1 %call2
 }
 

test/CodeGen/X86/statepoint-invoke.ll

entry:
   ; CHECK: callq some_call
   %sp1 = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i64 addrspace(1)* null, i64 addrspace(1)* undef)
            to label %normal_return unwind label %exceptional_return
 
 normal_return:
   ; CHECK-LABEL: %normal_return
   ; CHECK: xorl %eax, %eax
   ; CHECK-NEXT: popq
+  ; CHECK-NEXT: :
+  ; CHECK-NEXT: .cfi_def_cfa_offset 8
   ; CHECK-NEXT: retq
   %null.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp1, i32 13, i32 13)
   %undef.relocated = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %sp1, i32 14, i32 14)
   ret i64 addrspace(1)* %null.relocated
 
 exceptional_return:
   %landing_pad = landingpad token
           cleanup
entry:
   %c = inttoptr i64 15 to i64 addrspace(1)*
   ; CHECK: callq
   %sp = invoke token (i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...) @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64 0, i32 0, void (i64 addrspace(1)*)* @some_call, i32 1, i32 0, i64 addrspace(1)* %val1, i32 0, i32 5, i32 0, i32 -1, i32 0, i32 0, i32 0, i32 addrspace(1)* %aa, i64 addrspace(1)* %c)
            to label %normal_return unwind label %exceptional_return
 
 normal_return:
   ; CHECK: leaq
   ; CHECK-NEXT: popq
+  ; CHECK-NEXT: :
+  ; CHECK-NEXT: .cfi_def_cfa_offset 8
   ; CHECK-NEXT: retq
   %aa.rel = call coldcc i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token %sp, i32 13, i32 13)
   %aa.converted = bitcast i32 addrspace(1)* %aa.rel to i64 addrspace(1)*
   ret i64 addrspace(1)* %aa.converted
 
 exceptional_return:
   ; CHECK: movl $15
   ; CHECK-NEXT: popq
+  ; CHECK-NEXT: :
+  ; CHECK-NEXT: .cfi_def_cfa_offset 8
   ; CHECK-NEXT: retq
   %landing_pad = landingpad token
           cleanup
   %aa.rel2 = call coldcc i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token %landing_pad, i32 14, i32 14)
   ret i64 addrspace(1)* %aa.rel2
 }
 
 declare token @llvm.experimental.gc.statepoint.p0f_isVoidp1i64f(i64, i32, void (i64 addrspace(1)*)*, i32, i32, ...)
 declare token @llvm.experimental.gc.statepoint.p0f_p1i64p1i64f(i64, i32, i64 addrspace(1)* (i64 addrspace(1)*)*, i32, i32, ...)
 
 declare i64 addrspace(1)* @llvm.experimental.gc.relocate.p1i64(token, i32, i32)
 declare i32 addrspace(1)* @llvm.experimental.gc.relocate.p1i32(token, i32, i32)
 declare i64 addrspace(1)* @llvm.experimental.gc.result.p1i64(token)

test/CodeGen/X86/throws-cfi-fp.ll

+; RUN: llc %s -o - | FileCheck %s
+
+; ModuleID = 'throws-cfi-fp.cpp'
+source_filename = "throws-cfi-fp.cpp"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+$__clang_call_terminate = comdat any
+
+@_ZL11ShouldThrow = internal unnamed_addr global i1 false, align 1
+@_ZTIi = external constant i8*
+@str = private unnamed_addr constant [20 x i8] c"Threw an exception!\00"
+
+; Function Attrs: uwtable
+define void @_Z6throwsv() #0 personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+
+; CHECK-LABEL:   _Z6throwsv:
+; CHECK:         popq %rbp
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
+; CHECK-NEXT:    .LBB0_1:
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa %rbp, 16
+
+entry:
+  %.b5 = load i1, i1* @_ZL11ShouldThrow, align 1
+  br i1 %.b5, label %if.then, label %try.cont
+
+if.then:                                          ; preds = %entry
+  %exception = tail call i8* @__cxa_allocate_exception(i64 4)
+  %0 = bitcast i8* %exception to i32*
+  store i32 1, i32* %0, align 16
+  invoke void @__cxa_throw(i8* %exception, i8* bitcast (i8** @_ZTIi to i8*), i8* null)
+          to label %unreachable unwind label %lpad
+
+lpad:                                             ; preds = %if.then
+  %1 = landingpad { i8*, i32 }
+          catch i8* null
+  %2 = extractvalue { i8*, i32 } %1, 0
+  %3 = tail call i8* @__cxa_begin_catch(i8* %2)
+  %puts = tail call i32 @puts(i8* getelementptr inbounds ([20 x i8], [20 x i8]* @str, i64 0, i64 0))
+  invoke void @__cxa_rethrow()
+          to label %unreachable unwind label %lpad1
+
+lpad1:                                            ; preds = %lpad
+  %4 = landingpad { i8*, i32 }
+          cleanup
+  invoke void @__cxa_end_catch()
+          to label %eh.resume unwind label %terminate.lpad
+
+try.cont:                                         ; preds = %entry
+  ret void
+
+eh.resume:                                        ; preds = %lpad1
+  resume { i8*, i32 } %4
+
+terminate.lpad:                                   ; preds = %lpad1
+  %5 = landingpad { i8*, i32 }
+          catch i8* null
+  %6 = extractvalue { i8*, i32 } %5, 0
+  tail call void @__clang_call_terminate(i8* %6)
+  unreachable
+
+unreachable:                                      ; preds = %lpad, %if.then
+  unreachable
+}
+
+declare i8* @__cxa_allocate_exception(i64)
+
+declare void @__cxa_throw(i8*, i8*, i8*)
+
+declare i32 @__gxx_personality_v0(...)
+
+declare i8* @__cxa_begin_catch(i8*)
+
+declare void @__cxa_rethrow()
+
+declare void @__cxa_end_catch()
+
+; Function Attrs: noinline noreturn nounwind
+declare void @__clang_call_terminate(i8*)
+
+declare void @_ZSt9terminatev()
+
+; Function Attrs: nounwind
+declare i32 @puts(i8* nocapture readonly)
+
+attributes #0 = { "no-frame-pointer-elim"="true" }
+
+!llvm.dbg.cu = !{!2}
+!llvm.module.flags = !{!8, !9, !10}
+
+!2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 6.0.0 (https://github.com/llvm-mirror/clang.git 316ebefb7fff8ad324a08a694347500b6cd7c95f) (https://github.com/llvm-mirror/llvm.git dcae9be81fc17cdfbe989402354d3c8ecd0a2c79)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5)
+!3 = !DIFile(filename: "throws-cfi-fp.cpp", directory: "epilogue-dwarf/test")
+!4 = !{}
+!5 = !{}
+!8 = !{i32 2, !"Dwarf Version", i32 4}
+!9 = !{i32 2, !"Debug Info Version", i32 3}
+!10 = !{i32 1, !"wchar_size", i32 4}

test/CodeGen/X86/throws-cfi-no-fp.ll

+; RUN: llc %s -o - | FileCheck %s
+
+; ModuleID = 'throws-cfi-no-fp.cpp'
+source_filename = "throws-cfi-no-fp.cpp"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+$__clang_call_terminate = comdat any
+
+@_ZL11ShouldThrow = internal unnamed_addr global i1 false, align 1
+@_ZTIi = external constant i8*
+@str = private unnamed_addr constant [20 x i8] c"Threw an exception!\00"
+
+; Function Attrs: uwtable
+define void @_Z6throwsv() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+
+; CHECK-LABEL:   _Z6throwsv:
+; CHECK:         popq %rbx
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+; CHECK-NEXT:    .LBB0_1:
+; CHECK-NEXT:    :
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+
+entry:
+  %.b5 = load i1, i1* @_ZL11ShouldThrow, align 1
+  br i1 %.b5, label %if.then, label %try.cont
+
+if.then:                                          ; preds = %entry
+  %exception = tail call i8* @__cxa_allocate_exception(i64 4)
+  %0 = bitcast i8* %exception to i32*
+  store i32 1, i32* %0, align 16
+  invoke void @__cxa_throw(i8* %exception, i8* bitcast (i8** @_ZTIi to i8*), i8* null)
+          to label %unreachable unwind label %lpad
+
+lpad:                                             ; preds = %if.then
+  %1 = landingpad { i8*, i32 }
+          catch i8* null
+  %2 = extractvalue { i8*, i32 } %1, 0
+  %3 = tail call i8* @__cxa_begin_catch(i8* %2)
+  %puts = tail call i32 @puts(i8* getelementptr inbounds ([20 x i8], [20 x i8]* @str, i64 0, i64 0))
+  invoke void @__cxa_rethrow() #4
+          to label %unreachable unwind label %lpad1
+
+lpad1:                                            ; preds = %lpad
+  %4 = landingpad { i8*, i32 }
+          cleanup
+  invoke void @__cxa_end_catch()
+          to label %eh.resume unwind label %terminate.lpad
+
+try.cont:                                         ; preds = %entry
+  ret void
+
+eh.resume:                                        ; preds = %lpad1
+  resume { i8*, i32 } %4
+
+terminate.lpad:                                   ; preds = %lpad1
+  %5 = landingpad { i8*, i32 }
+          catch i8* null
+  %6 = extractvalue { i8*, i32 } %5, 0
+  tail call void @__clang_call_terminate(i8* %6)
+  unreachable
+
+unreachable:                                      ; preds = %lpad, %if.then
+  unreachable
+}
+
+declare i8* @__cxa_allocate_exception(i64)
+
+declare void @__cxa_throw(i8*, i8*, i8*)
+
+declare i32 @__gxx_personality_v0(...)
+
+declare i8* @__cxa_begin_catch(i8*)
+
+declare void @__cxa_rethrow()
+
+declare void @__cxa_end_catch()
+
+; Function Attrs: noinline noreturn nounwind
+declare void @__clang_call_terminate(i8*)
+
+declare void @_ZSt9terminatev()
+
+
+; Function Attrs: nounwind
+declare i32 @puts(i8* nocapture readonly)
+
+!llvm.dbg.cu = !{!2}
+!llvm.module.flags = !{!8, !9, !10}
+
+!2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 6.0.0 (https://github.com/llvm-mirror/clang.git 316ebefb7fff8ad324a08a694347500b6cd7c95f) (https://github.com/llvm-mirror/llvm.git dcae9be81fc17cdfbe989402354d3c8ecd0a2c79)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5)
+!3 = !DIFile(filename: "throws-cfi-no-fp.cpp", directory: "epilogue-dwarf/test")
+!4 = !{}
+!5 = !{}
+!8 = !{i32 2, !"Dwarf Version", i32 4}
+!9 = !{i32 2, !"Debug Info Version", i32 3}
+!10 = !{i32 1, !"wchar_size", i32 4}

test/CodeGen/X86/vector-sext.ll

 ; AVX1-NEXT:    vpinsrw $5, %edx, %xmm1, %xmm1
 ; AVX1-NEXT:    shlq $57, %rsi
 ; AVX1-NEXT:    sarq $63, %rsi
 ; AVX1-NEXT:    vpinsrw $6, %esi, %xmm1, %xmm1
 ; AVX1-NEXT:    shrq $7, %rbp
 ; AVX1-NEXT:    vpinsrw $7, %ebp, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
 ; AVX1-NEXT:    popq %rbx
+; AVX1-NEXT:  .Lcfi12:
+; AVX1-NEXT:    .cfi_def_cfa_offset 48
 ; AVX1-NEXT:    popq %r12
+; AVX1-NEXT:  .Lcfi13:
+; AVX1-NEXT:    .cfi_def_cfa_offset 40
 ; AVX1-NEXT:    popq %r13
+; AVX1-NEXT:  .Lcfi14:
+; AVX1-NEXT:    .cfi_def_cfa_offset 32
 ; AVX1-NEXT:    popq %r14
+; AVX1-NEXT:  .Lcfi15:
+; AVX1-NEXT:    .cfi_def_cfa_offset 24
 ; AVX1-NEXT:    popq %r15
+; AVX1-NEXT:  .Lcfi16:
+; AVX1-NEXT:    .cfi_def_cfa_offset 16
 ; AVX1-NEXT:    popq %rbp
+; AVX1-NEXT:  .Lcfi17:
+; AVX1-NEXT:    .cfi_def_cfa_offset 8
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: load_sext_16i1_to_16i16:
 ; AVX2:       # BB#0: # %entry
 ; AVX2-NEXT:    pushq %rbp
 ; AVX2-NEXT:  .Lcfi0:
 ; AVX2-NEXT:    .cfi_def_cfa_offset 16
 ; AVX2-NEXT:    pushq %r15

 ; AVX2-NEXT:    vpinsrw $5, %edx, %xmm1, %xmm1
 ; AVX2-NEXT:    shlq $57, %rsi
 ; AVX2-NEXT:    sarq $63, %rsi
 ; AVX2-NEXT:    vpinsrw $6, %esi, %xmm1, %xmm1
 ; AVX2-NEXT:    shrq $7, %rbp
 ; AVX2-NEXT:    vpinsrw $7, %ebp, %xmm1, %xmm1
 ; AVX2-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
 ; AVX2-NEXT:    popq %rbx
+; AVX2-NEXT:  .Lcfi12:
+; AVX2-NEXT:    .cfi_def_cfa_offset 48
 ; AVX2-NEXT:    popq %r12
+; AVX2-NEXT:  .Lcfi13:
+; AVX2-NEXT:    .cfi_def_cfa_offset 40
 ; AVX2-NEXT:    popq %r13
+; AVX2-NEXT:  .Lcfi14:
+; AVX2-NEXT:    .cfi_def_cfa_offset 32
 ; AVX2-NEXT:    popq %r14
+; AVX2-NEXT:  .Lcfi15:
+; AVX2-NEXT:    .cfi_def_cfa_offset 24
 ; AVX2-NEXT:    popq %r15
+; AVX2-NEXT:  .Lcfi16:
+; AVX2-NEXT:    .cfi_def_cfa_offset 16
 ; AVX2-NEXT:    popq %rbp
+; AVX2-NEXT:  .Lcfi17:
+; AVX2-NEXT:    .cfi_def_cfa_offset 8
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: load_sext_16i1_to_16i16:
 ; AVX512F:       # BB#0: # %entry
 ; AVX512F-NEXT:    kmovw (%rdi), %k1
 ; AVX512F-NEXT:    vpternlogd $255, %zmm0, %zmm0, %zmm0 {%k1} {z}
 ; AVX512F-NEXT:    vpmovdw %zmm0, %ymm0
 ; AVX512F-NEXT:    retq

 ; X32-SSE41-LABEL: sext_2i8_to_i32:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    pushl %eax
 ; X32-SSE41-NEXT:  .Lcfi0:
 ; X32-SSE41-NEXT:    .cfi_def_cfa_offset 8
 ; X32-SSE41-NEXT:    pmovsxbw %xmm0, %xmm0
 ; X32-SSE41-NEXT:    movd %xmm0, %eax
 ; X32-SSE41-NEXT:    popl %ecx
+; X32-SSE41-NEXT:  .Lcfi1:
+; X32-SSE41-NEXT:    .cfi_def_cfa_offset 4
 ; X32-SSE41-NEXT:    retl
 entry:
   %Shuf = shufflevector <16 x i8> %A, <16 x i8> undef, <2 x i32> <i32 0, i32 1>
   %Ex = sext <2 x i8> %Shuf to <2 x i16>
   %Bc = bitcast <2 x i16> %Ex to i32
   ret i32 %Bc
 }
 

test/CodeGen/X86/vector-shuffle-avx512.ll

 ; KNL32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL32-NEXT:    andl $-32, %esp
 ; KNL32-NEXT:    subl $32, %esp
 ; KNL32-NEXT:    vpbroadcastw {{.*#+}} ymm3 = [255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255]
 ; KNL32-NEXT:    vpblendvb %ymm3, %ymm2, %ymm0, %ymm0
 ; KNL32-NEXT:    vpblendvb %ymm3, 8(%ebp), %ymm1, %ymm1
 ; KNL32-NEXT:    movl %ebp, %esp
 ; KNL32-NEXT:    popl %ebp
+; KNL32-NEXT:  .Lcfi3:
+; KNL32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL32-NEXT:    retl
 entry:
   %0 = shufflevector <64 x i8> %A, <64 x i8> %W, <64 x i32>  <i32 64, i32 1, i32 66, i32 3, i32 68, i32 5, i32 70, i32 7, i32 72, i32 9, i32 74, i32 11, i32 76, i32 13, i32 78, i32 15, i32 80, i32 17, i32 82, i32 19, i32 84, i32 21, i32 86, i32 23, i32 88, i32 25, i32 90, i32 27, i32 92, i32 29, i32 94, i32 31, i32 96, i32 33, i32 98, i32 35, i32 100, i32 37, i32 102, i32 39, i32 104, i32 41, i32 106, i32 43, i32 108, i32 45, i32 110, i32 47, i32 112, i32 49, i32 114, i32 51, i32 116, i32 53, i32 118, i32 55, i32 120, i32 57, i32 122, i32 59, i32 124, i32 61, i32 126, i32 63>
   ret <64 x i8> %0
 }
 
 define <32 x i16> @test_mm512_mask_blend_epi16(<32 x i16> %A, <32 x i16> %W){
 ; SKX64-LABEL: test_mm512_mask_blend_epi16:

 ; SKX32-NEXT:    movl $-1431655766, %eax # imm = 0xAAAAAAAA
 ; SKX32-NEXT:    kmovd %eax, %k1
 ; SKX32-NEXT:    vpblendmw %zmm0, %zmm1, %zmm0 {%k1}
 ; SKX32-NEXT:    retl
 ;
 ; KNL32-LABEL: test_mm512_mask_blend_epi16:
 ; KNL32:       # BB#0: # %entry
 ; KNL32-NEXT:    pushl %ebp
-; KNL32-NEXT:  .Lcfi3:
-; KNL32-NEXT:    .cfi_def_cfa_offset 8
 ; KNL32-NEXT:  .Lcfi4:
+; KNL32-NEXT:    .cfi_def_cfa_offset 8
+; KNL32-NEXT:  .Lcfi5:
 ; KNL32-NEXT:    .cfi_offset %ebp, -8
 ; KNL32-NEXT:    movl %esp, %ebp
-; KNL32-NEXT:  .Lcfi5:
+; KNL32-NEXT:  .Lcfi6:
 ; KNL32-NEXT:    .cfi_def_cfa_register %ebp
 ; KNL32-NEXT:    andl $-32, %esp
 ; KNL32-NEXT:    subl $32, %esp
 ; KNL32-NEXT:    vpblendw {{.*#+}} ymm0 = ymm2[0],ymm0[1],ymm2[2],ymm0[3],ymm2[4],ymm0[5],ymm2[6],ymm0[7],ymm2[8],ymm0[9],ymm2[10],ymm0[11],ymm2[12],ymm0[13],ymm2[14],ymm0[15]
 ; KNL32-NEXT:    vpblendw {{.*#+}} ymm1 = mem[0],ymm1[1],mem[2],ymm1[3],mem[4],ymm1[5],mem[6],ymm1[7],mem[8],ymm1[9],mem[10],ymm1[11],mem[12],ymm1[13],mem[14],ymm1[15]
 ; KNL32-NEXT:    movl %ebp, %esp
 ; KNL32-NEXT:    popl %ebp
+; KNL32-NEXT:  .Lcfi7:
+; KNL32-NEXT:    .cfi_def_cfa %esp, 4
 ; KNL32-NEXT:    retl
 entry:
   %0 = shufflevector <32 x i16> %A, <32 x i16> %W, <32 x i32>  <i32 32, i32 1, i32 34, i32 3, i32 36, i32 5, i32 38, i32 7, i32 40, i32 9, i32 42, i32 11, i32 44, i32 13, i32 46, i32 15, i32 48, i32 17, i32 50, i32 19, i32 52, i32 21, i32 54, i32 23, i32 56, i32 25, i32 58, i32 27, i32 60, i32 29, i32 62, i32 31>
   ret <32 x i16> %0
 }
 
 define <16 x i32> @test_mm512_mask_blend_epi32(<16 x i32> %A, <16 x i32> %W){
 ; SKX64-LABEL: test_mm512_mask_blend_epi32:

test/CodeGen/X86/vector-shuffle-v1.ll

 ; AVX512F-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; AVX512F-NEXT:    kmovw %k0, (%rsp)
 ; AVX512F-NEXT:    movl (%rsp), %ecx
 ; AVX512F-NEXT:    movq %rcx, %rax
 ; AVX512F-NEXT:    shlq $32, %rax
 ; AVX512F-NEXT:    orq %rcx, %rax
 ; AVX512F-NEXT:    movq %rbp, %rsp
 ; AVX512F-NEXT:    popq %rbp
+; AVX512F-NEXT:  .Lcfi3:
+; AVX512F-NEXT:    .cfi_def_cfa %rsp, 8
 ; AVX512F-NEXT:    vzeroupper
 ; AVX512F-NEXT:    retq
 ;
 ; VL_BW_DQ-LABEL: shuf64i1_zero:
 ; VL_BW_DQ:       # BB#0:
 ; VL_BW_DQ-NEXT:    kmovq %rdi, %k0
 ; VL_BW_DQ-NEXT:    vpmovm2b %k0, %zmm0
 ; VL_BW_DQ-NEXT:    vpbroadcastb %xmm0, %zmm0

test/CodeGen/X86/wide-integer-cmp.ll

 ; CHECK-NEXT:    cmpl {{[0-9]+}}(%esp), %edx
 ; CHECK-NEXT:    sbbl {{[0-9]+}}(%esp), %esi
 ; CHECK-NEXT:    sbbl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    sbbl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    jge .LBB4_2
 ; CHECK-NEXT:  # BB#1: # %bb1
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi2:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .LBB4_2: # %bb2
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    movl $2, %eax
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:  .Lcfi4:
+; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 entry:
   %cmp = icmp slt i128 %a, %b
   br i1 %cmp, label %bb1, label %bb2
 bb1:
   ret i32 1
 bb2:
   ret i32 2

test/CodeGen/X86/x86-framelowering-trap.ll

 ; RUN: llc %s -o - | FileCheck %s
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK-LABEL: bar:
 ; CHECK: pushq
 ; CHECK: ud2
 ; CHECK-NEXT: popq
+; CHECK-NEXT: :
+; CHECK-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-NEXT: retq
 define void @bar() {
 entry:
   call void @callee()
   call void @llvm.trap()
   ret void
 }
 
 ; Function Attrs: noreturn nounwind
 declare void @llvm.trap()
 
 declare void @callee()

test/CodeGen/X86/x86-no_caller_saved_registers-preserve.ll

 ; CHECK-NEXT:    .cfi_offset %rdx, -16
 ; CHECK-NEXT:  .Lcfi2:
 ; CHECK-NEXT:    .cfi_offset %xmm1, -32
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $4, %eax
 ; CHECK-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1 # 16-byte Reload
 ; CHECK-NEXT:    popq %rdx
+; CHECK-NEXT:  .Lcfi3:
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   call void asm sideeffect "", "~{rax},~{rdx},~{xmm1},~{rdi},~{rsi},~{xmm0}"()
   ret i32 4
 }
 
 ;; Because "bar" has 'no_caller_saved_registers' attribute, function "foo"
 ;; doesn't need to preserve registers except for the arguments passed 
 ;; to "bar" (%ESI, %EDI and %XMM0).