BPF: use Source instead of ILP scheduler for selection dag

[yonghong-song]

Summary

This is to fix a bug reported by

https://github.com/iovisor/bpftrace/issues/1305

For the following initial selection dag,

t79: ch,glue = BPFISD::CALL t78, Constant:i64<4>, Register:i64 $r1, Register:i64 $r2, Register:i64 $r3, t78:1
t80: ch,glue = callseq_end t79, TargetConstant:i64<0>, TargetConstant:i64<0>, t79:1 
  t81: i64,ch,glue = CopyFromReg t80, Register:i64 $r0, t80:1
t82: i64,ch = load<(dereferenceable load 8 from %ir."struct cgroup.kn")> t81:1, FrameIndex:i64<2>, undef:i64
          t83: ch = lifetime.end<0 to -1> t82:1, TargetFrameIndex:i64<2>
        t86: ch = lifetime.start<0 to -1> t83, TargetFrameIndex:i64<1> 
      t89: ch = store<(store 8 into %ir.20)> t86, Constant:i64<0>, FrameIndex:i64<1>, undef:i64
    t91: ch = lifetime.start<0 to -1> t89, TargetFrameIndex:i64<0>
  t92: ch,glue = callseq_start t91, TargetConstant:i64<0>, TargetConstant:i64<0>
t93: ch,glue = CopyToReg t92, Register:i64 $r1, FrameIndex:i64<0>
t94: ch,glue = CopyToReg t93, Register:i64 $r2, Constant:i64<8>, t93:1
  t87: i64 = add t82, Constant:i64<8>
t95: ch,glue = CopyToReg t94, Register:i64 $r3, t87, t94:1
t96: ch,glue = BPFISD::CALL t95, Constant:i64<4>, Register:i64 $r1, Register:i64 $r2, Register:i64 $r3, t95:1
t97: ch,glue = callseq_end t96, TargetConstant:i64<0>, TargetConstant:i64<0>, t96:1

Note that node t89 depends on t86 which recursively depends on t82. This will enforce
load happens before store.

The optimized dag becomes

t79: ch,glue = BPFISD::CALL t78, Constant:i64<4>, Register:i64 $r1, Register:i64 $r2, Register:i64 $r3, t78:1
t80: ch,glue = callseq_end t79, TargetConstant:i64<0>, TargetConstant:i64<0>, t79:1
              t81: i64,ch,glue = CopyFromReg t80, Register:i64 $r0, t80:1
            t131: ch = TokenFactor t81:1, t130:1
          t83: ch = lifetime.end<0 to -1> t131, TargetFrameIndex:i64<2>
        t86: ch = lifetime.start<0 to -1> t83, TargetFrameIndex:i64<1>
        t128: ch = store<(store 8 into %ir.20)> t80, Constant:i64<0>, FrameIndex:i64<1>, undef:i64
      t129: ch = TokenFactor t86, t128
    t91: ch = lifetime.start<0 to -1> t129, TargetFrameIndex:i64<0>
  t92: ch,glue = callseq_start t91, TargetConstant:i64<0>, TargetConstant:i64<0>
t93: ch,glue = CopyToReg t92, Register:i64 $r1, FrameIndex:i64<0>
t94: ch,glue = CopyToReg t93, Register:i64 $r2, Constant:i64<8>, t93:1
  t87: i64 = add t130, Constant:i64<8>

...

t130: i64,ch = load<(dereferenceable load 8 from %ir."struct cgroup.kn")> t80, FrameIndex:i64<2>, undef:i64

For the optimized tag, store t128 now depends on t80 and load t130 also
depends on t80. Depending on how schedule dag scheduler works,
this opens possibility that load and store may be reordered.

Note the above optimized dag code is generated for both bpf and x86.
But x86 actually generates correct code. The reason is that x86 (and many other
architectures) is using "Source" selection dag scheduler which favors
source order in case of multiple choices. The default for bpf is ILP
which tries to optimize for instruction parallelism.

I disabled "Source" scheduler for x86 and used "ILP" scheduler,
the generated code still correct. This is because x86 has different lowering
code and it happens to work. Tweaking "ILP" might not be a robust idea
for BPF, so this patch enabled BPF to use "Source" scheduler.

There are two different ways to use "Source" scheduler.
One is to do setSchedulingPreference() in backend which will only
affect selection dag scheduling, and the other is to do
enableMachineScheduler() which enables selection dag "Source"
scheduling, but also enables "Machine Scheduler" phase and impacts
register allocation (favoring global-scope register collescing over
local collescing).

This patch uses setSchedulingPreference() for selection dag scheduler only
as benefit from machine scheduler and additional register allocation
is not clear to bpf backend at this point.

[ast]

ouch. sounds like we don't have much choice. Let's figure out what happened with test_verif_scale3 first, but looks like we have to land regardless.

[yonghong-song]

removed RFC tag as bpf selftests are solved.
using setSchedulingPreference() instead of enableMachineScheduler() to limit the scope
Regarding previous reported failures for bpf selftest test_verif_scale3.o:

• current llvm-project master: processed 997315 insns
• enableMachineScheduler() processed 1005967 insns (hacked verifier to increase to permit 2M insns), resulted from more instructions due to enabled machine scheduling phase (and possibly additional regalloc functionality, not verified).
• setSchedulingPreference() processed 998991 insns

[ast]

looks like x86 64-bit is ILP, so it's not niche.
Looks like Source gets the least amount of testing (judging by archs that use it).
RegPressure is probably 2nd most used after ILP.
It feels that we should fix it inside ILP instead.

[yonghong-song]

In D88525#2304656, @ast wrote:

looks like x86 64-bit is ILP, so it's not niche.
Looks like Source gets the least amount of testing (judging by archs that use it).
RegPressure is probably 2nd most used after ILP.
It feels that we should fix it inside ILP instead.

The default for x86 is Source. See

ScheduleDAGSDNodes* createDefaultScheduler(SelectionDAGISel *IS,
                                           CodeGenOpt::Level OptLevel) {
  const TargetLowering *TLI = IS->TLI;
  const TargetSubtargetInfo &ST = IS->MF->getSubtarget();

  // Try first to see if the Target has its own way of selecting a scheduler
  if (auto *SchedulerCtor = ST.getDAGScheduler(OptLevel)) {
    return SchedulerCtor(IS, OptLevel);
  }
  
  if (OptLevel == CodeGenOpt::None ||
      (ST.enableMachineScheduler() && ST.enableMachineSchedDefaultSched()) ||
      TLI->getSchedulingPreference() == Sched::Source)
    return createSourceListDAGScheduler(IS, OptLevel);
  if (TLI->getSchedulingPreference() == Sched::RegPressure)
    return createBURRListDAGScheduler(IS, OptLevel);
  if (TLI->getSchedulingPreference() == Sched::Hybrid)
    return createHybridListDAGScheduler(IS, OptLevel);
  if (TLI->getSchedulingPreference() == Sched::VLIW)
    return createVLIWDAGScheduler(IS, OptLevel);
  assert(TLI->getSchedulingPreference() == Sched::ILP &&
         "Unknown sched type!");
  return createILPListDAGScheduler(IS, OptLevel);
}

If ST.enableMachineScheduler() && ST.enableMachineSchedDefaultSched()
is true, it does not matter whether what target specifies SchedulingPreference,
it will use Source. For x86, it has enableMachineScheduler() and somehow
enableMachineSchedDefaultSched() is true, hence Source...

But this is only the default Subtarget scheduler, it is possible other
subtarget may use different schedulers, e.g., ILP.

Anyway, there is a discussion in https://bugs.llvm.org/show_bug.cgi?id=47591.
Possibly there is a real bug in selection dag. It would be the best the bug can
be fixed there.

[yonghong-song]

Looks like https://reviews.llvm.org/D91833 fixed the issue. Abandon this.

llvm/lib/Target/BPF/BPFISelLowering.cpp

if (BPFExpandMemcpyInOrder) {
     unsigned CommonMaxStores =
       STI.getSelectionDAGInfo()->getCommonMaxStoresPerMemFunc();
 
     MaxStoresPerMemset = MaxStoresPerMemsetOptSize = CommonMaxStores;
     MaxStoresPerMemcpy = MaxStoresPerMemcpyOptSize = CommonMaxStores;
     MaxStoresPerMemmove = MaxStoresPerMemmoveOptSize = CommonMaxStores;
   }
 
+  setSchedulingPreference(Sched::Source);
+
   // CPU/Feature control
   HasAlu32 = STI.getHasAlu32();
   HasJmp32 = STI.getHasJmp32();
   HasJmpExt = STI.getHasJmpExt();
 }
 
 bool BPFTargetLowering::isOffsetFoldingLegal(const GlobalAddressSDNode *GA) const {
   return false;

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-byte-size-1.ll

entry:
   %add = add i32 %4, %2, !dbg !43
   %add4 = add i32 %add, %6, !dbg !44
   %add5 = add i32 %add4, %8, !dbg !45
   ret i32 %add5, !dbg !46
 }
 
 ; CHECK:             r1 = 4
 ; CHECK:             r0 = 4
+; CHECK:             r2 = 4
+; CHECK:             r3 = 4
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 4
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 4
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=43
 ; CHECK:             .ascii  "0:1:0"                 # string offset=49
 ; CHECK:             .ascii  "0:1:1"                 # string offset=92
 ; CHECK:             .ascii  "0:1:2"                 # string offset=98

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-byte-size-2.ll

entry:
   call void @llvm.dbg.value(metadata i32 %5, metadata !30, metadata !DIExpression()), !dbg !31
   %add = add i32 %3, %1, !dbg !38
   %add3 = add i32 %add, %5, !dbg !39
   ret i32 %add3, !dbg !40
 }
 
 ; CHECK:             r1 = 8
 ; CHECK:             r0 = 4
+; CHECK:             r2 = 1
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 1
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=42
 ; CHECK:             .ascii  "0:1"                   # string offset=48
 ; CHECK:             .ascii  "0:1:0"                 # string offset=89
 ; CHECK:             .ascii  "0:1:1"                 # string offset=95

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-existence-1.ll

entry:
   %add = add i32 %3, %1, !dbg !44
   %add1 = add i32 %add, %5, !dbg !45
   %add2 = add i32 %add1, %8, !dbg !46
   ret i32 %add2, !dbg !47
 }
 
 ; CHECK:             r1 = 1
 ; CHECK:             r0 = 1
+; CHECK:             r2 = 1
+; CHECK:             r3 = 1
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 1
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 1
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_STRUCT(id = 2)
 ; CHECK:             .long   37                      # BTF_KIND_UNION(id = 7)
 ; CHECK:             .ascii  "s1"                    # string offset=1
 ; CHECK:             .ascii  "u1"                    # string offset=37
 ; CHECK:             .ascii  ".text"                 # string offset=64
 ; CHECK:             .ascii  "0:0"                   # string offset=70

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-lshift-1-bpfeb.ll

entry:
   %add = add i32 %4, %2, !dbg !43
   %add4 = add i32 %add, %6, !dbg !44
   %add5 = add i32 %add4, %8, !dbg !45
   ret i32 %add5, !dbg !46
 }
 
 ; CHECK-EB:          r1 = 32
 ; CHECK-EB:          r0 = 39
+; CHECK-EB:          r2 = 43
+; CHECK-EB:          r3 = 48
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK-EB:          r1 = 43
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK-EB:          r1 = 48
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=43
 ; CHECK:             .ascii  "0:1:0"                 # string offset=49
 ; CHECK:             .ascii  "0:1:1"                 # string offset=92
 ; CHECK:             .ascii  "0:1:2"                 # string offset=98

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-lshift-1.ll

entry:
   %add = add i32 %4, %2, !dbg !43
   %add4 = add i32 %add, %6, !dbg !44
   %add5 = add i32 %add4, %8, !dbg !45
   ret i32 %add5, !dbg !46
 }
 
 ; CHECK-EL:          r1 = 57
 ; CHECK-EL:          r0 = 53
+; CHECK-EL:          r2 = 48
+; CHECK-EL:          r3 = 32
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK-EL:          r1 = 48
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK-EL:          r1 = 32
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=43
 ; CHECK:             .ascii  "0:1:0"                 # string offset=49
 ; CHECK:             .ascii  "0:1:1"                 # string offset=92
 ; CHECK:             .ascii  "0:1:2"                 # string offset=98

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-rshift-1.ll

entry:
   %add = add i32 %4, %2, !dbg !43
   %add4 = add i32 %add, %6, !dbg !44
   %add5 = add i32 %add4, %8, !dbg !45
   ret i32 %add5, !dbg !46
 }
 
 ; CHECK:             r1 = 57
 ; CHECK:             r0 = 60
+; CHECK:             r2 = 59
+; CHECK:             r3 = 48
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 59
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 48
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=43
 ; CHECK:             .ascii  "0:1:0"                 # string offset=49
 ; CHECK:             .ascii  "0:1:1"                 # string offset=92
 ; CHECK:             .ascii  "0:1:2"                 # string offset=98

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-signedness-1.ll

entry:
   %add = add i32 %3, %1, !dbg !44
   %add1 = add i32 %add, %5, !dbg !45
   %add2 = add i32 %add1, %8, !dbg !46
   ret i32 %add2, !dbg !47
 }
 
 ; CHECK:             r1 = 1
 ; CHECK:             r0 = 0
+; CHECK:             r2 = 1
+; CHECK:             r3 = 0
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 1
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 0
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
+; CHECK-ALU64:       r0 += r3
+; CHECK-ALU32:       w0 += w3
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_STRUCT(id = 2)
 ; CHECK:             .long   37                      # BTF_KIND_UNION(id = 7)
 ; CHECK:             .ascii  "s1"                    # string offset=1
 ; CHECK:             .ascii  "u1"                    # string offset=37
 ; CHECK:             .ascii  ".text"                 # string offset=64
 ; CHECK:             .ascii  "0:0"                   # string offset=70

llvm/test/CodeGen/BPF/CORE/intrinsic-fieldinfo-signedness-2.ll

entry:
   call void @llvm.dbg.value(metadata i32 %6, metadata !40, metadata !DIExpression()), !dbg !41
   %add = add i32 %4, %2, !dbg !49
   %add3 = add i32 %add, %6, !dbg !50
   ret i32 %add3, !dbg !51
 }
 
 ; CHECK:             r1 = 1
 ; CHECK:             r0 = 0
+; CHECK:             r2 = 1
 ; CHECK-ALU64:       r0 += r1
 ; CHECK-ALU32:       w0 += w1
-; CHECK:             r1 = 1
-; CHECK-ALU64:       r0 += r1
-; CHECK-ALU32:       w0 += w1
+; CHECK-ALU64:       r0 += r2
+; CHECK-ALU32:       w0 += w2
 ; CHECK:             exit
 
 ; CHECK:             .long   1                       # BTF_KIND_UNION(id = 2)
 ; CHECK:             .ascii  "u1"                    # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=65
 ; CHECK:             .ascii  "0:1:0"                 # string offset=71
 ; CHECK:             .ascii  "0:1:1"                 # string offset=114
 ; CHECK:             .ascii  "0:1:2"                 # string offset=120

llvm/test/CodeGen/BPF/CORE/offset-reloc-fieldinfo-1.ll

entry:
   %retval.0 = trunc i64 %retval.0.in to i32, !dbg !37
   call void @llvm.lifetime.end.p0i8(i64 8, i8* nonnull %0) #5, !dbg !54
   ret i32 %retval.0, !dbg !54
 }
 
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK:             r{{[0-9]+}} <<= 51
+; CHECK:             r{{[0-9]+}} = 1
 ; CHECK64:           r{{[0-9]+}} s>>= 60
 ; CHECK64:           r{{[0-9]+}} >>= 60
-; CHECK32:           r{{[0-9]+}} >>= 60
 ; CHECK32:           r{{[0-9]+}} s>>= 60
-; CHECK:             r{{[0-9]+}} = 1
+; CHECK32:           r{{[0-9]+}} >>= 60
 
 ; CHECK:             .byte   115                     # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=30
 ; CHECK:             .ascii  "0:2"                   # string offset=73
 
 ; CHECK:             .long   16                      # FieldReloc
 ; CHECK-NEXT:        .long   30                      # Field reloc section string offset=30
 ; CHECK-NEXT:        .long   6
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
 ; CHECK-NEXT:        .long   0
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
 ; CHECK-NEXT:        .long   1
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
 ; CHECK-NEXT:        .long   4
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
-; CHECK-NEXT:        .long   5
+; CHECK-NEXT:        .long   3
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
 ; CHECK-NEXT:        .long   5
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   73
-; CHECK-NEXT:        .long   3
+; CHECK-NEXT:        .long   5
 
 ; Function Attrs: argmemonly nounwind willreturn
 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture) #1
 
 ; Function Attrs: nounwind readnone
 declare i16* @llvm.preserve.struct.access.index.p0i16.p0s_struct.ss(%struct.s*, i32, i32) #2
 
 ; Function Attrs: nounwind readnone

llvm/test/CodeGen/BPF/CORE/offset-reloc-fieldinfo-2-bpfeb.ll

sw.epilog:                                        ; preds = %entry, %sw.bb9, %sw.bb5, %sw.bb1, %sw.bb
   %retval.0.in = select i1 %tobool, i64 %shr15, i64 %shr, !dbg !79
   %retval.0 = trunc i64 %retval.0.in to i32, !dbg !41
   ret i32 %retval.0, !dbg !80
 }
 
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK-EB:          r{{[0-9]+}} <<= 41
+; CHECK:             r{{[0-9]+}} = 1
 ; CHECK64:           r{{[0-9]+}} s>>= 60
 ; CHECK64:           r{{[0-9]+}} >>= 60
-; CHECK32:           r{{[0-9]+}} >>= 60
 ; CHECK32:           r{{[0-9]+}} s>>= 60
-; CHECK:             r{{[0-9]+}} = 1
+; CHECK32:           r{{[0-9]+}} >>= 60
 
 ; CHECK:             .long   1                       # BTF_KIND_STRUCT(id = 2)
 ; CHECK:             .byte   115                     # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=30
 ; CHECK:             .ascii  "0:2"                   # string offset=36
 
 ; CHECK:             .long   16                      # FieldReloc
 ; CHECK-NEXT:        .long   30                      # Field reloc section string offset=30

 ; CHECK-NEXT:        .long   0
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
 ; CHECK-NEXT:        .long   4
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
-; CHECK-NEXT:        .long   5
+; CHECK-NEXT:        .long   3
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
 ; CHECK-NEXT:        .long   5
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
-; CHECK-NEXT:        .long   3
+; CHECK-NEXT:        .long   5
 
 ; Function Attrs: nounwind readnone
 declare i16* @llvm.preserve.struct.access.index.p0i16.p0s_struct.ss(%struct.s*, i32, i32) #1
 
 ; Function Attrs: nounwind readnone
 declare i32 @llvm.bpf.preserve.field.info.p0i16(i16*, i64) #1
 
 ; Function Attrs: nounwind readnone speculatable willreturn

llvm/test/CodeGen/BPF/CORE/offset-reloc-fieldinfo-2.ll

sw.epilog:                                        ; preds = %entry, %sw.bb9, %sw.bb5, %sw.bb1, %sw.bb
   %retval.0.in = select i1 %tobool, i64 %shr15, i64 %shr, !dbg !79
   %retval.0 = trunc i64 %retval.0.in to i32, !dbg !41
   ret i32 %retval.0, !dbg !80
 }
 
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK:             r{{[0-9]+}} = 4
 ; CHECK-EL:          r{{[0-9]+}} <<= 51
+; CHECK:             r{{[0-9]+}} = 1
 ; CHECK64:           r{{[0-9]+}} s>>= 60
 ; CHECK64:           r{{[0-9]+}} >>= 60
-; CHECK32:           r{{[0-9]+}} >>= 60
 ; CHECK32:           r{{[0-9]+}} s>>= 60
-; CHECK:             r{{[0-9]+}} = 1
+; CHECK32:           r{{[0-9]+}} >>= 60
 
 ; CHECK:             .long   1                       # BTF_KIND_STRUCT(id = 2)
 ; CHECK:             .byte   115                     # string offset=1
 ; CHECK:             .ascii  ".text"                 # string offset=30
 ; CHECK:             .ascii  "0:2"                   # string offset=36
 
 ; CHECK:             .long   16                      # FieldReloc
 ; CHECK-NEXT:        .long   30                      # Field reloc section string offset=30

 ; CHECK-NEXT:        .long   0
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
 ; CHECK-NEXT:        .long   4
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
-; CHECK-NEXT:        .long   5
+; CHECK-NEXT:        .long   3
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
 ; CHECK-NEXT:        .long   5
 ; CHECK-NEXT:        .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:        .long   2
 ; CHECK-NEXT:        .long   36
-; CHECK-NEXT:        .long   3
+; CHECK-NEXT:        .long   5
 
 ; Function Attrs: nounwind readnone
 declare i16* @llvm.preserve.struct.access.index.p0i16.p0s_struct.ss(%struct.s*, i32, i32) #1
 
 ; Function Attrs: nounwind readnone
 declare i32 @llvm.bpf.preserve.field.info.p0i16(i16*, i64) #1
 
 ; Function Attrs: nounwind readnone speculatable willreturn

llvm/test/CodeGen/BPF/CORE/offset-reloc-global-3.ll

entry:
   %call = tail call i32 @get_value(i32* %1) #3, !dbg !25
   ret i32 %call, !dbg !26
 }
 
 ; CHECK:              r2 = 4
 ; CHECK:              r1 += r2
 ; CHECK:              call get_value
 
-; CHECK:              .long   16                      # BTF_KIND_STRUCT(id = [[TID1:[0-9]+]])
+; CHECK:              .long   58                      # BTF_KIND_STRUCT(id = [[TID1:[0-9]+]])
 
 ; CHECK:              .ascii  ".text"                 # string offset=10
-; CHECK:              .ascii  "v3"                    # string offset=16
-; CHECK:              .ascii  "0:1"                   # string offset=23
+; CHECK:              .ascii  "v3"                    # string offset=58
+; CHECK:              .ascii  "0:1"                   # string offset=65
 
 ; CHECK:              .long   16                      # FieldReloc
 ; CHECK-NEXT:         .long   10                      # Field reloc section string offset=10
 ; CHECK-NEXT:         .long   1
 ; CHECK-NEXT:         .long   .Ltmp{{[0-9]+}}
 ; CHECK-NEXT:         .long   [[TID1]]
-; CHECK-NEXT:         .long   23
+; CHECK-NEXT:         .long   65
 ; CHECK-NEXT:         .long   0
 
 declare dso_local i32 @get_value(i32*) local_unnamed_addr #1
 
 ; Function Attrs: nounwind readnone
 declare i32* @llvm.preserve.struct.access.index.p0i32.p0s_struct.v3s(%struct.v3*, i32, i32) #2
 
 attributes #0 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "unsafe-fp-math"="false" "use-soft-float"="false" }

llvm/test/CodeGen/BPF/intrinsics.ll

 define i32 @ld_h(i8* %ctx, i8* %ctx2, i32 %foo) #0 {
   %1 = tail call i64 @llvm.bpf.load.half(i8* %ctx, i64 123) #2
   %2 = sext i32 %foo to i64
   %3 = tail call i64 @llvm.bpf.load.half(i8* %ctx2, i64 %2) #2
   %4 = add i64 %3, %1
   %5 = trunc i64 %4 to i32
   ret i32 %5
 ; CHECK-LABEL: ld_h:
-; CHECK-EL: r0 = *(u16 *)skb[r
 ; CHECK-EL: r0 = *(u16 *)skb[123]
-; CHECK-EB: r0 = *(u16 *)skb[r
+; CHECK-EL: r0 = *(u16 *)skb[r
 ; CHECK-EB: r0 = *(u16 *)skb[123]
+; CHECK-EB: r0 = *(u16 *)skb[r
 }
 
 declare i64 @llvm.bpf.load.half(i8*, i64) #1
 
 ; Function Attrs: nounwind uwtable
 define i32 @ld_w(i8* %ctx, i8* %ctx2, i32 %foo) #0 {
   %1 = tail call i64 @llvm.bpf.load.word(i8* %ctx, i64 123) #2
   %2 = sext i32 %foo to i64
   %3 = tail call i64 @llvm.bpf.load.word(i8* %ctx2, i64 %2) #2
   %4 = add i64 %3, %1
   %5 = trunc i64 %4 to i32
   ret i32 %5
 ; CHECK-LABEL: ld_w:
-; CHECK-EL: r0 = *(u32 *)skb[r
 ; CHECK-EL: r0 = *(u32 *)skb[123]
-; CHECK-EB: r0 = *(u32 *)skb[r
+; CHECK-EL: r0 = *(u32 *)skb[r
 ; CHECK-EB: r0 = *(u32 *)skb[123]
+; CHECK-EB: r0 = *(u32 *)skb[r
 }
 
 declare i64 @llvm.bpf.load.word(i8*, i64) #1
 
 define i32 @ld_pseudo() #0 {
 entry:
   %call = tail call i64 @llvm.bpf.pseudo(i64 2, i64 3)
   tail call void inttoptr (i64 4 to void (i64, i32)*)(i64 %call, i32 4) #2

llvm/test/CodeGen/BPF/lifetime.ll

+; RUN: llc -O2 -march=bpfel -mcpu=v3 < %s | FileCheck %s
+;
+; more info: https://bugs.llvm.org/show_bug.cgi?id=47591
+
+%printf_t = type { i64, i64 }
+
+define i64 @"kprobe:blk_update_request"(i8* %0) local_unnamed_addr section "s_kprobe:blk_update_request_1" {
+entry:
+  %"struct kernfs_node.parent" = alloca i64, align 8
+  %printf_args = alloca %printf_t, align 8
+  %"struct cgroup.kn" = alloca i64, align 8
+  %"struct cgroup_subsys_state.cgroup" = alloca i64, align 8
+  %"struct blkcg_gq.blkcg" = alloca i64, align 8
+  %"struct bio.bi_blkg" = alloca i64, align 8
+  %"struct request.bio" = alloca i64, align 8
+  %1 = getelementptr i8, i8* %0, i64 112
+  %2 = bitcast i8* %1 to i64*
+  %arg0 = load volatile i64, i64* %2, align 8
+  %3 = add i64 %arg0, 56
+  %4 = bitcast i64* %"struct request.bio" to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %4)
+  %probe_read = call i64 inttoptr (i64 10 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct request.bio", i32 8, i64 %3)
+  %5 = load i64, i64* %"struct request.bio", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %4)
+  %6 = add i64 %5, 72
+  %7 = bitcast i64* %"struct bio.bi_blkg" to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %7)
+  %probe_read1 = call i64 inttoptr (i64 11 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct bio.bi_blkg", i32 8, i64 %6)
+  %8 = load i64, i64* %"struct bio.bi_blkg", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %7)
+  %9 = add i64 %8, 40
+  %10 = bitcast i64* %"struct blkcg_gq.blkcg" to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %10)
+  %probe_read2 = call i64 inttoptr (i64 12 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct blkcg_gq.blkcg", i32 8, i64 %9)
+  %11 = load i64, i64* %"struct blkcg_gq.blkcg", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %10)
+  %12 = bitcast i64* %"struct cgroup_subsys_state.cgroup" to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %12)
+  %probe_read3 = call i64 inttoptr (i64 13 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct cgroup_subsys_state.cgroup", i32 8, i64 %11)
+  %13 = load i64, i64* %"struct cgroup_subsys_state.cgroup", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %12)
+  %14 = add i64 %13, 288
+  %15 = bitcast i64* %"struct cgroup.kn" to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %15)
+  %probe_read4 = call i64 inttoptr (i64 14 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct cgroup.kn", i32 8, i64 %14)
+  %16 = load i64, i64* %"struct cgroup.kn", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %15)
+  %17 = bitcast %printf_t* %printf_args to i8*
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %17)
+  %18 = add i64 %16, 8
+  %19 = bitcast i64* %"struct kernfs_node.parent" to i8*
+  %20 = getelementptr inbounds %printf_t, %printf_t* %printf_args, i64 0, i32 0
+  store i64 0, i64* %20, align 8
+
+; CHECK:      call 14
+; CHECK-NEXT: r{{[0-9]+}} = *(u64 *)(r10 - 24)
+; CHECK:      r{{[0-9]+}} = 0
+; CHECK-NEXT: *(u64 *)(r10 - 24) = r{{[0-9]+}}
+; CHECK-NOT:  r{{[0-9]+}} = *(u64 *)(r10 - 24)
+
+  call void @llvm.lifetime.start.p0i8(i64 -1, i8* nonnull %19)
+  %probe_read5 = call i64 inttoptr (i64 15 to i64 (i64*, i32, i64)*)(i64* nonnull %"struct kernfs_node.parent", i32 8, i64 %18)
+  %21 = load i64, i64* %"struct kernfs_node.parent", align 8
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %19)
+  %22 = getelementptr inbounds %printf_t, %printf_t* %printf_args, i64 0, i32 1
+  store i64 %21, i64* %22, align 8
+  %get_cpu_id = call i64 inttoptr (i64 15 to i64 ()*)()
+  %perf_event_output = call i64 inttoptr (i64 17 to i64 (i8*, i64, i64, %printf_t*, i64)*)(i8* %0, i64 1, i64 %get_cpu_id, %printf_t* nonnull %printf_args, i64 16)
+  call void @llvm.lifetime.end.p0i8(i64 -1, i8* nonnull %17)
+  ret i64 0
+}
+
+; Function Attrs: argmemonly nounwind willreturn
+declare void @llvm.lifetime.start.p0i8(i64 immarg %0, i8* nocapture %1) #1
+
+; Function Attrs: argmemonly nounwind willreturn
+declare void @llvm.lifetime.end.p0i8(i64 immarg %0, i8* nocapture %1) #1
+
+attributes #0 = { nounwind }
+attributes #1 = { argmemonly nounwind willreturn }

llvm/test/CodeGen/BPF/objdump_cond_op.ll

 ; CHECK: r0 <<= 1
 ; CHECK: goto +7 <LBB0_4>
 
 ; <label>:11:                                     ; preds = %8
   %12 = shl nsw i32 %10, 2
   br label %13
 
 ; CHECK-LABEL: <LBB0_2>:
-; CHECK: r3 = 0 ll
-; CHECK: r0 = *(u32 *)(r3 + 0)
 ; CHECK: r2 <<= 32
 ; CHECK: r2 >>= 32
+; CHECK: r3 = 0 ll
+; CHECK: r0 = *(u32 *)(r3 + 0)
 ; CHECK: if r1 == r2 goto +4 <LBB0_5>
 ; CHECK: r0 <<= 2
 
 ; <label>:13:                                     ; preds = %4, %11
   %14 = phi i32 [ %12, %11 ], [ %7, %4 ]
   store i32 %14, i32* @gbl, align 4
   br label %15
 ; CHECK-LABEL: <LBB0_4>:

llvm/test/CodeGen/BPF/objdump_intrinsics.ll

 define i32 @ld_h(i8* %ctx, i8* %ctx2, i32 %foo) #0 {
   %1 = tail call i64 @llvm.bpf.load.half(i8* %ctx, i64 123) #2
   %2 = sext i32 %foo to i64
   %3 = tail call i64 @llvm.bpf.load.half(i8* %ctx2, i64 %2) #2
   %4 = add i64 %3, %1
   %5 = trunc i64 %4 to i32
   ret i32 %5
 ; CHECK-LABEL: ld_h:
-; CHECK-EL: r0 = *(u16 *)skb[r
 ; CHECK-EL: r0 = *(u16 *)skb[123]
-; CHECK-EB: r0 = *(u16 *)skb[r
+; CHECK-EL: r0 = *(u16 *)skb[r
 ; CHECK-EB: r0 = *(u16 *)skb[123]
+; CHECK-EB: r0 = *(u16 *)skb[r
 }
 
 declare i64 @llvm.bpf.load.half(i8*, i64) #1
 
 ; Function Attrs: nounwind uwtable
 define i32 @ld_w(i8* %ctx, i8* %ctx2, i32 %foo) #0 {
   %1 = tail call i64 @llvm.bpf.load.word(i8* %ctx, i64 123) #2
   %2 = sext i32 %foo to i64
   %3 = tail call i64 @llvm.bpf.load.word(i8* %ctx2, i64 %2) #2
   %4 = add i64 %3, %1
   %5 = trunc i64 %4 to i32
   ret i32 %5
 ; CHECK-LABEL: ld_w:
-; CHECK-EL: r0 = *(u32 *)skb[r
 ; CHECK-EL: r0 = *(u32 *)skb[123]
-; CHECK-EB: r0 = *(u32 *)skb[r
+; CHECK-EL: r0 = *(u32 *)skb[r
 ; CHECK-EB: r0 = *(u32 *)skb[123]
+; CHECK-EB: r0 = *(u32 *)skb[r
 }
 
 declare i64 @llvm.bpf.load.word(i8*, i64) #1
 
 define i32 @ld_pseudo() #0 {
 entry:
   %call = tail call i64 @llvm.bpf.pseudo(i64 2, i64 3)
   tail call void inttoptr (i64 4 to void (i64, i32)*)(i64 %call, i32 4) #2

llvm/test/CodeGen/BPF/objdump_nop.ll

 ; RUN: llc -march=bpfel -filetype=obj -o - %s | llvm-objdump -d - | FileCheck %s
 ;
 ; Source:
 ;   int test() {
-;     asm volatile("r0 = r0" ::);
+;     asm volatile("r0 = r0" :::"r0");
 ;     return 0;
 ;   }
 ; Compilation flag:
 ;   clang -target bpf -O2 -S -emit-llvm t.c
 
 ; Function Attrs: nounwind
 define dso_local i32 @test() local_unnamed_addr {
 entry:
-  tail call void asm sideeffect "r0 = r0", ""()
+  tail call void asm sideeffect "r0 = r0", "~{r0}"()
   ret i32 0
 }
 ; CHECK-LABEL: test
 ; CHECK:       r0 = r0
 ; CHECK:       r0 = 0

llvm/test/CodeGen/BPF/remove_truncate_3.ll

 ; <label>:28:                                     ; preds = %4, %23, %11
   %29 = phi i32 [ 3, %4 ], [ 2, %11 ], [ %27, %23 ]
   ret i32 %29
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define i32 @rol32(i32, i32) local_unnamed_addr #1 {
   %3 = shl i32 %0, %1
-; CHECK: r1 <<= 32
-; CHECK: r1 >>= 32
+; CHECK: r{{[0-9]+}} <<= 32
+; CHECK: r{{[0-9]+}} >>= 32
   %4 = sub i32 0, %1
   %5 = and i32 %4, 31
   %6 = lshr i32 %0, %5
-; CHECK: r0 <<= 32
-; CHECK: r0 >>= 32
+; CHECK: r{{[0-9]+}} <<= 32
+; CHECK: r{{[0-9]+}} >>= 32
   %7 = or i32 %6, %3
   ret i32 %7
 }
 
 attributes #0 = { norecurse nounwind }
 attributes #1 = { norecurse nounwind readnone }

llvm/test/CodeGen/BPF/rodata_5.ll

entry:
   %v1 = alloca [3 x i8], align 1
   %v1.sub = getelementptr inbounds [3 x i8], [3 x i8]* %v1, i64 0, i64 0
   call void @llvm.lifetime.start.p0i8(i64 3, i8* nonnull %v1.sub)
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 1 dereferenceable(3) %v1.sub, i8* nonnull align 1 dereferenceable(3) getelementptr inbounds (%struct.t, %struct.t* @__const.test.v, i64 0, i32 0), i64 3, i1 false)
   call void @foo(i8* nonnull %v1.sub)
   call void @llvm.lifetime.end.p0i8(i64 3, i8* nonnull %v1.sub)
   ret i32 0
 }
-; CHECK-NOT:    w{{[0-9]+}} = *(u16 *)
 ; CHECK-NOT:    w{{[0-9]+}} = *(u8 *)
-; CHECK:        *(u16 *)(r10 - 4) = w{{[0-9]+}}
+; CHECK-NOT:    w{{[0-9]+}} = *(u16 *)
 ; CHECK:        *(u8 *)(r10 - 2) = w{{[0-9]+}}
+; CHECK:        *(u16 *)(r10 - 4) = w{{[0-9]+}}
 
 ; Function Attrs: argmemonly nounwind willreturn
 declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture)
 
 ; Function Attrs: argmemonly nounwind willreturn
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg)
 
 declare dso_local void @foo(i8*) local_unnamed_addr
 
 ; Function Attrs: argmemonly nounwind willreturn
 declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture)