[BPF] Attribute preserve_static_offset for structs

[eddyz87]

Summary

This commit adds a new BPF specific structure attribte
__attribute__((preserve_static_offset)) and a pass to deal with it.

This attribute may be attached to a struct or union declaration, where
it notifies the compiler that this structure is a "context" structure.
The following limitations apply to context structures:

• runtime environment might patch access to the fields of this type by updating the field offset;
  
  BPF verifier limits access patterns allowed for certain data types. E.g. struct __sk_buff and struct bpf_sock_ops. For these types only LD/ST <reg> <static-offset> memory loads and stores are allowed.
  
  This is so because offsets of the fields of these structures do not match real offsets in the running kernel. During BPF program load/verification loads and stores to the fields of these types are rewritten so that offsets match real offsets. For this rewrite to happen static offsets have to be encoded in the instructions.
  
  See kernel/bpf/verifier.c:convert_ctx_access function in the Linux kernel source tree for details.

• runtime environment might disallow access to the field of the type through modified pointers.
  
  During BPF program verification a tag PTR_TO_CTX is tracked for register values. In case if register with such tag is modified BPF programs are not allowed to read or write memory using register. See kernel/bpf/verifier.c:check_mem_access function in the Linux kernel source tree for details.

Access to the structure fields is translated to IR as a sequence:

• (load (getelementptr %ptr %offset)) or
• (store (getelementptr %ptr %offset))

During instruction selection phase such sequences are translated as a
single load instruction with embedded offset, e.g. LDW %ptr, %offset,
which matches access pattern necessary for the restricted
set of types described above (when %offset is static).

Multiple optimizer passes might separate these instructions, this
includes:

• SimplifyCFGPass (sinking)
• InstCombine (sinking)
• GVN (hoisting)

The preserve_static_offset attribute marks structures for which the
following transformations happen:

• at the early IR processing stage:
  • (load (getelementptr ...)) replaced by call to intrinsic llvm.bpf.getelementptr.and.load;
  • (store (getelementptr ...)) replaced by call to intrinsic llvm.bpf.getelementptr.and.store;
• at the late IR processing stage this modification is undone.

Such handling prevents various optimizer passes from generating
sequences of instructions that would be rejected by BPF verifier.

The attribute((preserve_static_offset)) has a priority over
attribute((preserve_access_index)). When preserve_access_index
attribute is present preserve access index transformations are not
applied.

This addresses the issue reported by the following thread:

https://lore.kernel.org/bpf/CAA-VZPmxh8o8EBcJ=[e-mail]/T/#m4b9ce2ce73b34f34172328f975235fc6f19841b6

[eddyz87]

• Added folding for GEP chains with constant indices:
  • inline anonymous unions work as expected
  • type casts to char* with consequitive array access work as expected
• Moved context rewrite to a late callback function processing pipeline stage:
  • necessary to run the rewrite after loop unrolling, as rewrite can't handle non-constant indices for non structural GEP chains

[eddyz87]

• added GVN pass after context rewrite to handle cases clobbered by marker calls
• added marker simplification and context rewrite passes for opt pipeline
• more test cases

[eddyz87]

Merged simplify and rewrite passes, various adjustments to unclobber CSE and GVN opportunities

[eddyz87]

Merge getelementptr and get.and.load/store calls for function inlining

[eddyz87]

Force doesNotAccessMemory for context marker function

[eddyz87]

Moved readonly/writeonly attrs as properties of call site to handle volatile load and stores

[eddyz87]

Rebase, replaced .c based back-end changes by .ll based ones.

[eddyz87]

• renamed llvm.bpf.context.marker to llvm.context.marker.bpf (to avoid issues with intrinsic being looked up in the wrong table and not getting an ID / Attrs for CallInst);
• use new pass manager for tests.

[eddyz87]

• removed lots of uses of auto
• removed usage of std::set

[eddyz87]

Rebase.

[eddyz87]

Generate immarg for index arguments of gep.and.load, gep.and.store.

[eddyz87]

Hi Yonghong,

Could you please take a look at this revision? I tried to describe the mechanics in the description / commit message and comments. Below are details regarding testing.
I tested these changes using a Linux Kernel special branch that declares decl_tag("ctx") for relevant functions (available here).
While comparing object files generated without decl_tag("ctx") and with decl_tag("ctx") I found some differences in 5 tests. (I use "with ctx" and "without ctx" below as a shorthand).

bpf_flow.bpf.o

Without ctx: 2713 instructions
With ctx: 2711 instructions

Difference: 2 load instructions appear in different BB and this requires two additional register to register assignments in "without ctx" case.
Difference is caused by GVNPass::PerformLoadPRE sub-pass and disappears if -mllvm -enable-load-pre=false option is passed. As the name implies this sub-pass operates only on load instructions.

connect4_prog.bpf.o

Without ctx: 351 instructions
With ctx: 351 instructions

Difference: one redundant load instruction is put in it's own basic block when compiled without ctx.

c
static __inline int set_notsent_lowat(struct bpf_sock_addr *ctx)
{ ... ctx->type ... }

int connect_v4_prog(struct bpf_sock_addr *ctx)
{
	...
	if (set_notsent_lowat(ctx))
		return 0;

	if (ctx->type ...) // ctx->type load is partially redundat after inlining
		return 0;
}

In the without ctx this is done by a part of JumpThreadingPass:

c++
/// simplifyPartiallyRedundantLoad - If LoadI is an obviously partially
/// redundant load instruction, eliminate it by replacing it with a PHI node.
/// This is an important optimization that encourages jump threading, and needs
/// to be run interlaced with other jump threading tasks.
bool JumpThreadingPass::simplifyPartiallyRedundantLoad(LoadInst *LoadI)

test_cls_redirect.bpf.o

Without ctx: 1616 instructions
With ctx: 1615 instructions

Difference: some differences in basic blocks placement. IR basic block structure is identical up until 'Branch Probability Basic Block Placement' machine pass.
Before that in without ctx case there is one additional basic block, which is introduced by GVNPass on process_icmpv4 function. Differences caused by GVNPass::PerformLoadPRE sub-pass which calls
llvm::SplitCriticalEdge that introduces the basic block.

test_misc_tcp_hdr_options.bpf.o

Without ctx: 611 instructions
With ctx: 612 instructions

Difference: one more load instruction generated in with ctx case. Happens in the following code snippet:

c
unsigned int nr_data = 0;
...
static int check_active_hdr_in(struct bpf_sock_ops *skops)
{
	...
	if (... < skops->skb_len)  // (1)
		nr_data++;
	...
	if (... == skops->skb_len) // (2)
		nr_pure_ack++;
	...
}

When compiled without ctx EarlyCSEPass reuses skops->skb_len computed at (1) for point (2). However, when compiled with ctx this does not happen. This happens because clang is not sure if skops, passed as parameter to getelementptr.and.load shares memory with nr_data. The following modification removes all differences:

c
static int check_active_hdr_in(struct bpf_sock_ops * restrict skops)

test_tcpnotify_kern.bpf.o

Without ctx: 57 instructions
With ctx: 58 instructions

Difference: one more store instruction generated, CFG structure differs slightly.
Difference is caused by SimplifyCFGPass, difference disappears if -mllvm -sink-common-insts=false option is supplied. The following IR snippet is the cause:

llvm
...
if.then:                                          ; preds = %entry
  %1 = getelementptr inbounds %struct.bpf_sock_ops, ptr %skops, i64 0, i32 1
  store i32 -1, ptr %1, align 4, !tbaa !240
  br label %cleanup30
...
...
sw.epilog:                                        ; preds = %if.end, ...
  %rv.0 = phi i32 [ -1, %sw.default ], ...
  %7 = getelementptr inbounds %struct.bpf_sock_ops, ptr %skops, i64 0, i32 1
  store i32 %rv.0, ptr %7, align 4, !tbaa !240
  br label %cleanup30, !dbg !278

cleanup30:                                        ; preds = %sw.epilog, %if.then
  %retval.0 = phi i32 [ 0, %if.then ], [ 1, %sw.epilog ], !dbg !226
  ret i32 %retval.0

Without ctx SimplifyCFGPass sinks getelementptr and store instructions to cleanup30 BB:

llvm
cleanup30:                                        ; preds = %sw.bb19, ...
  %rv.0.sink = phi i32 [ -1, %entry ], ...
  %retval.0 = phi i32 [ 0, %entry ], ...
  %6 = getelementptr inbounds %struct.bpf_sock_ops, ptr %skops, i64 0, i32 1, !dbg !226
  store i32 %rv.0.sink, ptr %6, align 4, !dbg !226, !tbaa !270
  ret i32 %retval.0, !dbg !271

For ctx case snippet looks as follows:

llvm
if.then:                                          ; preds = %entry
  tail call void ... @llvm.bpf.getelementptr.and.store.i32
    (i32 -1, ptr nonnull writeonly elementtype(%struct.bpf_sock_ops) %skops, ...
  br label %cleanup30
...
...
sw.epilog:                                        ; preds = %if.end ...
  %rv.0 = phi i32 [ -1, %sw.default ], ...
  call void ... @llvm.bpf.getelementptr.and.store.i32
    (i32 %rv.0, ptr nonnull writeonly elementtype(%struct.bpf_sock_ops) %skops, ...
  br label %cleanup30

cleanup30:                                        ; preds = %sw.epilog, %if.then
  %retval.0 = phi i32 [ 0, %if.then ], [ 1, %sw.epilog ], !dbg !228
  ret i32 %retval.0, !dbg !279

Note that in if.then the instruction is tail call while for sw.epilog the instruction is call. These instructions are considered not to be the same by SimplifyCFG.cpp:canSinkInstructions. I have an old merge request that fixes this: https://reviews.llvm.org/D134743 . Need to rebase it and ping someone to review.

[eddyz87]

• Removed special Sema processing and btf_decl_tag("ctx") prioritization in CGExpr, preserve access index calls are now handled by BPFContextMarker transformation.
• Correctness fixes for isPointerOperand().

[yonghong-song]

llvm/include/llvm/IR/Intrinsics.td

1966

Is it possible to make this builtin as BPF specific one?

[eddyz87]

llvm/include/llvm/IR/Intrinsics.td

1966

Currently llvm::Intrinsic::context_marker_bpf gets defined in llvm/IR/Intrinsics.h (via include of generated file IntrinsicEnums.inc, same as preserve_{struct,union,array}_access_index).

BPF specific intrinsics are defined in llvm/IR/IntrinsicsBPF.h (generated directly w/o .inc intermediary).

Thus, if I move context_marker_bpf to IntrinsicsBPF.td I would have to include IntrinsicsBPF.h in CGExpr.cpp. However, I don't see any target specific includes in that file.

[eddyz87]

git-clang-format fixes.

[yonghong-song]

I will try to review other parts of code in the next few days.

llvm/include/llvm/IR/Intrinsics.td

1966

I went through the related clang code and indeed found it is hard to get a BPF target defined function in CGF or CGM. On the other hand, we can consider this new builtin under the umbrella "Intrinsics that are used to preserve debug information". Maybe we can rename the intrinsic name
to 'int_preserve_context_marker'? The goal of this builtin to preserve certain load/store which should be immune from optimizations. I try to generalize this so your function name 'wrapWithBPFContextMarker' can be renamed to
'wrapWithContextMarker'. There is no need to mention BPF any more.

In the commit message, you can mention that
similar to int_preserve_array_access_index, int_preserve_context_marker is only implemented in BPF backend. But other architectures can implement processing these intrinsics if they want to achieve some results similar to bpf backend.

WDYT?

[eddyz87]

llvm/include/llvm/IR/Intrinsics.td

1966

I can rename these things, but tbh I don't think this functionality would be useful anywhere outside BPF, thus such renaming would be kind-of deceptive (and in case it would be useful, the renaming could be done at the time of second use).

---

Something more generic might look like !btf_decl_tag <str-value> metadata node attached to something. However, in the current form this would require transfer of such decl tag from type to function parameters and variables, e.g.:

struct foo {  ... } __attribute__((btf_decl_tag("ctx")));
void bar(struct foo *p) { ... }

During code-gen for bar use rule like "if function parameter has type annotated with btf_decl_tag, attach metadata to such parameter":

define void @bar(ptr %p !btf_decl_tag !1) { ... }
!1 = { "ctx" }

Such rule looks a bit weird:

• tag is transferred from type to it's usage
• what usages should be annotated? we care about function parameters but from generic point of view allocas or field accesses should be annotated as well.

---

The same metadata approach but with "ctx" attributes annotating function parameters (as you suggested originally, if I recall correctly) seems to be most generic and least controversial of all, e.g.:

void bar(struct foo *p __attribute__((btf_decl_tag("ctx")))) { ... }

Converted to:

define void @bar(ptr %p !btf_decl_tag !1) { ... }
!1 = { "ctx" }

However, this is less ergonomic for BPF, because user will have to annotate function parameters. (On the other hand, no changes in the kernel headers would be necessary).

[yonghong-song]

I can rename these things, but tbh I don't think this functionality would be useful anywhere outside BPF, thus such renaming would be kind-of deceptive (and in case it would be useful, the renaming could be done at the time of second use).

Then let us keep this for now. We can decide what to do after clang frontend people reviewed this code.

[ast]

I can rename these things, but tbh I don't think this functionality would be useful anywhere outside BPF, thus such renaming would be kind-of deceptive (and in case it would be useful, the renaming could be done at the time of second use).

I agree that it's not useful outside of BPF, but it's useful outside of 'ctx'. I think 'preserve_constant_field_offset' would be more accurate description of the restriction.
We can expand in the doc that it's a constant offset when field of the struct is accessed.

Also instead of btf_tag it would be better to add another builtin similar to preserve_access_index.
Currently we add attribute((preserve_access_index)) to trigger CO-RE.
This one will be a new attribute((preserve_constant_field_offset)) that will be specified manually either in uapi/bpf.h or in vmlinux.h on some structs
and it will have precedence over preserve_access_index, so
(attribute((preserve_access_index)), apply_to = record) in vmlinux.h will be ignored on such structs.
Otherwise it's a bit odd that special names inside btf_tag have stronger rules than other attribute((preserve_access_index)).

[eddyz87]

In D133361#4628951, @ast wrote:

I agree that it's not useful outside of BPF, but it's useful outside of 'ctx'. I think 'preserve_constant_field_offset' would be more accurate description of the restriction.
We can expand in the doc that it's a constant offset when field of the struct is accessed.

Also instead of btf_tag it would be better to add another builtin similar to preserve_access_index.
Currently we add attribute((preserve_access_index)) to trigger CO-RE.
This one will be a new attribute((preserve_constant_field_offset)) that will be specified manually either in uapi/bpf.h or in vmlinux.h on some structs

This makes sense, I'll adjust the implementation to use new attribute (also will try to use metadata instead of intrinsic call, replace by intrinsic on the back-end side).

and it will have precedence over preserve_access_index, so
(attribute((preserve_access_index)), apply_to = record) in vmlinux.h will be ignored on such structs.
Otherwise it's a bit odd that special names inside btf_tag have stronger rules than other attribute((preserve_access_index)).

Note on current propagation logic: whenever there is an expression E of type T, where T is a struct annotated with btf_decl_tag("ctx"), all usages of E are traversed recursively visiting getelementptr and calls to preserve_{struct,union,array}_index, the latter are replaced by getelementptr. E.g.:

#define __pai __attribute__((preserve_access_context));
#define __ctx __attribute__((btf_decl_tag("ctx")))
struct bar { int a; int b; } __pai;
struct foo {
  struct bar b;
} __ctx __pai;
... struct foo f; ...
... f.b.bb ...

The call to preserve_struct_index generated for bb in f.b.bb would be replaced by getelementptr and later by getelementptr.and.load.
However, context structures don't have nested structures at the moment. The list of context structures is:

• __sk_buff
• bpf_cgroup_dev_ctx
• bpf_sk_lookup
• bpf_sock
• bpf_sock_addr
• bpf_sock_ops
• bpf_sockopt
• bpf_sysctl
• sk_msg_md
• sk_reuseport_md
• xdp_md

[ast]

In D133361#4629081, @eddyz87 wrote:

Note on current propagation logic: whenever there is an expression E of type T, where T is a struct annotated with btf_decl_tag("ctx"), all usages of E are traversed recursively visiting getelementptr and calls to preserve_{struct,union,array}_index, the latter are replaced by getelementptr. E.g.:

#define __pai __attribute__((preserve_access_context));
#define __ctx __attribute__((btf_decl_tag("ctx")))
struct bar { int a; int b; } __pai;
struct foo {
  struct bar b;
} __ctx __pai;
... struct foo f; ...
... f.b.bb ...

The call to preserve_struct_index generated for bb in f.b.bb would be replaced by getelementptr and later by getelementptr.and.load.
However, context structures don't have nested structures at the moment. The list of context structures is:

• __sk_buff
• bpf_cgroup_dev_ctx
• bpf_sk_lookup
• bpf_sock
• bpf_sock_addr
• bpf_sock_ops
• bpf_sockopt
• bpf_sysctl
• sk_msg_md
• sk_reuseport_md
• xdp_md

Right. Such recursive propagation of PAI is necessary. For btf_tag we cannot do it. Always propagating it won't be correct.
New preserve_const_field_offset would need to be propagated too and we actually have nested unions.
__bpf_md_ptr is such example. btf_tag wouldn't propagate into that union, but attr(preserve_const_field_offset) should.

[eddyz87]

After retesting kernel build with LLVM=1 and libbpf patch to reconstruct btf_decl_tag [1], statistics for BPF selftests looks as follows:

• out of 653 object files 13 have some differences with and w/o this change;
• for 2 programs there is small instruction count increase (+2 insn total);
• for 5 programs there is small instruction decrease (-6 insn total);
• 6 programs differ slightly but number of instructions is the same.

(The differences are insignificant, the rest of the comment could be skipped as it is probably not interesting for anyone but me).

---

Differences for first 5 programs were already described in this comment. The rest of the differences in described below.

netns_cookie_prog.bpf.o

Without ctx: 46 instructions
With ctx: 46 instructions

Instruction reordering:

 <get_netns_cookie_sk_msg>:
 	r6 = r1
-	r2 = *(u64 *)(r6 + 0x48)
 	r1 = *(u32 *)(r6 + 0x10)
-	if w1 != 0xa goto +0xb <LBB1_4>
+	if w1 != 0xa goto +0xc <LBB1_4>
+	r2 = *(u64 *)(r6 + 0x48)
 	if r2 == 0x0 goto +0xa <LBB1_4>
 	r1 = 0x0 ll
 	r3 = 0x0

The difference is introduced by "Machine code sinking" transformation. Before the transformation both 0x48 and 0x10 loads reside in the same basic block:

;; Old:
bb.0.entry:
  ...
  %0:gpr = CORE_LD64 345, %2:gpr, @"llvm.sk_msg_md:0:72$0:10:0"
  %9:gpr32 = CORE_LD32 350, %2:gpr, @"llvm.sk_msg_md:0:16$0:2"
  JNE_ri_32 killed %9:gpr32, 10, %bb.3

;; New:
bb.0.entry:
  ...
  %0:gpr = LDD %2:gpr, 72
  %3:gpr32 = LDW32 %2:gpr, 16
  JNE_ri_32 killed %3:gpr32, 10, %bb.3

Note: CORE pseudo-instructions are replaced by regular loads because btf_decl_tag("ctx") has priority over preserve_access_index attribute. The "Machine code sinking" transformation (MachineSink.cpp) can move LDD, LDW instructions, but can't move CORE_LD* because CORE_LD* instructions are marked as MCID::UnmodeledSideEffects in BPFGenInstrInfo.inc (maybe something to adjust):

// called from MachineSinking::SinkInstruction
bool MachineInstr::isSafeToMove(AAResults *AA, bool &SawStore) const {
  if (... hasUnmodeledSideEffects())
    return false;
  ...
}

sock_destroy_prog.bpf.o

Without ctx: 102 instructions
With ctx: 101 instructions

In the following code fragment:

	if (ctx->protocol == IPPROTO_TCP)
		bpf_map_update_elem(&tcp_conn_sockets, &key, &sock_cookie, 0);
	else if (ctx->protocol == IPPROTO_UDP)
		bpf_map_update_elem(&udp_conn_sockets, &keyc, &sock_cookie, 0);
	else
		return 1;

Version w/o btf_decl_tag("ctx") keeps two loads for ctx->protocol because of the llvm.bpf.passthrough call. Version with btf_decl_tag("ctx") eliminates second load via a combination of EarlyCSEPass/InstCombinePass/SimplifyCFGPass passes.

socket_cookie_prog.bpf.o

Without ctx: 66 instructions
With ctx: 66 instructions

For the following C code fragment:

SEC("sockops")
int update_cookie_sockops(struct bpf_sock_ops *ctx)
{
	struct bpf_sock *sk = ctx->sk;
	struct socket_cookie *p;

	if (ctx->family != AF_INET6)
		return 1;

	if (ctx->op != BPF_SOCK_OPS_TCP_CONNECT_CB)
		return 1;

	if (!sk)
		return 1;
    ...
}

Code with decl_tag("ctx") does reordering for ctx->sk load relative to ctx->family and ctx->op loads:

//  old                                       new
 <update_cookie_sockops>:                  <update_cookie_sockops>:
    r6 = r1                                   r6 = r1
-   r2 = *(u64 *)(r6 + 0xb8)
    r1 = *(u32 *)(r6 + 0x14)                  r1 = *(u32 *)(r6 + 0x14)
    if w1 != 0xa goto +0x13 <LBB1_6>          if w1 != 0xa goto +0x14 <LBB1_6>
    r1 = *(u32 *)(r6 + 0x0)                   r1 = *(u32 *)(r6 + 0x0)
    if w1 != 0x3 goto +0x11 <LBB1_6>          if w1 != 0x3 goto +0x12 <LBB1_6>
                                          +   r2 = *(u64 *)(r6 + 0xb8)
    if r2 == 0x0 goto +0x10 <LBB1_6>          if r2 == 0x0 goto +0x10 <LBB1_6>
    r1 = 0x0 ll                               r1 = 0x0 ll
    r3 = 0x0                                  r3 = 0x0

Code w/o decl_tag("ctx") uses CORE_LD* instructions for these loads and does not reorder loads due to reasons as in netns_cookie_prog.bpf.o.

test_lwt_reroute.bpf.o

Without ctx: 18 instructions
With ctx: 17 instructions

The difference boils down EarlyCSEPass being able to remove last load in the store/load pair:

llvm
; Before EarlyCSEPass
  store i32 %and, ptr %mark24, align 8
  %mark25 = getelementptr inbounds %struct.__sk_buff, ptr %skb, i32 0, i32 2
  %19 = load i32, ptr %mark25, align 8
  %cmp26 = icmp eq i32 %19, 0
; After EarlyCSEPass
  %and = and i32 %cond, 255
  store i32 %and, ptr %mark, align 8
  %cmp26 = icmp eq i32 %and, 0

And unable to do so when get.element.and.{store,load} intrinsics are used. Which leads to slight codegen differences downstream.

test_sockmap_invalid_update.bpf.o

Without ctx: 13 instructions
With ctx: 12 instructions

In the following C fragment:

c
	if (skops->sk)
		bpf_map_update_elem(&map, &key, skops->sk, 0);

Code with decl_tag("ctx") loads skops->sk only once. Code w/o decl_tag("ctx") uses CO-RE relocations and does load twice. As with sock_destroy_prog.bpf.o, EarlyCSEPass does not consolidate identical %x = call llvm.bpf.passthrough; load %x pairs.

type_cast.bpf.o

Without ctx: 96 instructions
With ctx: 96 instructions

__builtin_memcpy(name, dev->name, IFNAMSIZ) is unrolled in a different order. No idea why.

core_kern.bpf.o, test_verif_scale2.bpf.o

For both programs number of instructions is unchanged (11249, 12286). Some instructions have different order after DAG->DAG Pattern Instruction Selection. Instruction selection with CO-RE and non-CO-RE loads produces slightly different result.

[eddyz87]

In D133361#4629292, @ast wrote:

...
Right. Such recursive propagation of PAI is necessary. For btf_tag we cannot do it. Always propagating it won't be correct.
New preserve_const_field_offset would need to be propagated too and we actually have nested unions.
__bpf_md_ptr is such example. btf_tag wouldn't propagate into that union, but attr(preserve_const_field_offset) should.

Hi Alexei,

It just occurred to me that such an attribute would also require DWARF and BTF encoding in order to get reflected in vmlinux.h (which we already have for btf_decl_tag). Given this I think we can rename decl tag "ctx" to btf_decl_tag("preserve_const_field_offset") but we should still keep it a btf_decl_tag. I'll try to replace usage of bpf_context_marker intrinsic by metadata, if that fails will just rename the intrinsic to preserve_const_field_offset.
What do you think? (Sorry, I should have thought this through last week).

[ast]

In D133361#4637008, @eddyz87 wrote:

In D133361#4629292, @ast wrote:

...
Right. Such recursive propagation of PAI is necessary. For btf_tag we cannot do it. Always propagating it won't be correct.
New preserve_const_field_offset would need to be propagated too and we actually have nested unions.
__bpf_md_ptr is such example. btf_tag wouldn't propagate into that union, but attr(preserve_const_field_offset) should.

Hi Alexei,

It just occurred to me that such an attribute would also require DWARF and BTF encoding in order to get reflected in vmlinux.h (which we already have for btf_decl_tag). Given this I think we can rename decl tag "ctx" to btf_decl_tag("preserve_const_field_offset") but we should still keep it a btf_decl_tag. I'll try to replace usage of bpf_context_marker intrinsic by metadata, if that fails will just rename the intrinsic to preserve_const_field_offset.
What do you think? (Sorry, I should have thought this through last week).

I still feel that new attr is much cleaner from llvm implementation/design perspective and vmlinux.h inconvenience should be low priority in this considerations.
Since ctx only applies to uapi/bpf.h header the users don't have to use vmlinux.h. I know that today we have pains combining uapi headers and vmlinux.h and several solutions were
proposed. None were accepted yet, but that shouldn't mean that we should sacrifice llvm implementation due to orthogonal issues.
As a temporary workaround for vmlinux.h we can have uapi/bpf.h to do attr(preserve_const_field_offset) _and_ btf_decl_tag("bpf_ctx_struct"). Then teach pahole to emit attr(preserve_const_field_offset)
when it sees btf_decl_tag("bpf_ctx_struct") in vmlinux BTF.
Other workarounds are possible.

[eddyz87]

• Use (new) attribute((preserve_static_offset)) instead of attribute((btf_decl_tag("ctx"))).
• Rename files, passes, etc accordingly.

[eddyz87]

Fix for failed unit test pragma-attribute-supported-attributes-list.test

[aaron.ballman]

I don't have a whole lot of opinions on the attribute itself (I'm not super familiar with BPF), but did spot some things to do on the Clang side. This will also need reviewers for the LLVM changes -- any ideas on who usually reviews BPF-related changes in LLVM?

clang/lib/Sema/SemaDeclAttr.cpp

7274–7275

8504

clang/test/Sema/bpf-attr-preserve-static-offset.c

1 ↗

(On Diff #556858)

You should also add a -verify test that verifies we diagnose applying the attribute to something other than a structure or union, accepts no arguments, is diagnosed on a non-BPF target, etc to ensure we've got correct diagnostic behavior.

[erichkeane]

The CFE stuff is pretty innocuous and I don't see a reason to stop this on our accord, but would like someone familiar with the LLVM stuff to review this at one point.

clang/lib/CodeGen/CGExpr.cpp

3721

3725

You can use E->getType()->getPointeeType() here instead, unless you REALLY care that this is only a normal-pointer deref (and not a PMF or reference type).

3816

I'd suggest just making hasBPFPreserveStaticOffset be nullptr tolerant and remove the 1st half of this.

clang/lib/Sema/SemaDeclAttr.cpp

7278

This should use BPFPreserveStaticOffsetAttr::Create instead of using placement new. We've been meaning to translate these all at one point, but never got around to it.

8504

Or this, this is probably better.

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

Are we sure we want to do something like this? It seems this both depends on YOUR computer AND us never releasing Clang 18.

[eddyz87]

Rebase, changes as requested by @aaron.ballman and @erichkeane.

Hi @aaron.ballman, @erichkeane,

Thank you for taking a look.
I beleive this commit covers all feedback except "clang version"
metadata comment by @erichkeane, I added inline reply there.

This will also need reviewers for the LLVM changes -- any ideas on
who usually reviews BPF-related changes in LLVM?

I'll communicate with @ast and @yonghong-song.

[erichkeane]

In D133361#4652102, @eddyz87 wrote:

Rebase, changes as requested by @aaron.ballman and @erichkeane.

Hi @aaron.ballman, @erichkeane,

Thank you for taking a look.
I beleive this commit covers all feedback except "clang version"
metadata comment by @erichkeane, I added inline reply there.

This will also need reviewers for the LLVM changes -- any ideas on
who usually reviews BPF-related changes in LLVM?

I'll communicate with @ast and @yonghong-song.

I don't see the comment response you had to me.

clang/lib/CodeGen/CGExpr.cpp

3727

getPointeeType can also return nullptr, so unless you have a test elsewhere to ensure it isn't, you likely have to do a little more work here (and if so, I'd need an assert).

[eddyz87]

In D133361#4652368, @erichkeane wrote:

I don't see the comment response you had to me.

Sorry, forgot to click submit.

clang/lib/Sema/SemaDeclAttr.cpp

7278

Replaced by a call to handleSimpleAttribute as suggested by Aaron.

clang/test/Sema/bpf-attr-preserve-static-offset.c

1 ↗

(On Diff #556858)

-verify looks neat, thank you.

I've added two test cases:

• clang/test/Sema/bpf-attr-preserve-static-offset-warns.c
• clang/test/Sema/bpf-attr-preserve-static-offset-warns-nonbpf.c

Those check for things you listed:

• can't be used for non-bpf target
• can't take parameters
• is error to put it on something other than struct / union.

Tbh, I don't know if there is anything else to test in this regard.

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

Are you sure this would be an issue?
The specific line is not a part of a CHECK and I tried the following command using my system's llvm 16 opt:

opt -O2 -mtriple=bpf-pc-linux -S -o - load-align.ll

And module was loaded / processed w/o any issues.
In general grepping shows that people don't usually mask these in tests:

$ cd llvm/test/CodeGen/
$ ag '{!"clang version' | wc -l
452

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3727

Is it? I actually double-checked this before pushing an update, clangd jumps to the following definition:

QualType Type::getPointeeType() const {
  if (const auto *PT = getAs<PointerType>())
    return PT->getPointeeType();
  ...
  return {};
}

The getAsRecordDecl() can return null indeed, but that null is checked.

[erichkeane]

clang/lib/CodeGen/CGExpr.cpp

3727

More correctly, it returns an empty QualType. The operator-> on that will cause a nullptr, causing the call to Type::getAsRecordDecl to have a nullptr this.

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

I don't write LLVM tests ever, so I'm not sure. It just seems odd to provide that much irrelevant info, perhaps one of hte LLVM reviewers can comment. Also, look at those ~450 and see what they contain?

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3727

Oh, right, I'll add an additional check, thank you.

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

Also, look at those ~450 and see what they contain?

Same random clang versions:

$ ag '{!"clang version' | head
X86/debug-loclists.ll:129:!6 = !{!"clang version 10.0.0 (trunk 374581) (llvm/trunk 374579)"}
X86/dbg-combine.ll:84:!11 = !{!"clang version 3.7.0 (trunk 227074)"}
X86/debuginfo-locations-dce.ll:46:!6 = !{!"clang version 8.0.0 (trunk 339665)"}
X86/pr31242.ll:48:!5 = !{!"clang version 4.0.0 (trunk 288844)"}
X86/catchpad-regmask.ll:140:!1 = !{!"clang version 3.8.0 "}
X86/debug-nodebug-crash.ll:48:!5 = !{!"clang version 4.0.0"}
X86/limit-split-cost.mir:64:  !2 = !{!"clang version 7.0.0 (trunk 335057)"}
X86/swap.ll:169:!1 = !{!"clang version 9.0.0 (trunk 352631) (llvm/trunk 352632)"}
X86/dwarf-aranges-available-externally.ll:65:!16 = !{!"clang version 15.0.0 (https://github.com/llvm/llvm-project.git 2f52a868225755ebfa5242992d3a650ac6aadce7)"}
X86/label-annotation.ll:96:!7 = !{!"clang version 9.0.0 ([e-mail]:llvm/llvm-project.git 7f9a008a2db285aca57bfa0c09858c9527a7aa98)"}

[erichkeane]

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

It seems at least removing your home-path would be a good idea, but I can't really review these either.

Note the 'trunk #' is from our SVN days, so that isn't particularly useful at all. IMO everything in the parens is worthless in the tests, but hopefully someone familiar with the LLVM tests can stop by and correct me.

[eddyz87]

llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll

61 ↗

(On Diff #556858)

Most of the tests use generated IR as a starting point and it looks like people don't bother to hide these details. But I checked and test passes if I replace this string with just "clang", I'll update the tests, not a big deal.

[eddyz87]

Additional changes to address feedback from @erichkeane:

• Update for hasBPFPreserveStaticOffset: added null check for getPointeeType() result (note: currently this function is only called for array base, so getPointeeType() always returns a non-null but I think that having a null check instead of assert is still better in this context).
• Update for all .ll test cases to remove references to clang version or compilation directory from all metadata.

[erichkeane]

2 nits on the CFE, else LGTM! Obviously you still need an LLVM reviewer for the backend changes.

clang/lib/CodeGen/CGExpr.cpp

3728

We override operator bool to make this work.

3730

if possible?

[eddyz87]

Rebase, added "const" qualifier in hasBPFPreserveStaticOffset().

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3728

Sorry, just to clarify, currently such modification fails with the following error:

clang/lib/CodeGen/CGExpr.cpp:3710:7: error: invalid argument type 'QualType' to unary expression
  if (!PointeeType)
      ^~~~~~~~~~~~
1 error generated.

And you want me to modify QualType as follows:

--- a/clang/include/clang/AST/Type.h
+++ b/clang/include/clang/AST/Type.h
@@ -796,6 +796,8 @@ public:
     return getTypePtr();
   }
 
+  explicit operator bool() const { return isNull(); }
+
   bool isCanonical() const;
   bool isCanonicalAsParam() const;

Right?

[erichkeane]

clang/lib/CodeGen/CGExpr.cpp

3728

No, don't do that, you can leave it just checking isNull. I could have sworn we already had that operator, but perhaps it was removed at one point.

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3728

Understood.
Thank you for the review.

[ast]

clang/lib/CodeGen/CGExpr.cpp

3817

If I'm reading this correctly wrapping with preserve_static_offset doesn't prevent further preserver_access_index wrapping which is a wasted effort for pai at the end ?

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3817

Yes, pai calls are undone in BPFPreserveStaticOffset.cpp:removePAICalls(). I can put back the logic that suppresses pai if preserve static offset is present.

[ast]

clang/lib/CodeGen/CGExpr.cpp

3817

I see. I guess I missed a previous discussion. Why this approach was chosen?

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3817

Initial version used __attribute__((btf_decl_tag("ctx"))) and Yonghong did not want to have prioritization between btf_decl_tag and preserve_access_index basing on decl tag string parameter. Now this limitation is gone (and I think this was one of your arguments in favor of separate attribute).

[ast]

clang/lib/CodeGen/CGExpr.cpp

3817

Ahh. Right. It made sense to avoid special treatment of strings in decl_tag, but now it's gone and PSO takes precedence over PAI. Here we're adding PAI just to remove it later. Looks like a waste of cpu cycles and code. Unless applying PAI to outer struct and PSO in inner makes implementation tricky. I doubt we need to support such combo though. I'm fine cleaning this up in a follow up. If such cleanup makes sense at all.

[eddyz87]

clang/lib/CodeGen/CGExpr.cpp

3817

If such cleanup makes sense at all.

I think it does, I'll implement this change and push an update.

[yonghong-song]

I went through LLVM part of the change, mostly look good to me from high-level and I need to go through another pass with details.
It would be great if you can post corresponding kernel patches which utilizes this functionality?

clang/include/clang/Basic/AttrDocs.td

2227 ↗

(On Diff #557580)

g->b.a ?

llvm/lib/Target/BPF/BPFPreserveStaticOffset.cpp

82 ↗

(On Diff #557580)

Is it possible that such a pattern already changed by GVN/CSE so later on converting to preserve_static_offset becomes impossible? I guess it is unlikely but want to double check. Maybe add a comment to clarify.

102 ↗

(On Diff #557580)

So the above implies that this is bad for CSE/GVN due to different representation, right? But we want to do (1) since we want to avoid later CSE/GVN etc, so argument of clobbering is not that important, right?

[yonghong-song]

I tried a couple of examples with both preserve_access_index and preserve_static_offset (both 'preserve_access_index preserve_static_offset' and 'preserve_static_offset preserve_access_index'). Looks they cooperate with each other properly. preserve_static_offset is for base pointer while
preserve_access_index is for offset. So the current implementation seems OKAY. But preserve_static_offset actually means 'static offset' (no offset change),
so preserve_access_index seems not necessary, so if possible maybe clang frontend should make a choice to remove preserve_access_index if
preserve_static_offset is there. But this can be done later (need frontend review again, need to update doc etc.)

[eddyz87]

Rebase.

[eddyz87]

Fix for documentation type g->a.b -> g->b.a.

[eddyz87]

llvm/lib/Target/BPF/BPFPreserveStaticOffset.cpp

82 ↗

(On Diff #557580)

Actually yes, that is possible. It can probably be mitigated by allowing non-constant indices in the get.and.{load,store} functions. As we already agreed to land this, I'll do this as a separate MR.

102 ↗

(On Diff #557580)

We want to forbid some changes by CSE/GVN but allow others. E.g. suppose the following example:

struct foo {
  int a;
  int b;
} __attribute__((preserve_static_offset));

void consume(int);

static void bar(int *p) { consume(*p); }
void foo(struct foo *foo) {
  consume(foo->b);
  bar(&foo->b);
}

After inlining of bar to foo it would look like:

consume(foo->b);
consume(foo->b);

And I attempted not to break CSE/GVN for such situations.

[eddyz87]

Rebase, hope for green CI.

[eddyz87]

Reopening revision to fix errors reported by testbot.

[eddyz87]

Fixes for issues reported by testbot:

• bpf-preserve-static-offset-bitfield.c updated to use -target bpfel instead of -target bpf to avoid different code generation between big and little endian machines.
• BPFPreserveStaticOffset.cpp:removePAICalls() updated to avoid use after free for WorkList elements: series of if (isXXX(V)) removeXXX(V) modified to be else-if statement.

[eddyz87]

Rebase.

clang/lib/CodeGen/CGExpr.cpp

static QualType getFixedSizeElementType(const ASTContext &ctx,
                                         const VariableArrayType *vla) {
   QualType eltType;
   do {
     eltType = vla->getElementType();
   } while ((vla = ctx.getAsVariableArrayType(eltType)));
   return eltType;
 }
 
+// Returns true if D is marked with __attribute__((btf_decl_tag("ctx")))
+static bool hasCtxBTFDeclTagAttr(const Decl *D) {
+  if (auto *Attr = D->getAttr<BTFDeclTagAttr>())
+    return Attr->getBTFDeclTag().equals("ctx");
+
+  return false;
+}
+
+static bool pointeeHasCtxBTFDeclTagAttr(const Expr *E) {
+  if (auto *PtrType = dyn_cast<PointerType>(E->getType()))
+    if (auto *BaseDecl = PtrType->getPointeeType()->getAsRecordDecl())
+      return hasCtxBTFDeclTagAttr(BaseDecl);
+
+  return false;
+}
+
+// Returns true if declaration is marked
+//         with __attribute__((preserve_access_index))
+// but NOT with __attribute__((btf_decl_tag("ctx")))
+static bool hasBPFPreserveAccessIndexAttr(const Decl *D) {
+  return D->hasAttr<BPFPreserveAccessIndexAttr>() && !hasCtxBTFDeclTagAttr(D);
+}
+
+// Wraps Addr with a call to llvm.bpf.context.marker intrinsic.
+// This is used for target specific rewrites, for details see
+// llvm/lib/Target/BPF/BPFContextMarker.cpp
+static Address wrapWithBPFContextMarker(CodeGenFunction &CGF,
+                                        Address &Addr) {
+  llvm::Function *Fn =
+    CGF.CGM.getIntrinsic(llvm::Intrinsic::bpf_context_marker,
+                         {Addr.getType(), Addr.getType()});
+  llvm::CallInst *Call = CGF.Builder.CreateCall(Fn, {Addr.getPointer()});
+  return Address(Call, Addr.getElementType(), Addr.getAlignment());
+}
+
 /// Given an array base, check whether its member access belongs to a record
 /// with preserve_access_index attribute or not.
 static bool IsPreserveAIArrayBase(CodeGenFunction &CGF, const Expr *ArrayBase) {
   if (!ArrayBase || !CGF.getDebugInfo())
     return false;
 
   // Only support base as either a MemberExpr or DeclRefExpr.
   // DeclRefExpr to cover cases like:
   //    struct s { int a; int b[10]; };
   //    struct s *p;
   //    p[1].a
   // p[1] will generate a DeclRefExpr and p[1].a is a MemberExpr.
   // p->b[5] is a MemberExpr example.
   const Expr *E = ArrayBase->IgnoreImpCasts();
   if (const auto *ME = dyn_cast<MemberExpr>(E))
-    return ME->getMemberDecl()->hasAttr<BPFPreserveAccessIndexAttr>();
+    return hasBPFPreserveAccessIndexAttr(ME->getMemberDecl());
 
   if (const auto *DRE = dyn_cast<DeclRefExpr>(E)) {
     const auto *VarDef = dyn_cast<VarDecl>(DRE->getDecl());
     if (!VarDef)
       return false;
 
     const auto *PtrT = VarDef->getType()->getAs<PointerType>();
     if (!PtrT)
       return false;
 
     const auto *PointeeT = PtrT->getPointeeType()
                              ->getUnqualifiedDesugaredType();
     if (const auto *RecT = dyn_cast<RecordType>(PointeeT))
-      return RecT->getDecl()->hasAttr<BPFPreserveAccessIndexAttr>();
+      return hasBPFPreserveAccessIndexAttr(RecT->getDecl());
     return false;
   }
 
   return false;
 }
 
 static Address emitArraySubscriptGEP(CodeGenFunction &CGF, Address addr,
                                      ArrayRef<llvm::Value *> indices,
if (auto vla = CGF.getContext().getAsVariableArrayType(eltType)) {
     eltType = getFixedSizeElementType(CGF.getContext(), vla);
   }
 
   // We can use that to compute the best alignment of the element.
   CharUnits eltSize = CGF.getContext().getTypeSizeInChars(eltType);
   CharUnits eltAlign =
     getArrayElementAlign(addr.getAlignment(), indices.back(), eltSize);
 
+  if (Base && pointeeHasCtxBTFDeclTagAttr(Base))
+    addr = wrapWithBPFContextMarker(CGF, addr);
+
   llvm::Value *eltPtr;
   auto LastIndex = dyn_cast<llvm::ConstantInt>(indices.back());
   if (!LastIndex ||
       (!CGF.IsInPreservedAIRegion && !IsPreserveAIArrayBase(CGF, Base))) {
     eltPtr = emitArraySubscriptGEP(
         CGF, addr.getElementType(), addr.getPointer(), indices, inbounds,
         signedIndices, loc, name);
   } else {
const bool UseVolatile = isAAPCS(CGM.getTarget()) &&
                              CGM.getCodeGenOpts().AAPCSBitfieldWidth &&
                              Info.VolatileStorageSize != 0 &&
                              field->getType()
                                  .withCVRQualifiers(base.getVRQualifiers())
                                  .isVolatileQualified();
     Address Addr = base.getAddress(*this);
     unsigned Idx = RL.getLLVMFieldNo(field);
     const RecordDecl *rec = field->getParent();
+    if (hasCtxBTFDeclTagAttr(rec))
+      Addr = wrapWithBPFContextMarker(*this, Addr);
     if (!UseVolatile) {
       if (!IsInPreservedAIRegion &&
-          (!getDebugInfo() || !rec->hasAttr<BPFPreserveAccessIndexAttr>())) {
+          (!getDebugInfo() || !hasBPFPreserveAccessIndexAttr(rec))) {
         if (Idx != 0)
           // For structs, we GEP to the field that the record layout suggests.
           Addr = Builder.CreateStructGEP(Addr, Idx, field->getName());
       } else {
         llvm::DIType *DbgInfo = getDebugInfo()->getOrCreateRecordType(
             getContext().getRecordType(rec), rec->getLocation());
         Addr = Builder.CreatePreserveStructAccessIndex(
             Addr, Idx, getDebugInfoFIndex(rec, field->getFieldIndex()),
if (base.getTBAAInfo().isMayAlias() ||
 
     // Update the final access type and size.
     FieldTBAAInfo.AccessType = CGM.getTBAATypeInfo(FieldType);
     FieldTBAAInfo.Size =
         getContext().getTypeSizeInChars(FieldType).getQuantity();
   }
 
   Address addr = base.getAddress(*this);
+  if (hasCtxBTFDeclTagAttr(rec))
+    addr = wrapWithBPFContextMarker(*this, addr);
   if (auto *ClassDef = dyn_cast<CXXRecordDecl>(rec)) {
     if (CGM.getCodeGenOpts().StrictVTablePointers &&
         ClassDef->isDynamicClass()) {
       // Getting to any field of dynamic object requires stripping dynamic
       // information provided by invariant.group.  This is because accessing
       // fields may leak the real address of dynamic object, which could result
       // in miscompilation when leaked pointer would be compared.
       auto *stripped = Builder.CreateStripInvariantGroup(addr.getPointer());
       addr = Address(stripped, addr.getElementType(), addr.getAlignment());
     }
   }
 
   unsigned RecordCVR = base.getVRQualifiers();
   if (rec->isUnion()) {
     // For unions, there is no pointer adjustment.
     if (CGM.getCodeGenOpts().StrictVTablePointers &&
         hasAnyVptr(FieldType, getContext()))
       // Because unions can easily skip invariant.barriers, we need to add
       // a barrier every time CXXRecord field with vptr is referenced.
       addr = Builder.CreateLaunderInvariantGroup(addr);
 
     if (IsInPreservedAIRegion ||
-        (getDebugInfo() && rec->hasAttr<BPFPreserveAccessIndexAttr>())) {
+        (getDebugInfo() && hasBPFPreserveAccessIndexAttr(rec))) {
       // Remember the original union field index
       llvm::DIType *DbgInfo = getDebugInfo()->getOrCreateStandaloneType(base.getType(),
           rec->getLocation());
       addr = Address(
           Builder.CreatePreserveUnionAccessIndex(
               addr.getPointer(), getDebugInfoFIndex(rec, field->getFieldIndex()), DbgInfo),
           addr.getElementType(), addr.getAlignment());
     }
 
     if (FieldType->isReferenceType())
       addr = Builder.CreateElementBitCast(
           addr, CGM.getTypes().ConvertTypeForMem(FieldType), field->getName());
   } else {
     if (!IsInPreservedAIRegion &&
-        (!getDebugInfo() || !rec->hasAttr<BPFPreserveAccessIndexAttr>()))
+        (!getDebugInfo() || !hasBPFPreserveAccessIndexAttr(rec)))
       // For structs, we GEP to the field that the record layout suggests.
       addr = emitAddrOfFieldStorage(*this, addr, field);
     else
       // Remember the original struct field index
       addr = emitPreserveStructAccess(*this, base, addr, field);
   }
 
   // If this is a reference field, load the reference right now.

clang/lib/Sema/SemaDeclAttr.cpp

 }
 
 static void handleBPFPreserveAccessIndexAttr(Sema &S, Decl *D,
     const ParsedAttr &AL) {
   auto *Rec = cast<RecordDecl>(D);
   handleBPFPreserveAIRecord(S, Rec);
   Rec->addAttr(::new (S.Context) BPFPreserveAccessIndexAttr(S.Context, AL));
 }
 
 static bool hasBTFDeclTagAttr(Decl *D, StringRef Tag) {
   for (const auto *I : D->specific_attrs<BTFDeclTagAttr>()) {
     if (I->getBTFDeclTag() == Tag)
       return true;
   }
   return false;
 }
 
+static bool hasCtxBTFDeclTagAttr(Decl *D) {
+  return hasBTFDeclTagAttr(D, "ctx");
+}
+
+static void handleCtxBTFDeclTagAttr(Sema &S, RecordDecl *RD) {
+  // Add btf_decl_tag("ctx") attribute to all fields and inner records.
+  for (auto *D : RD->decls()) {
+    if (hasCtxBTFDeclTagAttr(D) || D->getDeclContext() != RD)
+      continue;
+
+    D->addAttr(BTFDeclTagAttr::CreateImplicit(S.Context, "ctx"));
+    if (auto *NestedRD = dyn_cast<RecordDecl>(D))
+      handleCtxBTFDeclTagAttr(S, NestedRD);
+  }
+}
+
 static void handleBTFDeclTagAttr(Sema &S, Decl *D, const ParsedAttr &AL) {
   StringRef Str;
   if (!S.checkStringLiteralArgumentAttr(AL, 0, Str))
     return;
   if (hasBTFDeclTagAttr(D, Str))
     return;
-
   D->addAttr(::new (S.Context) BTFDeclTagAttr(S.Context, AL, Str));
+  if (Str == "ctx" && isa<RecordDecl>(D))
+    handleCtxBTFDeclTagAttr(S, cast<RecordDecl>(D));
 }
 
 BTFDeclTagAttr *Sema::mergeBTFDeclTagAttr(Decl *D, const BTFDeclTagAttr &AL) {
   if (hasBTFDeclTagAttr(D, AL.getBTFDeclTag()))
     return nullptr;
   return ::new (Context) BTFDeclTagAttr(Context, AL, AL.getBTFDeclTag());
 }
 
case ParsedAttr::AT_AMDGPUNumVGPR:
     break;
   case ParsedAttr::AT_AVRSignal:
     handleAVRSignalAttr(S, D, AL);
     break;
   case ParsedAttr::AT_BPFPreserveAccessIndex:
     handleBPFPreserveAccessIndexAttr(S, D, AL);
     break;
   case ParsedAttr::AT_BTFDeclTag:
     handleBTFDeclTagAttr(S, D, AL);
     break;
   case ParsedAttr::AT_WebAssemblyExportName:
     handleWebAssemblyExportNameAttr(S, D, AL);
     break;
   case ParsedAttr::AT_WebAssemblyImportModule:
     handleWebAssemblyImportModuleAttr(S, D, AL);
     break;
   case ParsedAttr::AT_WebAssemblyImportName:

clang/test/CodeGen/bpf-decl-tag-ctx-pai.c

+// Check interaction between two attributes:
+//   __attribute__((btf_decl_tag("ctx")))
+//   __attribute__((preserve_access_index))
+//
+// First, check that calls to context.marker are inserted where expected:
+//
+// RUN: %clang -g -target bpf -Xclang -disable-llvm-passes -S -emit-llvm -o - %s | \
+// RUN:        FileCheck -check-prefix=FRONT %s
+//
+// Next, check that gep.and.load/store calls were generated:
+//
+// RUN: %clang -g -target bpf -O2 -S -emit-llvm -o - %s |  \
+// RUN:        FileCheck -check-prefix=MID %s
+//
+// For gep.and.load/store calls lowering refer to test the case
+// bpf-decl-tag-ctx.c
+
+#define __ctx__ __attribute__((btf_decl_tag("ctx")))
+#define __preserve_access_index__ __attribute__((preserve_access_index))
+
+extern void consume_int(int);
+extern void consume_ptr(void *);
+
+struct context1_inner2 {
+  int a;
+} __preserve_access_index__;
+
+struct context1_inner {
+  int a;
+  int b;
+  int c[5];
+  struct context1_inner2 *inner;
+  struct context1_inner2 inner_inline;
+} __ctx__ __preserve_access_index__;
+
+struct context1 {
+  int first;
+  struct context1_inner *inner;
+  struct context1_inner inner_inline;
+  int arr[4];
+  int *ptr;
+  int last;
+} __ctx__ __preserve_access_index__;
+
+// FRONT-LABEL: define {{.*}} @preserve_access_index1
+//   MID-LABEL: define {{.*}} @preserve_access_index1
+void preserve_access_index1(struct context1 *ctx) {
+  consume_int(ctx->first);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->arr[2]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 3
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r2]], i64 0, i64 2
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 3, i64 2)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_ptr(ctx->ptr);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r3]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 4)
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef [[r0]])
+
+  consume_int(ctx->last);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 5
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 5)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->inner->b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1_inner) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner->c[3]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r4]], i32 0, i32 2
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x i32], ptr [[r5]], i64 0, i64 3
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = load i32, ptr [[r6]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r7]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1_inner) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i64 3)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_inline.b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->inner_inline.c[3]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 2
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x i32], ptr [[r4]], i64 0, i64 3
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i32 2, i64 3)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx[17].arr[2]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i64 17
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r3]], i32 0, i32 3
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r4]], i64 0, i64 2
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i64 17, i32 3, i64 2)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->inner_inline.inner->a);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 3
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r4]]
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.preserve.struct.access.index.p0.p0(ptr elementtype(%struct.context1_inner2) [[r5]], i32 0, i32 0)
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = load i32, ptr [[r6]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r7]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2, i32 3)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i64, ptr @"llvm.context1_inner2:0:0$0:0"
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr i8, ptr [[r0]], i64 [[r1]]
+//   MID-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = tail call ptr @llvm.bpf.passthrough.p0.p0({{.*}}, ptr [[r2]])
+//   MID-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r4]])
+
+  consume_int(ctx->inner_inline.inner_inline.a);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 4
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.preserve.struct.access.index.p0.p0(ptr elementtype(%struct.context1_inner2) [[r4]], i32 0, i32 0)
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+// Note: because @"llvm.context1_inner2:0:0$0:0" represents a dynamic offset in bytes
+//       there is no way to reduce it to bpf.llvm.getelementptr.and.load as it's indices
+//       represent logical offsets (array element number, struct field index).
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 2, i32 4
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i64, ptr @"llvm.context1_inner2:0:0$0:0"
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr i8, ptr [[r0]], i64 [[r1]]
+//   MID-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = tail call ptr @llvm.bpf.passthrough.p0.p0({{.*}}, ptr [[r2]])
+//   MID-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r4]])
+
+  ctx->first = 1;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   store i32 1, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 1, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+
+  ctx->arr[2] = 2;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 3
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r2]], i64 0, i64 2
+// FRONT-NEXT:   store i32 2, ptr [[r3]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 3, i64 2)
+
+  ctx->ptr = (void*) 0;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   store ptr null, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (ptr, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.p0(ptr readnone null, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 4)
+
+  ctx->last = 3;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 5
+// FRONT-NEXT:   store i32 3, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 3, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 5)
+
+  ctx->inner->b = 4;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   store i32 4, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 4, ptr elementtype(%struct.context1_inner) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+
+  ctx->inner->c[3] = 5;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r4]], i32 0, i32 2
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x i32], ptr [[r5]], i64 0, i64 3
+// FRONT-NEXT:   store i32 5, ptr [[r6]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 5, ptr elementtype(%struct.context1_inner) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i64 3)
+
+  ctx->inner_inline.b = 6;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   store i32 6, ptr [[r4]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 6, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i32 1)
+
+  ctx->inner_inline.c[3] = 7;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 2
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x i32], ptr [[r4]], i64 0, i64 3
+// FRONT-NEXT:   store i32 7, ptr [[r5]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 7, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i32 2, i64 3)
+
+  ctx[19].arr[2] = 55;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i64 19
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r3]], i32 0, i32 3
+// Note: there is no marker call here, but this is fine: every array of a primitive
+//       type would have a base used as a member expression, thus a marker call always  would be
+//       somewhere above.
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r4]], i64 0, i64 2
+// FRONT-NEXT:   store i32 55, ptr [[r5]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 55, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i64 19, i32 3, i64 2)
+
+  ctx->inner_inline.inner->a = 34;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 3
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r4]]
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.preserve.struct.access.index.p0.p0(ptr elementtype(%struct.context1_inner2) [[r5]], i32 0, i32 0)
+// FRONT-NEXT:   store i32 34, ptr [[r6]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2, i32 3)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i64, ptr @"llvm.context1_inner2:0:0$0:0"
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr i8, ptr [[r0]], i64 [[r1]]
+//   MID-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = tail call ptr @llvm.bpf.passthrough.p0.p0({{.*}}, ptr [[r2]])
+//   MID-NEXT:   store i32 34, ptr [[r3]]
+
+  ctx->inner_inline.inner_inline.a = 76;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1_inner, ptr [[r3]], i32 0, i32 4
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.preserve.struct.access.index.p0.p0(ptr elementtype(%struct.context1_inner2) [[r4]], i32 0, i32 0)
+// FRONT-NEXT:   store i32 76, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load i64, ptr @"llvm.context1_inner2:0:0$0:0"
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr i8, ptr {{.*}}, i64 [[r0]]
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = tail call ptr @llvm.bpf.passthrough.p0.p0({{.*}}, ptr [[r1]])
+//   MID-NEXT:   store i32 76, ptr [[r2]]
+
+  consume_ptr(ctx);
+}
+
+struct context2 {
+  int _;
+  struct {
+    int _;
+    int b[10];
+  } anon_struct;
+  struct {
+    int _;
+    int b[10];
+  } anon_struct_arr[7];
+  struct { int _1; int injected_field; };
+  union { int _2; int injected_union_field; };
+} __ctx__ __preserve_access_index__;
+
+// FRONT-LABEL: define {{.*}} @nested_structs
+//   MID-LABEL: define {{.*}} @nested_structs
+void nested_structs(struct context2 *ctx) {
+  consume_int(ctx->anon_struct.b[8]);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.anon, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [10 x i32], ptr [[r4]], i64 0, i64 8
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1, i64 8)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->anon_struct_arr[5].b[7]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds [7 x %struct.anon.0], ptr [[r3]], i64 0, i64 5
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r4]])
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.anon.0, ptr [[r5]], i32 0, i32 1
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = getelementptr inbounds [10 x i32], ptr [[r6]], i64 0, i64 7
+// FRONT-NEXT:   [[r8:%[a-zA-Z0-9._]+]] = load i32, ptr [[r7]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r8]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i64 5, i32 1, i64 7)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->injected_field);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 3
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.anon.1, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 3, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+  consume_int(ctx->injected_union_field);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 4)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+}
+
+extern int magic2(int);
+
+struct bpf_sock {
+  int bound_dev_if;
+  int family;
+} __ctx__ __preserve_access_index__;
+
+struct bpf_sockopt {
+  struct bpf_sock *sk;
+  int level;
+  int optlen;
+} __ctx__ __preserve_access_index__;
+
+// FRONT-LABEL: define {{.*}} @known_load_sink_example_1
+//   MID-LABEL: define {{.*}} @known_load_sink_example_1
+int known_load_sink_example_1(struct bpf_sockopt *ctx)
+{
+  unsigned g = 0;
+  switch (ctx->level) {
+// MID:        [[level3:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+// MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+// MID-NEXT:   switch i32 [[level3]], label {{.*}} [
+// MID-NEXT:     i32 10, label
+// MID-NEXT:     i32 20, label
+// MID-NEXT:   ]
+
+  case 10:
+    g = magic2(ctx->sk->family);
+    break;
+// MID:      {{.*}}:
+// MID-NEXT:   [[sk4:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+// MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+// MID-NEXT:   [[family5:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+// MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sock) [[sk4]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+// MID:        br label
+
+  case 20:
+    g = magic2(ctx->optlen);
+    break;
+// MID:      {{.*}}:
+// MID-NEXT:   [[optlen6:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+// MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2)
+// MID:        br label
+  }
+// MID:      {{.*}}:
+// MID-NEXT:   [[optlen6_sink:%[a-zA-Z0-9._]+]] = phi i32
+// MID-NEXT:   [[call2:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[optlen6_sink]])
+// MID-NEXT:   [[phi_bo:%[a-zA-Z0-9._]+]] = and i32 [[call2]], 1
+// MID-NEXT:   br label
+
+  return g % 2;
+// MID:      {{.*}}:
+// MID-NEXT:   [[g_0:%[a-zA-Z0-9._]+]] = phi i32
+// MID:        ret i32 [[g_0]]
+}

clang/test/CodeGen/bpf-decl-tag-ctx-required.c

+// Check that passes handling __attribute__((btf_decl_tag("ctx"))) are
+// executed when -O0 is specified
+
+// RUN: %clang -target bpf -O0 -S -emit-llvm %s -o - |  \
+// RUN:        FileCheck %s
+
+#define __ctx__ __attribute__((btf_decl_tag("ctx")))
+
+extern void consume_int(int);
+
+struct foo {
+  int a;
+  int b;
+} __ctx__;
+
+struct bar {
+  int _;
+  struct foo foo;
+} __ctx__;
+
+void testfn(struct bar *ctx) {
+  consume_int(ctx->foo.b);
+// CHECK:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// CHECK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call i32 (ptr, i1, i8, i8, i8, i1, ...)
+// CHECK-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bar) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+// CHECK-NEXT:   call void @consume_int(i32 noundef [[r1]])
+
+  ctx->foo.b = 7;
+// CHECK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// CHECK-NEXT:   call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+// CHECK-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 7, ptr elementtype(%struct.bar) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+}

clang/test/CodeGen/bpf-decl-tag-ctx.c

+// This test covers various usages of __attribute__((btf_decl_tag("ctx"))).
+// This attribute is handled in three stages:
+// - context.marker intrinsic call is added while IR is generated from AST
+// - context.marker call is replaced with calls to either:
+//   llvm.bpf.getelementptr.and.load intrinsic, or
+//   llvm.bpf.getelementptr.and.store intrinsic
+//   at the early IR processing stage
+// - get and load/store intrinsics are lowered at the late IR processing stage
+//
+// The test verifies all three stages using three separate RUN commands.
+//
+// First, check that calls to context.marker intrinsic are inserted
+// where expected:
+//
+// RUN: %clang -target bpf -Xclang -disable-llvm-passes -S -emit-llvm -o - %s | \
+// RUN:        FileCheck -check-prefix=FRONT %s
+//
+// Next, check that gep_and_load/store calls were generated:
+//
+// RUN: %clang -target bpf -O2 -S -emit-llvm -o - %s | \
+// RUN:        FileCheck -check-prefix=MID %s
+//
+// Finally, verify that gep_and_load/store calls were rewritten:
+//
+// RUN: %clang -mllvm --print-after=bpf-check-and-opt-ir \
+// RUN:        -mllvm --stop-after=bpf-check-and-opt-ir \
+// RUN:        -target bpf -S -O2 -o /dev/null 2>&1 %s | \
+// RUN:        FileCheck -check-prefix=BACK %s
+
+#define __ctx__ __attribute__((btf_decl_tag("ctx")))
+#define __preserve_access_index__ __attribute__((preserve_access_index))
+
+struct inner_no_vof {
+  int a;
+  int b;
+};
+
+struct inner_vof {
+  int a;
+  int b;
+} __ctx__;
+
+struct context1 {
+   int first;
+   struct inner_no_vof inner_inline;
+   struct inner_no_vof *inner_ptr;
+   int butlast;
+   int last;
+} __ctx__;
+
+struct context2 {
+   int first;
+   struct inner_vof inner_inline;
+   struct inner_vof *inner_ptr;
+   int butlast;
+   int last;
+} __ctx__;
+
+extern void consume_int(int);
+extern void consume_float(float);
+extern void consume_ptr(void *);
+
+// FRONT-LABEL: define dso_local void @fields_access_1
+//   MID-LABEL: define dso_local void @fields_access_1
+//  BACK-LABEL: define dso_local void @fields_access_1
+void fields_access_1(struct context1 *ctx) {
+  consume_int(ctx->first);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_inline.a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r2]], i32 0, i32 0
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r4]])
+
+//   MID-NEXT:    [[a:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 0)
+//   MID-NEXT:    tail call void @consume_int(i32 noundef [[a]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 1, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_inline.b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r2]], i32 0, i32 1
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 1, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_ptr->a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 0
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[r1]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r2]])
+
+  consume_int(ctx->inner_ptr->b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 0, i32 1
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[r1]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r2]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 0, i32 1
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  ctx->first = 1;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   store i32 1, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 1, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   store i32 1, ptr [[r0]]
+
+  ctx->inner_inline.a = 2;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r2]], i32 0, i32 0
+// FRONT-NEXT:   store i32 2, ptr [[r3]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 0)
+
+//  BACK-NEXT:   [[r0:%[0-9]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 1, i32 0
+//  BACK-NEXT:   store i32 2, ptr [[r0]]
+
+  ctx->inner_inline.b = 3;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r2]], i32 0, i32 1
+// FRONT-NEXT:   store i32 3, ptr [[r3]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 3, ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+
+//  BACK-NEXT:   [[r0:%[0-9]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 1, i32 1
+//  BACK-NEXT:   store i32 3, ptr [[r0]]
+
+  ctx->inner_ptr->a = 4;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 0
+// FRONT-NEXT:   store i32 4, ptr [[r4]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   store i32 4, ptr [[r0]]
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   store i32 4, ptr [[r1]]
+
+  ctx->inner_ptr->b = 5;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 1
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 0, i32 1
+//   MID-NEXT:   store i32 5, ptr [[r1]]
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 0, i32 1
+//  BACK-NEXT:   store i32 5, ptr [[r2]]
+
+  consume_ptr(ctx);
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+// FRONT-LABEL: define dso_local void @fields_access_2
+//   MID-LABEL: define dso_local void @fields_access_2
+//  BACK-LABEL-LABEL: define dso_local void @fields_access_2
+void fields_access_2(struct context2 *ctx) {
+  consume_int(ctx->inner_inline.a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r3]], i32 0, i32 0
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 1, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_inline.b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 1, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->inner_ptr->a);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i32 0, i32 0
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 0
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  consume_int(ctx->inner_ptr->b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 1
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  ctx->inner_inline.a = 2;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r3]], i32 0, i32 0
+// FRONT-NEXT:   store i32 2, ptr [[r4]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 1, i32 0
+//  BACK-NEXT:   store i32 2, ptr [[r0]]
+
+  ctx->inner_inline.b = 3;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   store i32 3, ptr [[r4]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 3, ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 1, i32 1
+//  BACK-NEXT:   store i32 3, ptr [[r0]]
+
+  ctx->inner_ptr->a = 4;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i32 0, i32 0
+// FRONT-NEXT:   store i32 4, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 4, ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 0
+//  BACK-NEXT:   store i32 4, ptr [[r2]]
+
+  ctx->inner_ptr->b = 5;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   store i32 5, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 5, ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 1
+//  BACK-NEXT:   store i32 5, ptr [[r2]]
+
+  consume_ptr(ctx);
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+// FRONT-LABEL: define dso_local void @pointer_access_not_affected_1
+//   MID-LABEL: define dso_local void @pointer_access_not_affected_1
+//  BACK-LABEL: define dso_local void @pointer_access_not_affected_1
+void pointer_access_not_affected_1(struct context1 *ctx) {
+  consume_ptr(&ctx->inner_inline);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r2]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_inline.b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r2]], i32 0, i32 1
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r3]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 1, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 1, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_ptr);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r2]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 2
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 2
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_ptr->b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context1) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 0, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 0, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r2]])
+
+  consume_ptr(&ctx->last);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr [[r1]], i32 0, i32 4
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 4
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context1, ptr %ctx, i64 0, i32 4
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  FRONT:   ret void
+//    MID:   ret void
+//   BACK:   ret void
+}
+
+// FRONT-LABEL: define dso_local void @pointer_access_not_affected_2
+//   MID-LABEL: define dso_local void @pointer_access_not_affected_2
+//  BACK-LABEL: define dso_local void @pointer_access_not_affected_2
+void pointer_access_not_affected_2(struct context2 *ctx) {
+  consume_ptr(&ctx->inner_inline);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r2]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_inline.b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 1, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 1, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_ptr);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   call void @consume_ptr(ptr noundef [[r2]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 2
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i64 0, i32 2
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r0]])
+
+  consume_ptr(&ctx->inner_ptr->b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i32 0, i32 1
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context2) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 2)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r0]], i64 0, i32 1
+//   MID-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context2, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i64 0, i32 1
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef nonnull [[r2]])
+
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+struct context3 {
+  int first;
+  struct inner_no_vof *no_vof_ptr;
+  struct inner_no_vof  no_vof_arr[5];
+  struct inner_no_vof *no_vof_arr_ptr[7];
+  struct inner_vof *vof_ptr;
+  struct inner_vof  vof_arr[5];
+  struct inner_vof *vof_arr_ptr[7];
+  int arr[4];
+} __ctx__;
+
+// FRONT-LABEL: define dso_local void @array_access
+//   MID-LABEL: define dso_local void @array_access
+//  BACK-LABEL: define dso_local void @array_access
+void array_access(struct context3 *ctx) {
+  consume_int(ctx->no_vof_ptr[1].b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i64 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 1, i32 1
+//   MID-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[r1]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r2]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 1, i32 1
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  consume_int(ctx->no_vof_arr[2].b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x %struct.inner_no_vof], ptr [[r2]], i64 0, i64 2
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[r4]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r5]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i64 2, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr %ctx, i32 0, i32 2, i64 2, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->no_vof_arr_ptr[3]->a);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 3
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [7 x ptr], ptr [[r2]], i64 0, i64 3
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r3]]
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r4]], i32 0, i32 0
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 3, i64 3)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr %ctx, i32 0, i32 3, i64 3
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[r1]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r2]])
+
+  consume_int(ctx->vof_ptr[1].b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i64 1
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r5]])
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r6]], i32 0, i32 1
+// FRONT-NEXT:   [[r8:%[a-zA-Z0-9._]+]] = load i32, ptr [[r7]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r8]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 4)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i64 1, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 4
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i64 1, i32 1
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  consume_int(ctx->vof_arr[2].b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 5
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x %struct.inner_vof], ptr [[r3]], i64 0, i64 2
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r4]])
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r5]], i32 0, i32 1
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = load i32, ptr [[r6]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r7]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 5, i64 2, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 5, i64 2, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->vof_arr_ptr[3]->b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 6
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [7 x ptr], ptr [[r2]], i64 0, i64 3
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r3]]
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r4]])
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r5]], i32 0, i32 1
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = load i32, ptr [[r6]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r7]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 6, i64 3)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 6, i64 3
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 1
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r3]])
+
+  consume_int(ctx[12].arr[3]);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i64 12
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r3]], i32 0, i32 7
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r4]], i64 0, i64 3
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load i32, ptr [[r5]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r6]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i64 12, i32 7, i64 3)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr %ctx, i64 12, i32 7, i64 3
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  ctx->no_vof_ptr[1].b = 2;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i64 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   store i32 2, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 1, i32 1
+//   MID-NEXT:   store i32 2, ptr [[r1]]
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 1, i32 1
+//  BACK-NEXT:   store i32 2, ptr [[r2]]
+
+  ctx->no_vof_arr[2].b = 4;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x %struct.inner_no_vof], ptr [[r2]], i64 0, i64 2
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r3]], i32 0, i32 1
+// FRONT-NEXT:   store i32 4, ptr [[r4]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 4, ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2, i64 2, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 2, i64 2, i32 1
+//  BACK-NEXT:   store i32 4, ptr [[r0]]
+
+  ctx->no_vof_arr_ptr[3]->b = 6;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 3
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [7 x ptr], ptr [[r2]], i64 0, i64 3
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r3]]
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   store i32 6, ptr [[r5]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 3, i64 3)
+//   MID-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r0]], i64 0, i32 1
+//   MID-NEXT:   store i32 6, ptr [[r1]]
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 3, i64 3
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_no_vof, ptr [[r1]], i64 0, i32 1
+//  BACK-NEXT:   store i32 6, ptr [[r2]]
+
+  ctx->vof_ptr[1].b = 2;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r2]]
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r4]], i64 1
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r5]])
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r6]], i32 0, i32 1
+// FRONT-NEXT:   store i32 2, ptr [[r7]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 4)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i64 1, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 4
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r1]], i64 1, i32 1
+//  BACK-NEXT:   store i32 2, ptr [[r2]]
+
+  ctx->vof_arr[2].b = 4;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 5
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds [5 x %struct.inner_vof], ptr [[r3]], i64 0, i64 2
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r4]])
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r5]], i32 0, i32 1
+// FRONT-NEXT:   store i32 4, ptr [[r6]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 4, ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 5, i64 2, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 5, i64 2, i32 1
+//  BACK-NEXT:   store i32 4, ptr [[r0]]
+
+  ctx->vof_arr_ptr[3]->b = 6;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i32 0, i32 6
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = getelementptr inbounds [7 x ptr], ptr [[r2]], i64 0, i64 3
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r3]]
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r4]])
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.inner_vof, ptr [[r5]], i32 0, i32 1
+// FRONT-NEXT:   store i32 6, ptr [[r6]]
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 6, i64 3)
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 6, ptr elementtype(%struct.inner_vof) [[r0]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context3, ptr %ctx, i32 0, i32 6, i64 3
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[r0]]
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.inner_vof, ptr [[r1]], i32 0, i32 1
+//  BACK-NEXT:   store i32 6, ptr [[r2]]
+
+  ctx[15].arr[3] = 22;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r1]], i64 15
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr [[r3]], i32 0, i32 7
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = getelementptr inbounds [4 x i32], ptr [[r4]], i64 0, i64 3
+// FRONT-NEXT:   store i32 22, ptr [[r5]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 22, ptr elementtype(%struct.context3) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i64 15, i32 7, i64 3)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context3, ptr %ctx, i64 15, i32 7, i64 3
+//  BACK-NEXT:   store i32 22, ptr [[r0]]
+
+  consume_ptr(ctx);
+
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+typedef int aligned_int __attribute__((aligned(128)));
+
+struct context4 {
+  volatile int a;
+  aligned_int b;
+  volatile aligned_int c;
+} __ctx__;
+
+// FRONT-LABEL: define dso_local void @load_and_store_attrs
+//   MID-LABEL: define dso_local void @load_and_store_attrs
+//  BACK-LABEL: define dso_local void @load_and_store_attrs
+void load_and_store_attrs(struct context4 *ctx) {
+  int r;
+  consume_int(ctx->a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load volatile i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context4) %ctx, i1 true, i8 0, i8 1, i8 7, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load volatile i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context4) %ctx, i1 false, i8 0, i8 1, i8 7, i1 true, i32 0, i32 2)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context4, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_int(ctx->c);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load volatile i32, ptr [[r2]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r3]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context4) %ctx, i1 true, i8 0, i8 1, i8 7, i1 true, i32 0, i32 4)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 4
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load volatile i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  __atomic_load(&ctx->a, &r, 2);
+  consume_int(r);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load atomic volatile i32, ptr [[r2]] acquire
+// FRONT:   call void @consume_int
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context4) %ctx, i1 true, i8 4, i8 1, i8 7, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load atomic volatile i32, ptr [[r0]] acquire
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  ctx->a = 1;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   store volatile i32 1, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 1, ptr elementtype(%struct.context4) %ctx, i1 true, i8 0, i8 1, i8 7, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   store volatile i32 1, ptr [[r0]]
+
+  ctx->b = 2;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   store i32 2, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.context4) %ctx, i1 false, i8 0, i8 1, i8 7, i1 true, i32 0, i32 2)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context4, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   store i32 2, ptr [[r0]]
+
+  ctx->c = 3;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 4
+// FRONT-NEXT:   store volatile i32 3, ptr [[r2]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 3, ptr elementtype(%struct.context4) %ctx, i1 true, i8 0, i8 1, i8 7, i1 true, i32 0, i32 4)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 4
+//  BACK-NEXT:   store volatile i32 3, ptr [[r0]]
+
+  r = 7;
+  __atomic_store(&ctx->a, &r, 3);
+// FRONT-NEXT:   store i32 7, ptr %r
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr %r
+// FRONT-NEXT:   store atomic volatile i32 [[r3]], ptr [[r2]] release
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 7, ptr elementtype(%struct.context4) %ctx, i1 true, i8 5, i8 1, i8 7, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context4, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   store atomic volatile i32 7, ptr [[r0]] release
+//  BACK-NEXT:   tail call void @consume_ptr(ptr noundef %ctx)
+
+  consume_ptr(ctx);
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+struct context5 {
+  union {
+    int a;
+    float b;
+  } u;
+} __ctx__;
+
+// FRONT-LABEL: define dso_local void @union_access
+//   MID-LABEL: define dso_local void @union_access
+//  BACK-LABEL: define dso_local void @union_access
+void union_access(struct context5 *ctx1, struct context5 *ctx2) {
+  consume_int(ctx1->u.a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr %ctx1.addr
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context5, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load i32, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_int(i32 noundef [[r4]])
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.context5) %ctx1, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_int(i32 noundef [[r0]])
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context5, ptr %ctx1, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_int(i32 noundef [[r1]])
+
+  consume_float(ctx2->u.b);
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr %ctx2.addr
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context5, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = load float, ptr [[r3]]
+// FRONT-NEXT:   call void @consume_float(float noundef [[r4]])
+
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = tail call float (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.f32(ptr elementtype(%struct.context5) %ctx2, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   tail call void @consume_float(float noundef [[r0]])
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context5, ptr %ctx2, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load float, ptr [[r0]]
+//  BACK-NEXT:   tail call void @consume_float(float noundef [[r1]])
+
+  ctx1->u.a = 2;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr %ctx1.addr
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context5, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   store i32 2, ptr [[r3]]
+
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.i32(i32 2, ptr elementtype(%struct.context5) %ctx1, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context5, ptr %ctx1, i32 0, i32 0
+//  BACK-NEXT:   store i32 2, ptr [[r0]]
+
+  ctx2->u.b = 3.4f;
+// FRONT-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr %ctx2.addr
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context5, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r2]])
+// FRONT-NEXT:   store float {{.*}}, ptr [[r3]]
+
+//   MID-NEXT:   tail call void (float, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.store.f32(float {{.*}}, ptr elementtype(%struct.context5) %ctx2, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.context5, ptr %ctx2, i32 0, i32 0
+//  BACK-NEXT:   store float {{.*}}, ptr [[r0]]
+
+  consume_ptr(ctx1);
+  consume_ptr(ctx2);
+// FRONT:   ret void
+//   MID:   ret void
+//  BACK:   ret void
+}
+
+// TODO: add an offset at the start!
+struct context6 {
+  unsigned _;
+  unsigned a:1;
+  unsigned b:2;
+} __ctx__;
+
+// FRONT-LABEL: define dso_local void @bitfield_access
+//   MID-LABEL: define dso_local void @bitfield_access
+//  BACK-LABEL: define dso_local void @bitfield_access
+void bitfield_access(struct context6 *ctx) {
+  consume_int(ctx->a);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr {{.*}})
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr [[r0]], i32 0, i32 1
+// FRONT-NEXT:   load i8, ptr [[r1]]
+// FRONT:        @consume_int
+
+//   MID:        tail call i8 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i8(ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID:        @consume_int
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:        load i8, ptr [[r0]]
+//  BACK:        @consume_int
+
+  consume_int(ctx->b);
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr {{.*}})
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr [[r0]], i32 0, i32 1
+// FRONT-NEXT:   load i8, ptr [[r1]]
+// FRONT:        @consume_int
+
+//   MID:        [[r0:%[a-zA-Z0-9._]+]] = tail call i8 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i8(ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID:        @consume_int
+
+//  BACK:        [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:        load i8, ptr [[r0]]
+//  BACK:        tail call void @consume_int
+
+  ctx->a = 1;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr {{.*}})
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr [[r0]], i32 0, i32 1
+// FRONT-NEXT:   load i8, ptr [[r1]]
+// FRONT:        store {{.*}}, ptr [[r1]]
+
+//   MID-NEXT:  tail call i8 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i8(ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID:       tail call void (i8, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i8(i8 {{.*}}, ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+
+//  BACK:       [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:       load i8, ptr [[r0]]
+//  BACK:       [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:       store i8 {{.*}}, ptr [[r1]]
+
+  ctx->b = 2;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr {{.*}})
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr [[r0]], i32 0, i32 1
+// FRONT-NEXT:   load i8, ptr [[r1]]
+// FRONT:        store {{.*}}, ptr [[r1]]
+
+//   MID:       tail call i8 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i8(ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID:       tail call void (i8, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i8(i8 {{.*}}, ptr elementtype(%struct.context6) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+
+//  BACK:       [[r0:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:       load i8, ptr [[r0]]
+//  BACK:       [[r1:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.context6, ptr %ctx, i32 0, i32 1
+//  BACK:       store i8 {{.*}}, ptr [[r1]]
+
+  consume_ptr(ctx);
+// FRONT: ret void
+//   MID: ret void
+//  BACK: ret void
+}
+
+struct pattern_bug1 {
+  int src_port;
+  int dst_port;
+} __ctx__;
+
+// No need to check FRONT or BACK
+//   MID-LABEL: define dso_local i32 @pattern_bug1
+int pattern_bug1(struct pattern_bug1 *sk) {
+  int *port = &sk->dst_port;
+  return port[0] == 0x777;
+//   MID:        [[dst_port:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.pattern_bug1, ptr %sk, i64 0, i32 1
+//   MID-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load i32, ptr [[dst_port]]
+//   MID-NEXT:   [[cmp:%[a-zA-Z0-9._]+]] = icmp eq i32 [[r0]], 1911
+//   MID-NEXT:   [[conv:%[a-zA-Z0-9._]+]] = zext i1 [[cmp]] to i32
+//   MID-NEXT:   ret i32 [[conv]]
+}
+
+struct bpf_sock {
+  int bound_dev_if;
+  int family;
+} __ctx__;
+
+struct bpf_sockopt {
+  struct bpf_sock *sk;
+  int level;
+  int optlen;
+} __ctx__;
+
+extern int magic(void);
+extern int magic2(int);
+
+// The next test case is related to the issue described in the following thread:
+// https://lore.kernel.org/bpf/CAA-VZPmxh8o8EBcJ=[e-mail]/T/#m4b9ce2ce73b34f34172328f975235fc6f19841b6
+
+// FRONT-LABEL: define dso_local i32 @known_load_sink_example_1
+//   MID-LABEL: define dso_local i32 @known_load_sink_example_1
+//  BACK-LABEL: define dso_local i32 @known_load_sink_example_1
+int known_load_sink_example_1(struct bpf_sockopt *ctx)
+{
+  unsigned g = 0;
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[level:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.bpf_sockopt, ptr [[r1]], i32 0, i32 1
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[level]]
+// FRONT-NEXT:   switch i32 [[r2]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+// FRONT-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:   ]
+
+//   MID:        [[level3:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:      @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 1)
+//   MID-NEXT:   switch i32 [[level3]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//   MID-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:   ]
+
+//  BACK:        [[level39:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.bpf_sockopt, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load i32, ptr [[level39]]
+//  BACK-NEXT:   switch i32 [[r0]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//  BACK-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:   ]
+  switch (ctx->level) {
+  case 10:
+    g = magic2(ctx->sk->family);
+    break;
+// FRONT:        [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[sk:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.bpf_sockopt, ptr [[r4]], i32 0, i32 0
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load ptr, ptr [[sk]]
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r5]])
+// FRONT-NEXT:   [[family:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.bpf_sock, ptr [[r6]], i32 0, i32 1
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = load i32, ptr [[family]]
+// FRONT-NEXT:   [[call:%[a-zA-Z0-9._]+]] = call i32 @magic2(i32 noundef [[r7]])
+// FRONT-NEXT:   store i32 [[call]], ptr %g
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.bb:
+//   MID-NEXT:   [[sk4:%[a-zA-Z0-9._]+]] = tail call ptr (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.p0(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 3, i1 true, i32 0, i32 0)
+//   MID-NEXT:   [[family5:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sock) [[sk4]], i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   br label [[sw_epilog_sink_split:%[a-zA-Z0-9._]+]]
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[sk410:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.bpf_sockopt, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load ptr, ptr [[sk410]]
+//  BACK-NEXT:   [[family511:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.bpf_sock, ptr [[r1]], i32 0, i32 1
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[family511]]
+//  BACK-NEXT:   br label %sw.epilog.sink.split
+
+  case 20:
+    g = magic2(ctx->optlen);
+    break;
+// FRONT:        [[r8:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r9:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r8]])
+// FRONT-NEXT:   [[optlen:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.bpf_sockopt, ptr [[r9]], i32 0, i32 2
+// FRONT-NEXT:   [[r10:%[a-zA-Z0-9._]+]] = load i32, ptr [[optlen]]
+// FRONT-NEXT:   [[call2:%[a-zA-Z0-9._]+]] = call i32 @magic2(i32 noundef [[r10]])
+// FRONT-NEXT:   store i32 [[call2]], ptr %g
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.{{.*}}:
+//   MID-NEXT:   [[optlen6:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.bpf_sockopt) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2)
+//   MID-NEXT:   br label %sw.epilog.sink.split
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[optlen612:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.bpf_sockopt, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[optlen612]]
+//  BACK-NEXT:   br label %sw.epilog.sink.split
+  }
+// FRONT:      sw.epilog:
+// FRONT-NEXT:   [[r11:%[a-zA-Z0-9._]+]] = load i32, ptr %g
+// FRONT-NEXT:   [[rem:%[a-zA-Z0-9._]+]] = urem i32 [[r11]], 2
+// FRONT:        ret i32 [[rem]]
+
+//   MID:      sw.epilog.sink.split:
+//   MID-NEXT:   [[optlen6_sink:%[a-zA-Z0-9._]+]] = phi i32 [ [[optlen6]], [[sw_bb1]] ], [ [[family5]], [[sw_bb]] ]
+//   MID-NEXT:   [[call2:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[optlen6_sink]])
+//   MID-NEXT:   [[phi_bo:%[a-zA-Z0-9._]+]] = and i32 [[call2]], 1
+//   MID-NEXT:   br label %sw.epilog
+
+//   MID:      sw.epilog:
+//   MID-NEXT:   [[g_0:%[a-zA-Z0-9._]+]] = phi i32 [ 0, %entry ], [ %phi.bo, %sw.epilog.sink.split ]
+//   MID-NEXT:   ret i32 [[g_0]]
+
+//  BACK:      sw.epilog.sink.split:
+//  BACK-NEXT:   [[optlen6_sink:%[a-zA-Z0-9._]+]] = phi i32 [ [[r3]], [[sw_bb1]] ], [ [[r2]], [[sw_bb]] ]
+//  BACK-NEXT:   [[call2:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[optlen6_sink]])
+//  BACK-NEXT:   [[phi_bo:%[a-zA-Z0-9._]+]] = and i32 [[call2]], 1
+//  BACK-NEXT:   br label %sw.epilog
+
+//  BACK:      sw.epilog:
+//  BACK-NEXT:   [[g_0:%[a-zA-Z0-9._]+]] = phi
+//  BACK-NEXT:   ret i32 [[g_0]]
+
+  return g % 2;
+}
+
+struct __sk_buff {
+  int priority;
+  int mark;
+  int tc_index;
+} __ctx__;
+
+// FRONT-LABEL: define dso_local i32 @known_store_sink_example_1
+//   MID-LABEL: define dso_local i32 @known_store_sink_example_1
+//  BACK-LABEL: define dso_local i32 @known_store_sink_example_1
+int known_store_sink_example_1(struct __sk_buff *ctx) {
+  switch (ctx->priority) {
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[priority:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r1]], i32 0, i32 0
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[priority]]
+// FRONT-NEXT:   switch i32 [[r2]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+// FRONT-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:   ]
+
+//   MID:        [[priority3:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   switch i32 [[priority3]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//   MID-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:   ]
+
+//  BACK:        [[priority36:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load i32, ptr [[priority36]]
+//  BACK-NEXT:   switch i32 [[r0]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//  BACK-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:     i32 20, label [[sw_bb1:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:   ]
+  case 10:
+    ctx->mark = 3;
+    break;
+// FRONT:      sw.bb:
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[mark:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   store i32 3, ptr [[mark]]
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.bb:
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 3, ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   br label %sw.epilog
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   store i32 3, ptr [[r1]]
+//  BACK-NEXT:   br label %sw.epilog
+
+  case 20:
+    ctx->priority = 4;
+    break;
+// FRONT:      sw.bb1:
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r5]])
+// FRONT-NEXT:   [[priority2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r6]], i32 0, i32 0
+// FRONT-NEXT:   store i32 4, ptr [[priority2]]
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.bb1:
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 4, ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   br label %sw.epilog
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   store i32 4, ptr [[r2]]
+//  BACK-NEXT:   br label %sw.epilog
+  }
+// FRONT:      sw.epilog:
+// FRONT-NEXT:   ret i32 0
+//   MID:      sw.epilog:
+//   MID-NEXT:   ret i32 0
+//  BACK:      sw.epilog:
+//  BACK-NEXT:   ret i32 0
+  return 0;
+}
+
+// FRONT-LABEL: define dso_local i32 @known_store_sink_example_2
+//   MID-LABEL: define dso_local i32 @known_store_sink_example_2
+//  BACK-LABEL: define dso_local i32 @known_store_sink_example_2
+int known_store_sink_example_2(struct __sk_buff *ctx) {
+  switch (ctx->tc_index) {
+// FRONT:        [[r0:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r0]])
+// FRONT-NEXT:   [[tc_index:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r1]], i32 0, i32 2
+// FRONT-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = load i32, ptr [[tc_index]]
+// FRONT-NEXT:   switch i32 [[r2]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+// FRONT-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:     i32 20, label [[sw_bb3:%[a-zA-Z0-9._]+]]
+// FRONT-NEXT:   ]
+
+//   MID:        [[tc_index8:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 2)
+//   MID-NEXT:   switch i32 [[tc_index8]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//   MID-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:     i32 20, label [[sw_bb3:%[a-zA-Z0-9._]+]]
+//   MID-NEXT:   ]
+
+//  BACK:        [[tc_index814:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 2
+//  BACK-NEXT:   [[r0:%[a-zA-Z0-9._]+]] = load i32, ptr [[tc_index814]]
+//  BACK-NEXT:   switch i32 [[r0]], label [[sw_epilog:%[a-zA-Z0-9._]+]] [
+//  BACK-NEXT:     i32 10, label [[sw_bb:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:     i32 20, label [[sw_bb3:%[a-zA-Z0-9._]+]]
+//  BACK-NEXT:   ]
+
+  case 10:
+    magic2(ctx->mark);
+    ctx->mark = magic();
+    break;
+// FRONT:      sw.bb:
+// FRONT-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r3]])
+// FRONT-NEXT:   [[mark:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r4]], i32 0, i32 1
+// FRONT-NEXT:   [[r5:%[a-zA-Z0-9._]+]] = load i32, ptr [[mark]]
+// FRONT-NEXT:   [[call:%[a-zA-Z0-9._]+]] = call i32 @magic2(i32 noundef [[r5]])
+// FRONT-NEXT:   [[call1:%[a-zA-Z0-9._]+]] = call i32 @magic()
+// FRONT-NEXT:   [[r6:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r7:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r6]])
+// FRONT-NEXT:   [[mark2:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r7]], i32 0, i32 1
+// FRONT-NEXT:   store i32 [[call1]], ptr [[mark2]]
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.bb:
+//   MID-NEXT:   [[mark7:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   [[call:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[mark7]])
+//   MID-NEXT:   [[call1:%[a-zA-Z0-9._]+]] = tail call i32 @magic()
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 [[call1]], ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 1)
+//   MID-NEXT:   br label %sw.epilog
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[mark715:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   [[r1:%[a-zA-Z0-9._]+]] = load i32, ptr [[mark715]]
+//  BACK-NEXT:   [[call:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[r1]])
+//  BACK-NEXT:   [[call1:%[a-zA-Z0-9._]+]] = tail call i32 @magic()
+//  BACK-NEXT:   [[r2:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 1
+//  BACK-NEXT:   store i32 [[call1]], ptr [[r2]]
+//  BACK-NEXT:   br label %sw.epilog
+
+  case 20:
+    magic2(ctx->priority);
+    ctx->priority = magic();
+    break;
+// FRONT:      sw.{{.*}}:
+// FRONT-NEXT:   [[r8:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r9:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r8]])
+// FRONT-NEXT:   [[priority:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r9]], i32 0, i32 0
+// FRONT-NEXT:   [[r10:%[a-zA-Z0-9._]+]] = load i32, ptr [[priority]]
+// FRONT-NEXT:   [[call4:%[a-zA-Z0-9._]+]] = call i32 @magic2(i32 noundef [[r10]])
+// FRONT-NEXT:   [[call5:%[a-zA-Z0-9._]+]] = call i32 @magic()
+// FRONT-NEXT:   [[r11:%[a-zA-Z0-9._]+]] = load ptr, ptr {{.*}}
+// FRONT-NEXT:   [[r12:%[a-zA-Z0-9._]+]] = call ptr @llvm.bpf.context.marker.p0.p0(ptr [[r11]])
+// FRONT-NEXT:   [[priority6:%[a-zA-Z0-9._]+]] = getelementptr inbounds %struct.__sk_buff, ptr [[r12]], i32 0, i32 0
+// FRONT-NEXT:   store i32 [[call5]], ptr [[priority6]]
+// FRONT-NEXT:   br label %sw.epilog
+
+//   MID:      sw.{{.*}}:
+//   MID-NEXT:   [[priority9:%[a-zA-Z0-9._]+]] = tail call i32 (ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.load.i32(ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   [[call4:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[priority9]])
+//   MID-NEXT:   [[call5:%[a-zA-Z0-9._]+]] = tail call i32 @magic()
+//   MID-NEXT:   tail call void (i32, ptr, i1, i8, i8, i8, i1, ...)
+//   MID-SAME:     @llvm.bpf.getelementptr.and.store.i32(i32 [[call5]], ptr elementtype(%struct.__sk_buff) %ctx, i1 false, i8 0, i8 1, i8 2, i1 true, i32 0, i32 0)
+//   MID-NEXT:   br label %sw.epilog
+
+//  BACK:      sw.{{.*}}:
+//  BACK-NEXT:   [[priority916:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   [[r3:%[a-zA-Z0-9._]+]] = load i32, ptr [[priority916]]
+//  BACK-NEXT:   [[call4:%[a-zA-Z0-9._]+]] = tail call i32 @magic2(i32 noundef [[r3]])
+//  BACK-NEXT:   [[call5:%[a-zA-Z0-9._]+]] = tail call i32 @magic()
+//  BACK-NEXT:   [[r4:%[a-zA-Z0-9._]+]] = getelementptr  inbounds %struct.__sk_buff, ptr %ctx, i32 0, i32 0
+//  BACK-NEXT:   store i32 [[call5]], ptr [[r4]]
+//  BACK-NEXT:   br label %sw.epilog
+  }
+  return 0;
+// FRONT:      sw.epilog:
+// FRONT-NEXT:   ret i32 0
+//   MID:      sw.epilog:
+//   MID-NEXT:   ret i32 0
+//  BACK:      sw.epilog:
+//  BACK-NEXT:   ret i32 0
+}

llvm/include/llvm/IR/Intrinsics.td

def int_preserve_union_access_index : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                                 [IntrNoMem,
                                                  ImmArg<ArgIndex<1>>]>;
 def int_preserve_struct_access_index : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
                                                  [llvm_anyptr_ty, llvm_i32_ty,
                                                   llvm_i32_ty],
                                                  [IntrNoMem,
                                                   ImmArg<ArgIndex<1>>,
                                                   ImmArg<ArgIndex<2>>]>;
+def int_bpf_context_marker : DefaultAttrsIntrinsic<[llvm_anyptr_ty],
+                                                   [llvm_anyptr_ty],
+                                                   [IntrNoMem]>;
 
 //===------------ Intrinsics to perform common vector shuffles ------------===//
 
 def int_experimental_vector_reverse : DefaultAttrsIntrinsic<[llvm_anyvector_ty],
                                                             [LLVMMatchType<0>],
                                                             [IntrNoMem]>;
 
 def int_experimental_vector_splice : DefaultAttrsIntrinsic<[llvm_anyvector_ty],

llvm/include/llvm/IR/IntrinsicsBPF.td

let TargetPrefix = "bpf" in {  // All intrinsics start with "llvm.bpf."
   def int_bpf_preserve_enum_value : ClangBuiltin<"__builtin_bpf_preserve_enum_value">,
               Intrinsic<[llvm_i64_ty], [llvm_i32_ty, llvm_ptr_ty, llvm_i64_ty],
               [IntrNoMem]>;
   def int_bpf_passthrough : ClangBuiltin<"__builtin_bpf_passthrough">,
               Intrinsic<[llvm_any_ty], [llvm_i32_ty, llvm_any_ty], [IntrNoMem]>;
   def int_bpf_compare : ClangBuiltin<"__builtin_bpf_compare">,
               Intrinsic<[llvm_i1_ty], [llvm_i32_ty, llvm_anyint_ty, llvm_anyint_ty],
               [IntrNoMem]>;
+  def int_bpf_getelementptr_and_load : ClangBuiltin<"__builtin_bpf_getelementptr_and_load">,
+              Intrinsic<[llvm_any_ty],
+                        [llvm_ptr_ty,     // base ptr for getelementptr
+                         llvm_i1_ty,      // volatile
+                         llvm_i8_ty,      // atomic order
+                         llvm_i8_ty,      // synscope id
+                         llvm_i8_ty,      // alignment
+                         llvm_i1_ty,      // inbounds
+                         llvm_vararg_ty], // indices for getelementptr insn
+                        [IntrReadMem,
+                         IntrArgMemOnly,
+                         NoCapture <ArgIndex<0>>,
+                         ReadOnly  <ArgIndex<0>>,
+                         ImmArg    <ArgIndex<1>>, // volatile
+                         ImmArg    <ArgIndex<2>>, // atomic order
+                         ImmArg    <ArgIndex<3>>, // synscope id
+                         ImmArg    <ArgIndex<4>>, // alignment
+                         ImmArg    <ArgIndex<5>>, // inbounds
+                        ]>;
+  def int_bpf_getelementptr_and_store : ClangBuiltin<"__builtin_bpf_getelementptr_and_store">,
+              Intrinsic<[],
+                        [llvm_any_ty,     // value to store
+                         llvm_ptr_ty,     // base ptr for getelementptr
+                         llvm_i1_ty,      // volatile
+                         llvm_i8_ty,      // atomic order
+                         llvm_i8_ty,      // syncscope id
+                         llvm_i8_ty,      // alignment
+                         llvm_i1_ty,      // inbounds
+                         llvm_vararg_ty], // indexes for getelementptr insn
+                        [IntrWriteMem,
+                         IntrArgMemOnly,
+                         NoCapture <ArgIndex<1>>,
+                         WriteOnly <ArgIndex<1>>,
+                         ImmArg    <ArgIndex<2>>, // volatile
+                         ImmArg    <ArgIndex<3>>, // atomic order
+                         ImmArg    <ArgIndex<4>>, // syncscope id
+                         ImmArg    <ArgIndex<5>>, // alignment
+                         ImmArg    <ArgIndex<6>>, // inbounds
+                        ]>;
 }

llvm/lib/Target/BPF/BPF.h

 FunctionPass *createBPFPreserveDIType();
 FunctionPass *createBPFIRPeephole();
 FunctionPass *createBPFISelDag(BPFTargetMachine &TM);
 FunctionPass *createBPFMISimplifyPatchablePass();
 FunctionPass *createBPFMIPeepholePass();
 FunctionPass *createBPFMIPeepholeTruncElimPass();
 FunctionPass *createBPFMIPreEmitPeepholePass();
 FunctionPass *createBPFMIPreEmitCheckingPass();
+FunctionPass *createBPFRewriteContextAccessPass();
 
 void initializeBPFAdjustOptPass(PassRegistry&);
 void initializeBPFCheckAndAdjustIRPass(PassRegistry&);
 
 void initializeBPFAbstractMemberAccessLegacyPassPass(PassRegistry &);
 void initializeBPFPreserveDITypePass(PassRegistry&);
 void initializeBPFIRPeepholePass(PassRegistry&);
 void initializeBPFMISimplifyPatchablePass(PassRegistry&);
 void initializeBPFMIPeepholePass(PassRegistry&);
 void initializeBPFMIPeepholeTruncElimPass(PassRegistry&);
 void initializeBPFMIPreEmitPeepholePass(PassRegistry&);
 void initializeBPFMIPreEmitCheckingPass(PassRegistry&);
+void initializeBPFRewriteContextAccessLegacyPassPass(PassRegistry &);
 
 class BPFAbstractMemberAccessPass
     : public PassInfoMixin<BPFAbstractMemberAccessPass> {
   BPFTargetMachine *TM;
 
 public:
   BPFAbstractMemberAccessPass(BPFTargetMachine *TM) : TM(TM) {}
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
public:
 
   static bool isRequired() { return true; }
 };
 
 class BPFAdjustOptPass : public PassInfoMixin<BPFAdjustOptPass> {
 public:
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
 };
+
+class BPFRewriteContextAccessPass
+    : public PassInfoMixin<BPFRewriteContextAccessPass> {
+public:
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+
+  static bool isRequired() { return true; }
+};
+
 } // namespace llvm
 
 #endif

llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp

 //===------------ BPFCheckAndAdjustIR.cpp - Check and Adjust IR -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Check IR and adjust IR for verifier friendly codes.
 // The following are done for IR checking:
 //   - no relocation globals in PHI node.
 // The following are done for IR adjustment:
 //   - remove __builtin_bpf_passthrough builtins. Target independent IR
 //     optimizations are done and those builtins can be removed.
+//   - remove llvm.bpf.getelementptr.and.load builtins.
+//   - remove llvm.bpf.getelementptr.and.store builtins.
 //
 //===----------------------------------------------------------------------===//
 
 #include "BPF.h"
 #include "BPFCORE.h"
 #include "BPFTargetMachine.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicsBPF.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Pass.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 
 #define DEBUG_TYPE "bpf-check-and-opt-ir"
public:
   static char ID;
   BPFCheckAndAdjustIR() : ModulePass(ID) {}
 
 private:
   void checkIR(Module &M);
   bool adjustIR(Module &M);
   bool removePassThroughBuiltin(Module &M);
   bool removeCompareBuiltin(Module &M);
+  bool removeGEPBuiltins(Module &M);
 };
 } // End anonymous namespace
 
 char BPFCheckAndAdjustIR::ID = 0;
 INITIALIZE_PASS(BPFCheckAndAdjustIR, DEBUG_TYPE, "BPF Check And Adjust IR",
                 false, false)
 
 ModulePass *llvm::createBPFCheckAndAdjustIR() {
for (auto &BB : F)
         BB.getInstList().insert(Call->getIterator(), ICmp);
 
         Call->replaceAllUsesWith(ICmp);
         ToBeDeleted = Call;
       }
   return Changed;
 }
 
+static unsigned getOperandAsUnsigned(CallInst *Call, unsigned ArgNo) {
+  if (auto *Int = dyn_cast<ConstantInt>(Call->getOperand(ArgNo)))
+    return Int->getValue().getZExtValue();
+  std::string Report;
+  raw_string_ostream ReportS(Report);
+  ReportS << "Expecting ConstantInt as argument #" << ArgNo << " of " << *Call
+          << "\n";
+  report_fatal_error(StringRef(Report));
+}
+
+static GetElementPtrInst *reconstructGEP(CallInst *Call, int Delta) {
+  SmallVector<Value *> Indices;
+  Indices.append(Call->data_operands_begin() + 6 + Delta,
+                 Call->data_operands_end());
+  auto *GEPPointeeType = Call->getParamElementType(Delta);
+  auto *GEP = GetElementPtrInst::Create(GEPPointeeType, Call->getOperand(Delta),
+                                        ArrayRef<Value *>(Indices),
+                                        Call->getName(), Call);
+  GEP->setIsInBounds(getOperandAsUnsigned(Call, 5 + Delta));
+  return GEP;
+}
+
+template <class T = std::disjunction<LoadInst, StoreInst>>
+static void unrollCommon(CallInst *Call, GetElementPtrInst *GEP, T *Insn,
+                         int Delta) {
+  Insn->setVolatile(getOperandAsUnsigned(Call, 1 + Delta));
+  Insn->setOrdering((AtomicOrdering)getOperandAsUnsigned(Call, 2 + Delta));
+  Insn->setSyncScopeID(getOperandAsUnsigned(Call, 3 + Delta));
+  auto AlignShiftValue = getOperandAsUnsigned(Call, 4 + Delta);
+  Insn->setAlignment(Align(uint64_t(1) << AlignShiftValue));
+  GEP->setDebugLoc(Call->getDebugLoc());
+  Insn->setDebugLoc(Call->getDebugLoc());
+  Call->replaceAllUsesWith(Insn);
+  Call->eraseFromParent();
+}
+
+static void unrollGEPLoad(CallInst *Call) {
+  auto *GEP = reconstructGEP(Call, 0);
+  auto *ReturnType = Call->getFunctionType()->getReturnType();
+  auto *Load = new LoadInst(ReturnType, GEP, "", Call);
+  unrollCommon(Call, GEP, Load, 0);
+}
+
+static void unrollGEPStore(CallInst *Call) {
+  auto *GEP = reconstructGEP(Call, 1);
+  auto *Store = new StoreInst(Call->getOperand(0), GEP, Call);
+  unrollCommon(Call, GEP, Store, 1);
+}
+
+static bool removeGEPBuiltinsInFunc(Function &F) {
+  SmallVector<CallInst *> GEPLoads;
+  SmallVector<CallInst *> GEPStores;
+  for (auto &BB : F)
+    for (auto &Insn : BB)
+      if (auto *Call = dyn_cast<CallInst>(&Insn))
+        if (auto *Called = Call->getCalledFunction())
+          switch (Called->getIntrinsicID()) {
+          case Intrinsic::bpf_getelementptr_and_load:
+            GEPLoads.push_back(Call);
+            break;
+          case Intrinsic::bpf_getelementptr_and_store:
+            GEPStores.push_back(Call);
+            break;
+          }
+
+  if (GEPLoads.empty() && GEPStores.empty())
+    return false;
+
+  for_each(GEPLoads, unrollGEPLoad);
+  for_each(GEPStores, unrollGEPStore);
+
+  return true;
+}
+
+// Rewrites the following builtins:
+// - llvm.bpf.getelementptr.and.load
+// - llvm.bpf.getelementptr.and.store
+// As (load (getelementptr ...)) or (store (getelementptr ...)).
+bool BPFCheckAndAdjustIR::removeGEPBuiltins(Module &M) {
+  bool Changed = false;
+  for (auto &F : M)
+    Changed = removeGEPBuiltinsInFunc(F) || Changed;
+  return Changed;
+}
+
 bool BPFCheckAndAdjustIR::adjustIR(Module &M) {
   bool Changed = removePassThroughBuiltin(M);
   Changed = removeCompareBuiltin(M) || Changed;
+  Changed = removeGEPBuiltins(M) || Changed;
   return Changed;
 }
 
 bool BPFCheckAndAdjustIR::runOnModule(Module &M) {
   checkIR(M);
   return adjustIR(M);
 }

llvm/lib/Target/BPF/BPFContextMarker.cpp

+//===------ BPFContextMarker.cpp ------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// TLDR: replaces llvm.bpf.context.marker + GEP + load / store
+//           with llvm.bpf.getelementptr.and.load / store
+//
+// This implementations BPFRewriteContextAccessPass that address two
+// BPF verifier specific issues:
+//
+// (a) Access to some structural types is allowed only using
+//     LD / ST instructions with static immediate offsets;
+//
+//     Examples of such types are `struct __sk_buff` and `struct bpf_sock_ops`.
+//     This is so because offsets of the fields of these structures do
+//     not match real offsets in the running kernel. During BPF
+//     program load/verification loads and stores to the fields of
+//     these types are rewritten so that offsets match real
+//     offsets. For this rewrite to happen static offsets have to be
+//     encoded in the instructions.
+//
+//     See kernel/bpf/verifier.c:convert_ctx_access function in the
+//     Linux kernel source tree for details.
+//
+// (b) Pointers to context parameters of BPF programs must not be
+//     modified before access.
+//
+//     During BPF program verification a tag PTR_TO_CTX is tracked for
+//     register values. In case if register with such tag is modified
+//     BPF programs are not allowed to read or write memory using
+//     register. See kernel/bpf/verifier.c:check_mem_access function in the
+//     Linux kernel source tree for details.
+//
+// Both cases limit the ways structural types could be accessed.
+// After AST translation the IR for structural types access has the
+// following form:
+//
+//   %x = getelementptr %ptr, %offset
+//   %y = load %x
+//
+// Instruction selection for BPF translates such series of
+// instructions as a single instruction:
+//
+//   LDW %ptr, %offset
+//
+// However, several optimization passes might sink `load` instruction
+// or hoist `getelementptr` instruction so that the instructions are
+// no longer in sequence. Examples of such passes are:
+// SimplifyCFGPass, InstCombinePass, GVNPass.
+// After such modification the verifier would reject the BPF program.
+//
+// To avoid this issue the patterns like (load/store (getelementptr ...))
+// are replaced by calls to BPF specific intrinsic functions:
+// - llvm.bpf.getelementptr.and.load
+// - llvm.bpf.getelementptr.and.store
+//
+// These calls are lowered back to (load/store (getelementptr ...))
+// by BPFCheckAndAdjustIR pass right before the translation from IR to
+// machine instructions.
+//
+// The BPFRewriteContextAccessPass is responsible for this
+// transformation.  It uses calls to llvm.bpf.context.marker to
+// identify starting points of (load/store (gep (gep ...))) chains and
+// replaces these chains with calls to intrinsics.
+
+#include "BPF.h"
+#include "BPFCORE.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/IntrinsicsBPF.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include <set>
+
+#define DEBUG_TYPE "bpf-context-marker"
+
+using namespace llvm;
+
+static bool isNamedFuncCall(Value *I, const char *Name) {
+  auto *Call = dyn_cast<CallInst>(I);
+  if (!Call)
+    return false;
+  auto *Func = Call->getCalledFunction();
+  if (!Func)
+    return false;
+  return Func->getName().startswith(Name);
+}
+
+static bool isContextMarkerCall(Value *I) {
+  return isNamedFuncCall(I, "llvm.bpf.context.marker");
+}
+
+template <class T = Instruction>
+static const DILocation *mergeDILocations(SmallVector<T *> &Insns) {
+  const DILocation *Merged = (*Insns.begin())->getDebugLoc();
+  for (auto *I : Insns)
+    Merged = DILocation::getMergedLocation(Merged, I->getDebugLoc());
+  return Merged;
+}
+
+static CallInst *makeIntrinsicCall(Module *M,
+                                   Intrinsic::BPFIntrinsics Intrinsic,
+                                   ArrayRef<Type *> Types,
+                                   SmallVector<Value *> &Args) {
+
+  Function *Fn =
+    Intrinsic::getDeclaration(M, Intrinsic, ArrayRef<Type *>(Types));
+  return CallInst::Create(Fn, ArrayRef<Value *>(Args));
+}
+
+static void setParamElementType(CallInst *Call, unsigned ArgNo, Type *Type) {
+  auto &C = Call->getContext();
+  Call->addParamAttr(ArgNo, Attribute::get(C, Attribute::ElementType, Type));
+}
+
+static void setParamReadNone(CallInst *Call, unsigned ArgNo) {
+  auto &C = Call->getContext();
+  Call->addParamAttr(ArgNo, Attribute::get(C, Attribute::ReadNone));
+}
+
+namespace {
+struct GEPChainInfo {
+  bool InBounds = true;
+  Type *SourceElementType;
+  Type *ResultElementType;
+  SmallVector<Value *> Indices;
+  SmallVector<GetElementPtrInst *> Members;
+};
+} // Anonymous namespace
+
+template <class T = std::disjunction<LoadInst, StoreInst>>
+static void fillCommonArgs(SmallVector<Value *> &Args,
+                           GEPChainInfo &GEP,
+                           T *Insn) {
+  auto &C = Insn->getContext();
+  auto *Int8Ty = Type::getInt8Ty(C);
+  auto *Int1Ty = Type::getInt1Ty(C);
+  // Implementation of Align guarantees that ShiftValue < 64
+  auto AlignShiftValue = Log2_64(Insn->getAlign().value());
+  Args.push_back(GEP.Members[0]->getPointerOperand());
+  Args.push_back(ConstantInt::get(Int1Ty, Insn->isVolatile()));
+  Args.push_back(ConstantInt::get(Int8Ty, (unsigned) Insn->getOrdering()));
+  Args.push_back(ConstantInt::get(Int8Ty, (unsigned) Insn->getSyncScopeID()));
+  Args.push_back(ConstantInt::get(Int8Ty, AlignShiftValue));
+  Args.push_back(ConstantInt::get(Int1Ty, GEP.InBounds));
+  Args.append(GEP.Indices.begin(), GEP.Indices.end());
+}
+
+static Instruction *makeGEPAndLoad(GEPChainInfo &GEP,
+                                   LoadInst *Load) {
+  auto *M = Load->getModule();
+  SmallVector<Value *> Args;
+  fillCommonArgs(Args, GEP, Load);
+  auto *Call = makeIntrinsicCall(M, Intrinsic::bpf_getelementptr_and_load,
+                                 {Load->getType()}, Args);
+  setParamElementType(Call, 0, GEP.SourceElementType);
+  Call->applyMergedLocation(mergeDILocations(GEP.Members),
+                            Load->getDebugLoc());
+  Call->setName((*GEP.Members.rbegin())->getName());
+  return Call;
+}
+
+static Instruction *makeGEPAndStore(GEPChainInfo &GEP,
+                                    StoreInst *Store) {
+  auto *M = Store->getModule();
+  SmallVector<Value *> Args;
+  Args.push_back(Store->getValueOperand());
+  fillCommonArgs(Args, GEP, Store);
+  auto *Call = makeIntrinsicCall(M, Intrinsic::bpf_getelementptr_and_store,
+                                 {Store->getValueOperand()->getType()}, Args);
+  setParamElementType(Call, 1, GEP.SourceElementType);
+  if (Store->getValueOperand()->getType()->isPointerTy())
+    setParamReadNone(Call, 0);
+  Call->applyMergedLocation(mergeDILocations(GEP.Members),
+                            Store->getDebugLoc());
+  return Call;
+}
+
+// Given a chain of GEP instructions collect information necessary to
+// merge this chain as a single instruction.
+// Return `true` is such merge is possible, `false` otherwise.
+static bool collectGEPChainInfo(SmallVector<GetElementPtrInst *> &GEPs,
+                                GEPChainInfo &Info) {
+  if (GEPs.empty())
+    return false;
+
+  auto *First = GEPs[0];
+  Info.InBounds = First->isInBounds();
+  Info.SourceElementType = First->getSourceElementType();
+  Info.ResultElementType = First->getResultElementType();
+  Info.Indices.append(First->idx_begin(), First->idx_end());
+  Info.Members.push_back(First);
+
+  for (auto *Iter = GEPs.begin() + 1; Iter != GEPs.end(); ++Iter) {
+    auto *GEP = *Iter;
+    auto *CI = dyn_cast<ConstantInt>(GEP->idx_begin());
+    if (!CI || !CI->isZero())
+      return false;
+    if (!GEP->getSourceElementType() ||
+        GEP->getSourceElementType() != Info.ResultElementType)
+      return false;
+    Info.InBounds &= GEP->isInBounds();
+    Info.Indices.append(GEP->idx_begin() + 1, GEP->idx_end());
+    Info.Members.push_back(GEP);
+    Info.ResultElementType = GEP->getResultElementType();
+  }
+
+  return true;
+}
+
+static bool isPointerOperand(Instruction *I, User *U) {
+  if (auto *L = dyn_cast<LoadInst>(U))
+    return L->getPointerOperand() == I;
+  if (auto *S = dyn_cast<StoreInst>(U))
+    return S->getPointerOperand() == I;
+  if (auto *GEP = dyn_cast<GetElementPtrInst>(U))
+    return GEP->getPointerOperand() == I;
+  return true;
+}
+
+// A DFS traversal of GEP chain trees starting from Root.
+//
+// Recursion descends through GEP instructions and
+// llvm.bpf.context.marker calls. Recursion stops at any other
+// instruction. If load or store instruction is reached at the end
+// it is replaced by the call to `llvm.bpf.getelementptr.and.load` or
+// `llvm.bpf.getelementptr.and.store` intrinsic.
+//
+// Parameters description:
+// - Root - a call to llvm.bpf.context.marker, the root of the tree
+// - Insn - current position in the tree
+// - GEPs - GEP instructions for the current branch
+// - Visited - a list of visited instructions in DFS order,
+//   order is important for unused instruction deletion.
+static void rewriteAccessChain(Instruction *Root,
+                               Instruction *Insn,
+                               SmallVector<GetElementPtrInst *> &GEPs,
+                               SmallVector<Instruction *> &Visited) {
+  auto MarkAndTraverseUses = [&]() {
+    Visited.push_back(Insn);
+    for (auto *U : Insn->users())
+      if (auto *UI = dyn_cast<Instruction>(U))
+        if (isPointerOperand(Insn, UI))
+          rewriteAccessChain(Root, UI, GEPs, Visited);
+  };
+  if (isa<LoadInst>(Insn) || isa<StoreInst>(Insn)) {
+    GEPChainInfo GEPChain;
+    if (!collectGEPChainInfo(GEPs, GEPChain)) {
+      errs() << "Can't generate gep and load/store for "
+             << *Root << ": irreducable GEP chain\n";
+      return;
+    }
+    Visited.push_back(Insn);
+    if (auto *Load = dyn_cast<LoadInst>(Insn)) {
+      auto *Replacement = makeGEPAndLoad(GEPChain, Load);
+      Replacement->insertBefore(Load);
+      Load->replaceAllUsesWith(Replacement);
+    }
+    if (auto *Store = dyn_cast<StoreInst>(Insn)) {
+      auto *Replacement = makeGEPAndStore(GEPChain, Store);
+      Replacement->insertBefore(Store);
+    }
+  } else if (auto *GEP = dyn_cast<GetElementPtrInst>(Insn)) {
+    GEPs.push_back(GEP);
+    MarkAndTraverseUses();
+    GEPs.pop_back();
+  }  else if (isContextMarkerCall(Insn)) {
+    MarkAndTraverseUses();
+  }
+}
+
+// Rewrite the GEP chains starting at Marker and delete instructions
+// that are not used after rewrite.
+static void rewriteAccessChain(Instruction *Marker) {
+  SmallVector<GetElementPtrInst *> GEPs;
+  SmallVector<Instruction *> Visited;
+  for (auto *U : Marker->users())
+    if (auto *UI = dyn_cast<Instruction>(U))
+      if (isPointerOperand(Marker, UI))
+        rewriteAccessChain(Marker, UI, GEPs, Visited);
+  // Check if Visited instructions could be removed, iterate in
+  // reverse to unblock instructions higher in the chain.
+  for (auto V = Visited.rbegin(); V != Visited.rend(); ++V)
+    if ((*V)->use_empty())
+      (*V)->eraseFromParent();
+}
+
+// Note: order matters, see comment in rewriteContextAccess below
+static std::vector<Instruction *> collectContextMarkers(Function &F) {
+  std::vector<Instruction *> Markers;
+  for (auto &Insn : instructions(F))
+    if (isContextMarkerCall(&Insn))
+      Markers.push_back(&Insn);
+  return Markers;
+}
+
+static void removeMarker(Instruction *Marker) {
+    Marker->replaceAllUsesWith(Marker->getOperand(0));
+    Marker->eraseFromParent();
+}
+
+static bool rewriteContextAccess(Function &F) {
+  LLVM_DEBUG(dbgs() << "********** Rewrite Context Access ************\n");
+
+  auto ContextMarkers = collectContextMarkers(F);
+
+  LLVM_DEBUG(dbgs()
+             << "There are " << ContextMarkers.size()
+             << " context markers\n");
+
+  if (ContextMarkers.empty())
+    return false;
+
+  for (auto *Marker : ContextMarkers) {
+    // For a series of nested accesses like `a.b.c` there would be
+    // multiple marker calls inserted when all accessed structures are
+    // marked with btf_decl_tag("ctx"), e.g.:
+    //
+    //       %m0 = call @llvm.bpf.context.marker(%a)
+    //   %b.addr = getelementptr %m0 ...
+    //       %m1 = call @llvm.bpf.context.marker(%b.addr)
+    //   %c.addr = getelementptr %m1 ...
+    //        %c = load %c.addr
+    //
+    // The `collectContextMarkers` will return a vector with %m0
+    // preceding the %m1. The `rewriteAccessChain` would remove %m1.
+    // Thus the parent check is needed to avoid removing %m1 twice.
+    if (!Marker->getParent())
+      continue;
+    rewriteAccessChain(Marker);
+    removeMarker(Marker);
+  }
+
+  return true;
+}
+
+namespace {
+
+class BPFRewriteContextAccessLegacyPass final : public FunctionPass {
+public:
+  static char ID;
+
+  BPFRewriteContextAccessLegacyPass() : FunctionPass(ID) {}
+
+  bool runOnFunction(Function &F) override {
+    return rewriteContextAccess(F);
+  }
+};
+
+} // End anonymous namespace
+
+char BPFRewriteContextAccessLegacyPass::ID = 0;
+INITIALIZE_PASS(BPFRewriteContextAccessLegacyPass, DEBUG_TYPE,
+                "BPF Rewrite Context Access", false, false)
+
+FunctionPass *llvm::createBPFRewriteContextAccessPass() {
+  return new BPFRewriteContextAccessLegacyPass();
+}
+
+PreservedAnalyses
+llvm::BPFRewriteContextAccessPass::run(Function &F, FunctionAnalysisManager &AM) {
+  return rewriteContextAccess(F)
+    ? PreservedAnalyses::none()
+    : PreservedAnalyses::all();
+}

llvm/lib/Target/BPF/BPFTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeBPFTarget() {
   PassRegistry &PR = *PassRegistry::getPassRegistry();
   initializeBPFAbstractMemberAccessLegacyPassPass(PR);
   initializeBPFPreserveDITypePass(PR);
   initializeBPFIRPeepholePass(PR);
   initializeBPFAdjustOptPass(PR);
   initializeBPFCheckAndAdjustIRPass(PR);
   initializeBPFMIPeepholePass(PR);
   initializeBPFMIPeepholeTruncElimPass(PR);
+  initializeBPFRewriteContextAccessLegacyPassPass(PR);
 }
 
 // DataLayout: little or big endian
 static std::string computeDataLayout(const Triple &TT) {
   if (TT.getArch() == Triple::bpfeb)
     return "E-m:e-p:64:64-i64:64-i128:128-n32:64-S128";
   else
     return "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128";

 };
 }
 
 TargetPassConfig *BPFTargetMachine::createPassConfig(PassManagerBase &PM) {
   return new BPFPassConfig(*this, PM);
 }
 
 void BPFTargetMachine::adjustPassManager(PassManagerBuilder &Builder) {
   Builder.addExtension(
       PassManagerBuilder::EP_EarlyAsPossible,
       [&](const PassManagerBuilder &, legacy::PassManagerBase &PM) {
         PM.add(createBPFAbstractMemberAccess(this));
         PM.add(createBPFPreserveDIType());
+        PM.add(createBPFRewriteContextAccessPass());
         PM.add(createBPFIRPeephole());
       });
 
   Builder.addExtension(
       PassManagerBuilder::EP_Peephole,
       [&](const PassManagerBuilder &, legacy::PassManagerBase &PM) {
         PM.add(createCFGSimplificationPass(
             SimplifyCFGOptions().hoistCommonInsts(true)));
       });
   Builder.addExtension(
       PassManagerBuilder::EP_ModuleOptimizerEarly,
       [&](const PassManagerBuilder &, legacy::PassManagerBase &PM) {
         PM.add(createBPFAdjustOpt());
       });
 }
 
 void BPFTargetMachine::registerPassBuilderCallbacks(PassBuilder &PB) {
   PB.registerPipelineStartEPCallback(
       [=](ModulePassManager &MPM, OptimizationLevel) {
         FunctionPassManager FPM;
         FPM.addPass(BPFAbstractMemberAccessPass(this));
         FPM.addPass(BPFPreserveDITypePass());
+        FPM.addPass(BPFRewriteContextAccessPass());
         FPM.addPass(BPFIRPeepholePass());
         MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
       });
   PB.registerPeepholeEPCallback([=](FunctionPassManager &FPM,
                                     OptimizationLevel Level) {
     FPM.addPass(SimplifyCFGPass(SimplifyCFGOptions().hoistCommonInsts(true)));
   });
   PB.registerPipelineEarlySimplificationEPCallback(

llvm/lib/Target/BPF/CMakeLists.txt

 
 add_public_tablegen_target(BPFCommonTableGen)
 
 add_llvm_target(BPFCodeGen
   BPFAbstractMemberAccess.cpp
   BPFAdjustOpt.cpp
   BPFAsmPrinter.cpp
   BPFCheckAndAdjustIR.cpp
+  BPFContextMarker.cpp
   BPFFrameLowering.cpp
   BPFInstrInfo.cpp
   BPFIRPeephole.cpp
   BPFISelDAGToDAG.cpp
   BPFISelLowering.cpp
   BPFMCInstLower.cpp
   BPFPreserveDIType.cpp
   BPFRegisterInfo.cpp