Page MenuHomePhabricator

BPF: make 32bit register spill with 64bit alignment
ClosedPublic

Authored by yonghong-song on Sep 1 2021, 10:20 AM.

Details

Summary

In llvm, for non-alu32 mode, the stack alignment is 64bit so only one
64bit spill per 64bit slot. For alu32 mode, the stack alignment
is 32bit, so it is possible to have two 32bit spills per
64bit slot.

Currently, bpf kernel verifier does not preserve register states
for 32bit spills. That is, one 32bit register may hold a constant
value or a bounded range before spill. After reload from the
stack, the information is lost and sometimes this may cause
verifier failure. For 64bit register spill, the verifier
indeed tries to preserve the register state for reloading.

The current verifier can be modestly changed to handle one
32bit spill per 64bit stack slot with state-preserving reload.
Handling two 32bit spills per 64bit stack slot will require
substantial changes.

This patch changes stack alignment for alu32 to be 64bit.
This way, for any 64bit slot in alu32 mode, only one
32bit or 64bit register values can be saved. Together
with previous-mentioned verifier enhancement, 32bit
spill can be handled with state preserving.

Note that llvm stack slot coallescing
seems only doing adjacent packing which may leave some holes
in the stack. For example,

stack slot 8   <== 8 bytes
stack slot 4   <== 8 bytes with 4 byte hole
stack slot 8   <== 8 bytes
stack slot 4   <== 4 bytes

Diff Detail

Event Timeline

yonghong-song created this revision.Sep 1 2021, 10:20 AM
yonghong-song requested review of this revision.Sep 1 2021, 10:20 AM
Herald added a project: Restricted Project. · View Herald TranscriptSep 1 2021, 10:20 AM

The following is the kernel bpf_verif_scale self test results w.r.t. stack size.

bpf selftest bpf_verif_scale (-mcpu=v3):
test_id  subtest name            old stack size   new stack size
12/1     loop3.o                 0                0
12/2     test_verif_scale1.o     8+0              8+0
12/3     test_verif_scale2.o     8                8
12/4     test_verif_scale3.o     8+0              8+0
12/5     pyperf_global.o         0+352            0+352
12/6     pyperf_subprogs.o       0+352+12         0+352+12
12/7     pyperf50.o              352              352
12/8     pyperf100.o             352              352
12/9     pyperf180.o             344              344
12/10    pyperf600.o             368              368
12/11    pyperf600_nounroll.o    320              320
12/12    loop1.o                 0                0
12/13    loop2.o                 0                0
12/14    loop4.o                 0                0
12/15    loop5.o                 0                0
12/16    loop6.o                 64               64
12/17    strobemeta.o            496              496
12/18    strobemeta_nounroll1.o  488              488
12/19    strobemeta_nounroll2.o  488              488
12/20    strobemeta_subprogs.o   24+416+0+32      24+416+0+32
12/21    test_sysctl_loop1.o     428              428
12/22    test_sysctl_loop2.o     300+80           300+80
12/23    test_xdp_loop.o         36               40
12/24    test_seg6_loop.o        88               96

Only test_xdp_loop.o and `test_seg6_loop.o' has a small increase. Overall, it seems the impact on stack size should be moderate.

@pchaigno this patch has a potential to increase stack size for alu32 (or -mcpu=v3) mode. I am not sure whether cilium uses -mcpu=v3 or not. Could you help check how much cilium bpf program stack size increases under -mcpu=v3 and whether the verification still succeed? You can use verifier log level 4 to get stats (see selftest bpf_verif_scale). Thanks!

ast accepted this revision.Sep 20 2021, 8:26 PM
This revision is now accepted and ready to land.Sep 20 2021, 8:26 PM
This revision was automatically updated to reflect the committed changes.