This matches GNU assembler behaviour.
Alex, I was looking at why the fuzzer did not detect this issue (the constrained fuzzer was not generating the modifier, but the unconstrained fuzzer is and should have generated that test case).
In the process I ran into this other example that the llvm assembler fails to assemble, but GAS is able to assemble.
echo "li x0,%lo(4)" |llvm-mc -triple riscv32
<stdin>:1:12: error: immediate must be an integer in the range [-2147483648, 4294967295]
With your patch, this example works fine. You might want to add a test case for it.