This matches the reference implementation:
https://github.com/Cyan4973/xxHash/blob/3064d42e7d74b0921bdd1818395d9cb37bb8976a/xxhash.c#L707
https://github.com/Cyan4973/xxHash/blob/3064d42e7d74b0921bdd1818395d9cb37bb8976a/xxhash.c#L147
and makes the behaviour independent of whether we are compiled with
-funsigned-char.