I believe we were previously calculating a pointer info with the scalar base and an offset of 0. But that's not really where the gather is pointing. The offset is a function of the indices of the GEP we looked through.
Also get rid of the constant memory stuff for gather because I'm not sure that was right either.
I'm also not sure the memory size for the gather/scatter is correct either since I think it using the vector VT, but we're really loading/storing multiple disjoint scalars.