Add the same optimization that we do loading from fixed alloca slots to