Switch the pointer-difference simplification logic to only work with