now that you can put a PointerIntPair in a SmallPtrSet, remove some