Switch some std::sets to SmallPtrSet. This speeds up