Fix an issue where assignments that caused a SmallPtrSet to become non-small