we don't need a smallptrset to detect duplicates, the values are