[LCG] Rather than doing a linear time SmallSetVector removal of each