Make SmallSet<whatever*, N> faster by transparently implementing it with