Use SmallVector and DenseMap in even more places.