Grow the map on entry so that we don't crash if joinIntervals never