Remove live interval entries for an interval if we're eliminating its only VN.