Fix up some asserts that are within an if statement. This removes the need