Improve the bail-out predicate to really only kick in when phi