Fix PR2506 by being a bit more careful about reverse fact propagation when