pm80xx: Remove unnecessary phy disconnect while link error