[USBPORT] Fix endless loop, occasionally happened on real hardware