Only offset the rect if it's not empty, otherwise we might return negative