Don't leak memory when destroying device info set and driver lists