It may indeed change its size when connecting to the audio driver which causes the DSP to adapts its audio I/O, but it should not be THAT slow. Are you running on Mac OS or Windows?