Sounds like you reduced the buffersize to a point where the PC is struggling to keep up. Getting the correct buffer is a bit back and forth at times. Bring the buffer back up, one increment at a time.
You can also download for windows a free utility that will analyze your pc to see if it will struggle with audio processing. Look for latencymon, and follow the directions on the product page.