Maybe I'm confused on your terminology, but it's the buffer size you want to reduce in order to reduce latency. I don't know, maybe block size and buffer are the same thing, but I do know that buffer and sample rate are not the same thing, so maybe you're changing the wrong parameter.
I'm using audiobox and this is what it looks like with that.
When recording, reduce buffer size, when mixing, increase buffer size.