I have also noticed this issue and really can't stand it. I noticed it very quickly because I like to trim loops down to one shots and play them like instruments sometimes.
It's important for this to work in a way that allows polyphony so we can play chords more naturally. Even a trained pianist won't hit every key of a chord at exactly the same moment, and shouldn't because it won't sound as natural or "human".
Also, if this follow song tempo setting worked the way it should, it'd make the sample duration more even as we play higher and lower pitches. So instead of a lower pitched note sounding for a longer time than a higher pitched note, they could last about the same amount of time.
If we want to prioritize one note at a time, we can set the voice polyphony to 1... simple...
I'm hoping the developers will address this soon, but I've seen people mentioning this issue since 2018... it may not come any time soon. No one seems to bump this issue like they're bumping up others.
Also, junhoshin, add some more tags to this post that are relevant. I noticed you haven't tagged sample one xt, but you tagged impact xt.