Depending on how far along you are in the process there are a couple options:
1. In the Track Inspector (right below the track name), set Tempo to "Don't Follow"
2. Remove the tempo change, bounce the audio tracks starting at the beginning of the song, then re-apply the tempo change. Since the audio is now just one clip, it will not be impacted by the tempo change