I can't just move the whole tempo track
You could. Place your cursor in the tempo track area on top, just above the individual tempo "events", then you can click and drag to select as many tempo "events" as you'd like. Once selected, you are able to move all the selected tempo changes as one.
I agree still that the this is not the ideal way. Perhaps the feature should be that after clicking on (highlighting) a tempo "event", CTRL+A should select all tempo events. (And that all tempo event should have the ability of being dragged past the boundaries of the project)