As in the tiniest amount? The problem will be in proportion to the degree of tininess.
To what extent can you guarantee the accuracy of your jig? Even if you can guarantee that both jig and trimmer are 100% vertical, if the jig is out laterally by even a quarter of a millimeter, because your tube and blank are held rigidly in your jig, that error is transmitted down the entire length of the tube, and you will have a tube that is oval not round.
Hold the blank by hand, and even if you mess up at the point of entry, the error will only be at the point of entry - the shaft will straighten things up because your hand is not a rigid platform. Therefore the length of the tube will still be round.
The end of the tube in your photo is getting on for square, suggesting that the relationship between your your jig and trimmer is inaccurate both vertically AND laterally.
Abandon the jig - you don't need it. I believe it is the root of your problem.
Please don't think I'm criticising your ability to build an accurate jig. It could just as easily be the accuracy of the drill that is at fault. It's the relationship between the two that is the problem, but either way, it is overcome by losing the jig.
