I suspect Eddie, it has something to do with the relationship between your jig and the trimmer in your drill, and whichever one is at fault you can easily overcome it with one simple adjustment to your method.
Abandon the jig!!
Keep the drill revs down to about 2 to 3 hundred, and just hold the blank by hand. Instead of lowering the chuck and trimmer into the tube, push the tube by hand onto the trimmer. The trimmer shaft will keep it square, so you don't need a jig to do it for you.
I suggest your problem is nothing to do with lathes, bushes, mandrels, spindle alignment and so on - I suggest you are mounting on your mandrel, a tube that is already mis-shapen, because you have distorted it in the process of trimming. The tube on your photo is clearly not round, and therefore your chances of turning a round pen barrel are extremely slim to say the least.
Give it a go without the jig and see if it makes a difference. You could just as easily put your trimmer in a jacobs chuck on the lathe and push the tube over the trimmer that way, which is what many, many people do. Its the same principle, but in both cases it's the trimmer shaft which keeps the whole thing true not an over reliance on engineering micro-details by attempting to in marry a jig to a drill in 3 planes.
