Do you sand with the bushes on? When you use the skew as a scraper, does the tool scrape the bushes as well as the wood? Do you use a CA finish, then remove the CA from the bushes to free the blank? Any of these will abrade the bushes and reduce their diameter, making them unreliable as guides for the finish diameter. If they are no longer accurate throw them in the bin and buy new ones. Or, as Grump says, use callipers as your guide - then you can carry on using them to compress the blank on the mandrel.
To find out whether your bushes are reliable take a pair of digital callipers (£7.50 at Wilkinson), measure the bush diameter, then measure the diameter of the centreband at the edge where it contacts the blank. If there is a significant difference (more than about 0.2 of a mm) the bushes are toast. Note that this only works for slimlines - most other pen bushes have different diameters at each end. For them you have to measure each individual bush and it's corresponding kit component.
Not to take away from anything already posted - just something else to consider!