Many will not be surprised at what I am about to say.
Why oh why do folk use ca to stick in tubes, it is not suitable. It is not gap filling, and is too time sensitive, and it is not suitable to glue metal to any surface as it is not porous - the subsequent adhesion may appear to be suitable but it is a very weak solution. You are in danger of getting caught with the tube stuck out if you revolve the tube to get good coverage as the glue can grab the tube and the setting time is unreliable as it is dependent on the moisture content of the wood.
As for the other issues, did you use the correct drill bits? The correct ones for the Roman harvest are 31 and 33 64ths I think from memory, not any metric equivalent!
It's a good idea if you are using soft woods to do a dry run with a tube to make sure that the wood has not expanded back into the hole and closed it up a bit as this can sometimes happen.
Sorry to hear about your predicament, you might get away with trimming a bit off the tube, but no guarantees, it may also mean if it is a rollerball that there is not much bouncy on the rollerball if the spring has been compressed that extra bit. Unless the wood is worth the risk I'd be tempted to strip it off and start again.