Welcome Simon. You don’t say if you are drilling on the lathe or using a pillar drill. If you drill on the lathe. If drilling on the lathe check your headstock / tailstock alignment by doing the kiss test. This involves putting your headstock drive in the spindle, bringing the tailstock up close to the drive & locking it to the bed. Slowly wind the tailstock quill out until the point of tour tailstock drive is nearly touching the point of your headstock drive. Check both vertically & horizontally to see if the points are in line. If they aren’t then adjust the headstock to line them up. If your lathe has a swivel head then unclamped the head line the points up then reclamation the head. Recheck the alignment to see if the head has moved when reclamping.
If you have a fixes head & the points don’t line up then loosen the bolts, line up the points then retighten the bolts, again recheck alignment. Also check your tailstock when unclamped to see how much side movement you have as thes can also have an effect. It tends to make the holes oversize even if using the right size drill. I know because I had this problem with one of my lathes which is now cured. The tubes were very slack in the holes & caused problems. It’s long winded but I hope it helps.
