Colin, if you are drilling plain timber blanks then it is not so important that the drilled hole is not that straight. If however you are drilling segmented blanks then straightness becomes crucial. I only drill on the lathe....I have a pretty good pillar drill but try as I might I was never 100% happy with the straightness.
I use these pen jaws on my lathe chuck....important that the blank is square.
[URL unfurl="true"]https://www.axminstertools.com/axminster-woodturning-chucks-pen-jaws-103670?queryID=a757969bef21731b0d60b1bd1484f6af[/URL]