What Bill said!
If your lathe is properly lined up, the blank is correctly held central in the chuck, ( what chuck & jaws are you using? ) then there can only be a couple of ithe things wrong.
e.g. Your drill bits are blunt or are poor quality and flex while you're drilling - sometimes can happen if you'r pushing too hard and not withdrawing to clear swarf.
I use a record SC3 chuck with pen jaws fitted for drilling as I find this better for holding blanks that aren't square and can drill wild grain hardwood that's bent like a banana and still get a decent resut and to date have never has an acrylic, coria or horn blank that's gone wrong when drilling on the lathe.
I do use a centre drill on acrilic but only because I have a spare chuck with one permanently fitted otherwise I wouldn't bother, neither do I use lubricant, just drill in one go slowly enough not to overheat and partially withdraw every 1" or so. Don't get blowouts either, you can hear when the drill is close to the other end or can wrap tape around the drill to mark and can be careful.
I'd do what Bill said first, check your lathe set up and if that's ok do the same test with a centre point fitted in your chuk, spin it by hand to make sure it's true, then check your drill bits.