As said it really boils down to personal preference. I drill all small stuff on the lathe, and only use the pillar drill to drill mortice holes on pieces like stool or chair posts. I can't agree that one method is better than the other for speed, it is all down to what you are happy, and experienced with.
Speed only comes with practice, whatever the method.