You can drill a lot of woods wood quite fast, but the denser the wood the slower the speed needs to be. Heat build up is your enemy, as is not withdrawing the bit and cleaning the flutes frequently. Acrylics need to be fairly slow otherwise there is a danger they will melt and fuse to the bit.
I drill everything on a pillar drill, and I tend to keep the same speed setting for everything, which is about 250rpm. Others will think that is quite slow, but I've never had a problem, and for the sake of a few seconds extra time, I'd rather be safe that sorry.