In looking at the pictures, the first thing I noticed is that the hole started out off center. That's a consequence of starting with the large bit.
My approach to this would involve two things. First, I would start the hole using a center bit. Center bits are intended primarily for metal work, but they are really useful when doing precision work in wood. Basically, they are very short tapered bits - a thick body to resist flexing, with a very short small diameter extension, and with a tapered section between the body and the tip that can also be used as a countersink. When I'm doing precision drilling, I use only the narrow tip to start the hole.
You can accomplish the same effect by using a small bit, but the problem is that small bits can flex. You can minimize flex by mounting the bit in the drill chuck so that only a short portion of the bit extends from the nose of the check. If you can't find a center bit, an alternative is to grind 3/4" off of the end of a standard bit so that it can be mounted in a chuck with only a very short length of flute exposed.
After starting the hole with a center bit, the next step is to chase that hole with a larger bit. A large 118 deg spiral bit can still wander around a small starter hole, so using a brad-point bit is preferable; the brad will lock in the small starter hole to keep the bit centered. Alternatively, you can drill the hole in several steps, increasing the diameter slightly with each drilling.
Finally, while I appreciate my drill press, I find that this kind of precision drilling is more easily done on the lathe. I routinely make 'throwaway' pens by redressing cheap Bic biros in a wooden sleeve. That requires that i drill a 9/16" hole 5" deep in blanks that can be as small as 3/8" in cross section. That would be impossible on a drill press, but I've done it on my lathe.
