Just a couple of things that help. First and foremost the drill bit must be sharpened correctly, both lips exact. If they are not one side of the bit takes off material at a different rate than the other and the bit will go off center. You want the drill to enter where you want it. Using a center drill has the wrong geometry. It is designed for a 60 degree center and your bit is going to be 118 or 135 and is not going to match. Either use a 118 spotting drill with a 118 drill bit, etc. Another trick is to use a a ball endmill to center the hole for the bit, works best for me. This may be obvious but the material must be clamped to the table so it cannot move, you cannot just hold it even in a vise.
As for 12" - the drill press must have a heck of a throw. Only way I can do deep long accurate holes is in a lathe.
Bob