First off, the
rotor is the rotating 'core' of the motor:
A Typical Single-Phase Induction Motor
As you can see, it is a rather substantial mass, usually consisting of iron (though other metals can also be used), that pretty well dwarfs the mass of your pulley. In fact, the rotor typically represents the bulk of the motor's weight (not that the stator or chassis is especially lightweight) -- it's considerable mass is the reason your power tool takes so long to spin down after switching it off! The outer stator windings induce a rotating magnetic field through the rotor, which both simultaneously pulls and pushes the rotor in the direction of rotation at a speed that is (more or less) phase-locked to the frequency of the incoming AC power. Which is to say, the rotational speed of an Induction motor is determined by the frequency of the incoming power and not the voltage (as would be the case with a universal motor). In actuality, it will always spin a little slower than ideal due to a phenomena known as "rotor slip".
I'm a bit concerned that you were able to install and remove the bearings without heat as that means the fit is not especially tight, which offers some possibilities that the shaft and the cup the bearing rests in could possibly allow for some vibration over time (it will all depend upon what sort of forces are in play). Typically you would need to heat the bearing -- or cool the shaft -- in order to install the bearing on the motor's shaft. You would then use a mallet (not a hammer) or a hydraulic press to press the bearing into it's 'cup' in the motor's chassis, taking care to do so only by the outside race and ensuring no force whatsoever is applied to the motor shaft and/or inside bearing race.
However, I suspect that what you are interpreting as arbor/bearing play is really
*pulley* play. If a pulley does not fit *tightly* over a motor's shaft (e.g. the pulley's bore is a bit oversized or the shaft a bit undersized) then the pulley can literally rock with every revolution, with the setscrew usually serving as the pivot point about which the pulley will rock.
What is likely happening is that, after your servicing, your pulley setscrew has been very well tightened. However, over time, due to the free-play between the two, the setscrew either loosens a tiny bit or the end of the setscrew begins to round over, making it easier to pivot about.
What you will hear is a knock-knock knock-knock (ad infinitum) as the motor spins. Not only is there often an audible knock-knock sound, but that knocking also causes vibration throughout the motor and belt assembly, which then transfers into the rest of the saw. It will not be especially noticeable (unless really bad) when the belt is removed, but will be quite noticeable with the belt installed (due to the lateral load the belt places on the assembly).
If you did not need a pulley/gear puller to remove the pulley... and a mallet (and some persuasion) to install it, then there is a very good chance you have found the cause of your problem. It could also, eventually, lead to bearing failure, but good bearings would take quite some time to be detectably affected by this (usually).
You might be able to shim the pulley to address the issue. Keeping it tight to the motor shaft is more important that whether it is perfectly centered. However, the best solution is a snug-fitting replacement pulley.
I would hold off on replacing the bearing until you have a new snug-fitting pulley so that you can see if that does not fix your problem. You may well discover that your existing bearings are just fine.
HTH