I lucked into an estate sale where I purchased a router table and over 150 high quality bits. I then sold the router table and router for what I paid for the entire grouping.
A lot depends on what you typically build.
I most often use roundover( multiple sizes), carbide upspiral and downspiral bits, rabbeting (Maloof) flush trim for following patterns various lengths. I also use chamfer bits at both 30 and 45 degrees often. I do not build a lot of doors but a rail and stile bit set is also handy. In making Maloof chairs, I use the specialty Maloof Bits from Woodline and I also use a cove bit for Maloof tables but these are specialty applications. An inlay kit is also useful.
I rarely use edge profile style bits like Ogee, etc. I purchased all of the $5 bits at Woodcraft a few years ago and most still have the wax coating.
I prefer 1/2 inch collet bits if possible.
My Favorite brands in order of preference.
CMT (awesome roundover bits)
Whiteside (great spiral flush trim bits and inlay kit)
Bosch (Good chamfer bits)
Freud (excellent profile bits)
Onsrud (High quality carbide upspiral and downspiral bits)
Woodline (Only supplier offering Maloof bits, mid priced)
Klingspor (mid priced, but excellent)
Woodcraft (mid priced,, I have only used a couple of the ones I own but they seem to work fine)
I have never tried Amana or Infinity but the reviews are great.
Rockler, MLCS and Eagle America are less expensive but do not seem to cut as cleanly as the others above.
I have had bad experience with unbranded or generic lower priced router bits as they do not retain the edge leaving a lot of tear out and tend to burn the wood