There is not much, if any, inherent different in wicking or breathablity between polyester and nylon. Those are attributes of type of fiber shaping or coating (wicking) and type of weave used (breathability). It is possible that a tightly woven nylon shirt verse a polyester of similar weave, may become slightly less breathable because the nylon fibers do absorb some significant amount of water vs polyester, and so they may swell slightly making the space between the fibers smaller?
As mentioned by others, a lot of button up nylon shirts are somewhat tightly woven and don't breathe as well as knit shirts or shirts less tightly woven. A lot of these also don't wick very well, though more companies are starting to come out with versions that do wick. Try to find the ones that do so permanently through shaped/altered fibers (Kuhl Wunderer), rather than a coating (REI Sahara-last i checked) because the coating will wash off eventually.
They are somewhat hard to find, but there are knit nylon garments that are fairly breathable but still offer decent Sun protection and some basic bug protection. The one that i have, Under Armour Iso-Chill ls shirt, also wicks very well and has decent odor prevention properties (part of that is the treatment, but as mentioned by others, nylon is less stinky and easier to wash out same than polyester). I rather like the shirt overall.
I haven't tested it much in very buggy conditions. I "suspect" that it won't work nearly as good as the typical somewhat tightly woven nylon shirts, but still offer some more casual protection (especially when moving). I also carry a wind shirt, so if bug protection is absolutely needed (stopped, overwhelming, etc) i will put that on.
I think polyester's only advantages over nylon, is that it will dry slightly quicker if everything else is equal, and it handles UV significantly better. If you set up a polyester tent/tarp and nylon tent/tarp in the Sun for awhile, the polyester one will fare much better over time. However, in normal clothes wear, probably not much of a noticeable difference unless you spend a lot of time in places like the desert or at high altitude near lower latitudes w/out shade.
But polyester is stinkier, heavier, and definitely less tough for same weight. I prefer nylon for various reasons. I would like to try shirts and pants with high percentage of nylon to low percentage of certain natural fibers (linen, hemp, or tencel) but they are not being made for the most part. I do have one shirt that has about 55% nylon to 45% tencel ratio, and i do like it a lot, but wish it was a bit more nylon and a bit less tencel for the muggy hot conditions that are common where i live/hike (ideal ratio of synthetic to tencel for a combo of comfort, wicking, and drying according to one Lenzing study is 70% synthetic to 30% tencel).