Even if you inflate a pad with outside air (i.e., NOT your breath) in the afternoon…by the middle of the night, when the air has cooled, the air volume inside the pad will have decreased. Some people might confuse this with an actual leak.
Additionally. if you inflate the pad with afternoon air (when the air is warmer and can hold more water vapor) then, in the middle of the night, once the air inside the pad has cooled due to the temperature having dropped, water vapor will have condensed into droplets inside the pad. My wife and I have seen the water droplets (some pads use a very translucent fabric) on many a trip.
So, when we get home, we inflate the pads (again, NOT using breath (we use the Exped shnozzle sack), We thenl et them sit for a while at room temp (to allow the droplets trapped inside the pad to evaporate (i.e., turn back into water vapor). Then, we deflate the pad. We do about three cycles of that.
So, even without using your breath at all you can have moisture collect in a pad. Whether that can lead to mold or not I do not know. But I do know that the baffles inside a pad can suffer (i.e. fail) from hydrolysis, caused by any water, moldy or not.
But there are measure you can take to try to get the moisture out.
P.S. We never breathe a breath into our schnozzle sacks, instead, we fill them with air by a quick flick of the wrists.