Once the shell becomes a cold, soaked, wet layer, it appears to become no longer breathable, in my experience. So even if you are just walking around, and not working up a sweat, condensation of your body vapor against that cold outer layer might become an issue. I don’t know if that is a likelihood in the amount of time you were walking around. What would make for a fascinating ( for me anyway) is to see what would happen if you had been wearing a vapor barrier shirt close to your skin. I wonder if that down would have been so soaked in that case? I generally do not have any condensation against my VB layers because they are kept warm. Condensation loves to happen on contact with icy cold outer layers. Moisture will only become a problem with my VB if I overheat and sweat, but even then it is still not condensation, it is sweat from too much heat. Plus, the wet against my skin is unpleasant, but warm, because it can not reach my insulation.
I have tested this by making no attempts to either vent or remove an insulation layer or the VB itself, and just pushed myself to sweat and soak my near skin layer on purpose. Insulation remained totally dry. I have tested this sense I have worked up so many sweats over the years no matter how supposedly breathable my gear was, getting my insulation wet. But I must say even that wet insulation ( usually from sweat ) has worked out for me, because it has almost always been fleece or some other synthetic, which precedes to dry even under my GTX or similar shell after I stop sweating. For example, going down a 3000 ft ski slope as fast as I am capable, maybe with the occasional fall buried in deep powder, working up a sweat. Only to get on a ski lift and sit fully exposed to below zero wind chills for however long it took to get back to the top, with or without blowing snow. But wearing my synthetic long johns and fleece layers, I never had a problem. At the end of the day, I would feel pretty dry. And usually stayed warm enough.
But whatever the actual cause, I bet most of us have either had it happen to us or our friends more than once over the years: whether from sweat, condensation, or exterior moisture making it’s way in: wet despite high dollar rain gear, and also cold depending on the insulation that was under that rain gear.