Due to the PNW often living up to its wet reputation, I was slow to convert to down. Seeing that my bargain 550 fill down sweater is 14oz, Costco down vest is in neighborhood of 6-8oz, and my fleece vest is >14oz (sorry for vague numbers/going from memory), I found down to be worth the risk.
My down sweater or vest and sleeping bag stay in a dry bag when I'm hiking. I only bust out the sweater or vest to stay warm during breaks as I rarely need more than a shell and base layer for anything above 20* as long as I'm moving. I wouldn't hike while wearing down due to perspiration issues as you mentioned. If I'm looking at sub-20* temperatures where it's likely that I'll hike with extra insulation on, I'd bring synthetic insulation in lieu of down. I'm a fan of the military style polypro for this purpose but I have a fleece vest too.

