A down sweater or a down inner jacket are intended to be inner layers beneath a top shell. In other words, the down sweater does not have a very tough shell of its own, and it is likely to get abraded or ripped in ordinary use unless it is protected by a top shell. You are likely to have a top shell anyway, and it is almost mandatory for rain and wind protection.
Your down garment has down baffles, and that makes it warmer than if it were sewn-through.
I was on Kili in June 2000, and I took a puffy down jacket in my duffle bag, and I fully expected to need it. However, it just never got that cold or windy, so I did not wear it. I was wearing several synthetic layers underneath my top shell. We had more wet weather than cold weather. Still, it could have turned cold on me and I would have had to grab the puffy down jacket. I would guess that you'll be OK if you augment that down garment with a thick synthetic layer or two.
For example, that thing has 900 fill power. That's nice, but it means that there really is not a lot of substance in the fill. It will be slightly fragile. If you keep it 100% dry, it will stay good. But if that thing starts accumulating any moisture (rain or sweat) then it is going to degrade. Synthetic layers above and/or below that would make it a bit more bulletproof.
–B.G.–