It doesn't mater what the meterial is. What is important is how much air is trapped in the fabric. It's the trapped air in the fabic that keeps you warm. The warmest jeckets are those that trap the most air and yet block the wind.
Wind has an easy time passing through a sweater so the slightest breaze will raplace all the warm air with cold air. Same goes for Fleece but fleece is generally denser which would reduce the effect of the wind. The denser the material the better. If you put on a wind shirt the sweater might be better then the fleece.
Years ago Polartech came out with a new fleece that was stated to be as light as regular 300 weight fleece but lighter. I purchased it but quickly found the claim was only right when you were completely out of the wind. The slightest breaze would quickly give you a chill and in the end I found it was no bettern than regular 100 weight fleece. Polartech stopped selling it and then came out with wind block fleece. The windblock fleece was no lighter then the old fleece but it did perform better with the wind block layer.