Hats take a beating, scrunched up in a pocket or pack and rarely threaded with the care of an expensive down jacket or bag. I have this pet therory about thin down garments loosing a higher proportion of loft: If a thick puffy loses 1/4" loft, you would hardly notice, but if you only have 1/2" to start with, that's a 50% drop. The assumption is that those outer tufts of down are taking the beating and with a thin garment, it's ALL outer layers.
Fleece is cheap and light, it wicks and dries well and it feels good. Wool is more durable. There are all kinds of wool/fleece hybrid beanies.
It really comes to how much loft you really need. I find that I usually need a little insulation on my head and will just be a sweaty mess if I overdo it. I have a light patagonia R.5 beanie and an Ibex wool beanie that are my first choices. The hood on my wind or rain shell is taking the brunt of the weather and the beanie under that just needs to keep a warm layer of air next to my skin
A light balaclava may be a better bet in cold weather as it is my face, chin, neck and ears that are feeling the cold. Likewise, a simple fleece headband will keep my ears warm without overheating my whole head.
A down beanie might be better in camp, with the same pros and cons as down vs synthetic jackets and sleeping bags. At that level of cold I would be wearing an insulated jacket with a hood anyway, with the option of a fleece or wool beanie under.
I use a bag with a hood, so it's the same scenario for sleep as with a hooded jacket. I would want a warm hat with a hoodless quilt for cold weather, but if it is that cold, a balaclava would be my choice— I have a thick head of hair and my face, chin and neck feel the cold more.
My real go to cold weather hat is a Peruvian style beanie with the long ear flaps in a Windstopper fleece. OR makes good ones. I'm using a North Face version now because I found one that fits my XL noggin better. If the Windstopper fleece and a hood won't cut it, I need a cabin with a fire!
Carhartt has a few models worth looking at. They have a few that are cut like the Peruvian caps, sans a chin strap. They are long in back and give excellent coverage down to your neck. They also make a helmet liner face shield that most of us would call a balaclava, but the chin portion is easy to pull down out of the way. The prices tend to be better than hiking clothing.