My standard backpacking clothing arrangement, from the waist up is this:
1) Synthetic T-Shirt
2) Puffy Hooded Jacket (specifically the BPL Cocoon Hoody)
3) Propore (e. g. O2, Drop Stoppers) Rain Jacket
4) Fleece Hat
5) Fleece Gloves
6) Rain Mitts
That's it. The Rain Jacket doubles as a bug shirt. Most of the time, during the day, I wear only the T-Shirt. When it gets cold (say, on top of a mountain while eating lunch) I put on the puffy. Overall, this extends my range quite a bit. I don't have to carry as big a sleeping bag because I can sleep with my puffy jacket. A fleece sweater (which I typically use for day hiking) breathes better, but is not as warm for the weight.
The "cost" of this setup is that I'm not as granular as I would be with fleece. In other words, sometimes it is too cold for just the T-Shirt, but too warm for the puffy jacket. This is a small price to pay, and doesn't happen that often. The fleece hat helps in that regard. Like any fleece, it can comfortably handle a wide range of temperatures. So I will sometimes hike with a T-Shirt, fleece gloves and fleece hat.
The one thing that I might add to this is a nice, fairly breathable wind shirt. Using a wind shirt can help cover that middle ground well, and would provide for a more breathable bug layer. Pro Mountain Sports (in Seattle) recommends these layers, on their website: http://promountainsports.com/index.php//clothing_systems
(notice, no fleece)