Warm neck makes me feel warmer than it has a right to.
When it is warmer (I know 40* has been used only as an example), I probably wouldn't have my down jacket. I would have a vest and a hooded windbreaker. The hood of that goes over a wool cap. I like the cap.
If it is colder, I would have a hooded jacket. Still like the wool cap unless I need the hood.
I started life hood adverse. Now, I am pro hood.