I rarely hike in anything more than a base layer. It just gets too hot. The closest I'll come is in that wonderful range between 30*F to 45/50*F with moderate to heavy rain. In the upper range I'll still sometimes just wear the base layer, but sometimes I'll wear a shell with the sleeves pushed up all the way. If it's not raining, I'm usually comfortable in just the baselayer down to around 15*F when I'm on the move. If it's breezy, a windbreaker… but after a couple miles I start overheating. In 0*F, give or take 10*F, I'll generally start out in a thin wool layer (140-190-ish weight) and a windbreaker, and if I'm feeling "wussy" a thin fleece vest, topped off with a lightweight hat and gloves. The vest is gone in a mile or so, the hat turned up or taken off shortly after. Whether or not the gloves stay depends on the wind.
So, in my experience, you're fine to around 0*F with those three combined… provided you're on the move and making time. I don't think just adding the down layer is optimal for camp, though. Depends on the person, of course. But I get too hot moving around camp, and like the flexibility that a midweight layer affords me. Think a 300-ish weight wool, or if you prefer synthetic a powerstretch piece. If you're trying to minimize as much as possible, you could consider that midlayer combined with a big down vest. Could keep you decently warm, relatively sedentary, into the teens or twenties…?
The trick, of course, is your "whilst walking" aspect of the question. While actually hiking, I'd imagine you'd overheat wearing the base/wind/rain in all but near-zero F conditions. Once you stop, a big poofy down piece would probably be warm enough, but you might find that adding a midlayer would add a lot of practicality and a bit of a safety margin (it's easier to control moisture exposure to a sleeping bag than a jacket, IMHO), which is why you might consider the down vest/midlayer option. (I mention "poofy" and "big" down because trying to get away with a thin down layer won't work as well for most in those lower temps, particularly if that's your only insulation.)