I can't really speak to whether using your sleeping bag as primary evening insulation is common technique or not. I do know that it's common theory, just not sure how many people actually suck it up and do it! I've mostly thought about it as an option if it needs to be done, vs my only plan. I'd say that very few hikers go out with in shoulder season without insulation enough to keep them warm after about 5pm while stationary. It could easily be too cold for your clothing setup after about that time.
Even if I hike a long day, there is usually an hour or so where I'm in camp and I don't yet want to be in my sleeping bag. And if I'm not dead tired from the hiking, I like to be out wandering around my camp checking things out until a little after dark.
If you are willing to sacrifice a LOT of comfort for the goal of a light pack, you may be able to get by with your current setup. All gear choices are compromises.
I'm generally not an advocate of buying a lot of stuff, but you could consider how a warm puffy layer complements an ultralight setup. For this type of trip, and other hiking in CA, I'd consider the Montbell Alpine Light as an option. As for your current down jacket, that's probably not something I'd want to carry outside of winter. Could you scrounge another thick fleece from somewhere? A down vest?
I'm heading out next week and I'll be bringing essentially what you are, plus a Montbell UL Down Inner jacket. I think I'll be cold for the minimal out of bag camp time that I'll have at ~12,000ft.
This could all be bad advice, it could be warmish at night (mid 30s) or it could be low teens.