Do you ever have to wear both T shirts and both pair of underwear at the same time? If not, you need only one of each. The only changes of clothing I take are socks. Everyone else on the trail smells just as bad! I leave a change of clothes and some moist towlettes in the car so I won't reek too badly when I get back to civilization. Should the worst come to the worst, it's no big deal to go without underwear for a couple of hours while the item that had to be rinsed out is drying (with lightweight synthetics, it won't take long).
I gave up liner socks when I switched from boots to trail runners–I haven't needed them. Trail runners have more "give" so there's far less chance of blisters. Your mileage may vary, but if you're still wearing boots, you might find trail runners with a single pair of medium or even lightweight socks a lot more comfortable and definitely lighter on your feet and legs.
A puffy top (such as the Montbell UL Thermawrap, if you can find it on sale) is definitely lighter than fleece.
Re the base layer, a lightweight polyester or merino wool fabric (the latter is usually more expensive), similar in weight to Patagonia's Capilene 2, will do the job. Since I prefer long sleeves (I'm allergic to most sunscreen and detest bug repellent), I use my baselayer top as a hiking shirt. However, this means the sleeping bag gets dirtier. I may wear my 2.5 oz. windshirt over it at night.
Just for comparison, my total clothing carried, including rain gear and a dry bag for my insulating clothing, suitable for late fall/early spring trips or for summer trips in the high northern Rockies (below freezing temps and occasional snow), is 41 oz. That's more clothing than you need for summer in the Sierra. Of course I'm short, but I'm unfortunately also pretty stout, so unless you're pretty big I doubt that similar clothing in your size would weigh more than 5-6 ounces more. And I'm definitely not an ultralight backpacker; my base weight for the conditions outlined above is 13.5 lbs.
What do you wear when it rains?
I strongly recommend setting up a spreadsheet with your complete gear list, including weights for each item. If you don't have a digital postage scale, that should be your next purchase. That's the best way to analyze your gear list and also gives you a checklist for each trip. Just comparing your list to other gear lists on this site will give you lots of ideas for paring weight. You can post the spreadsheet on your profile and let us whack away at it, if you don't mind some people hollering at you about using toilet paper. :-)