Steven,
I have to agree that it was probably your basic tenet to stay with/in the shelter that brought you undone here. Instead of using your bag as a bivvy, you used it as a sleeping bag cover.
A really good book to cover the whole idea of using a bivvy is Ronald Turnball's Book of the Bivvy. $14.33 including postage from http://www.bookdepository.co.uk
http://www.bookdepository.co.uk/book/9781852845612/The-Book-of-the-Bivvy
He also would have suggested dropping into a more sheltered spot in the woods nearby the shelter.
It's a great book and I highly recommend it if you're interested in getting into bivvying
As far as the situation you were in, and the wind blowing the warm air out of your bag, could you have shoved a jacket (shell?) in as a collar to reduce this. I also find it worth manufacturing a microclimate beside my head to minimise airflow directly across my face. I use my pack for this, but anything bulky enough will work. You can also turn the bivvy upside down with a small breathing hole facing away from the wind.
Just remember, in a cheap (ie non-eVent) bivvy, you can be warm and wet or cold and dry. Warm and wet is fine if you're only out for one or two nights.
Just remind your wife that the fact that you called her up for a lift shows how responsible you are. You could have toughed it out, and tried to walk out alone the next day in a dangerously depleted condition, but instead you chose the safe and sensible option. Pick a really crappy night sometime soon, and try the bivvy again in the back yard or with a tent along as back-up. She'll soon start to trust your judgement.
The comments at work are easy. Just invite any of them along next time. It's fantastic that you are getting out there in less than ideal conditions, since it gives you an insight into how others may be doing it, and what the possible consequences of not making smart choices can be.