Mike B,
Downproof fabric is only an issue with low volume down. When you get to 800+ cubic inch down, there are few quills to poke through the fabric. ‘Down’ per se has no quills. When using non-down-proof fabric, just pull out any random quills or feathers as you fill the chambers. This won’t work with low-volume ‘down’ which may have so many quills you will be left with little.
You might consider using a soft microfiber polyester instead of fleece to save significant weight. Also, try using a fleece liner in an existing bag for a few nights to see if you really want to do this. You may change your mind. Fleece can be clammy.