I was in the same situation a few years ago. I had a ULA Catalyst that I bought for week long hikes with a bear can or carrying 6 liters of water. Then I started doing 2 and 3 day trips and I found that it was barely half full. I got a smaller pack that was also quite a bit lighter. It was a good choice.
You'll find that smaller packs are also cheaper, so it's easier to justify even if you only use it 5 times a year.
Or stalk the used gear forum and pounce when you see something you want. Lots of people buy packs but find out they aren't what they are looking for, so they sell them almost new.