"I put the rolled Gossamer Gear sleep pad in. I put the Thermarest inside of that."
I'm guessing here…
You have a cylinder of sleep pads inside the pack, and you then stuff that volume with gear?
And it Sways side to side as you walk?
If that is an accurate description, try a different packing style.
Try to get the pack to resemble a rectangular box versus a cylinder.
At the extremes, a cylinder with a narrow contact area against the back will be hard to stabilize. A 12"+ wide contact area will do much better.
Try folding the pads, with one against the back and the other elsewhere. If they are both against your back you end up pushing the weight away, inducing more sway.
Does this pack have a waist belt with "tensioners" attached to pull the pack into your back?
How about compression straps between the top of the pack and the shoulder straps? (erroneously called load lifters, on a frameless pack)
As mentioned above, pictures, sketches, and more verbage will help nail this down.