Over any small interval, everything is linear. And while small differences in weight aren’t detectable from all the other variables in one measurement to another, they’re still contributing.
I’m pretty sensitive to pack weight and always have been. In the 1980’s when other people on the trip would have 45- and 60-pound packs, I was thinking, “if my pack weighed that much, I wouldn’t go backpacking.”
I’d feel as tired doing 15 miles per day carrying 30 pounds as doing 25 miles per day carrying 20 pounds. So 10 less pounds = 10 more miles. Hence, losing one pound lets me hike another mile that day (or lets me feel 1 mile less tired over the same distance).
But back to racing: is there also a psychological benefit? – I certainly notice when I put on my lightest trail shoes, I feel lighter on my feet. Baseball players swing a weighted bat before going to the plate. Swimming shave all their body hair before a big meet and *feel* slicker in addition to actually having less hydrodynamic drag.
And there’s the physiological effect on the other racers if they don’t have the best gear. The Nimbus 2000 / 2001 were the fastest racing brooms through 1992 until the Firebolt was developed by Randolph Spudmore in 1993 after which it came to dominate in International Quidditch events.