With 3 kids now…well, what I did when I had one child is much different than what I do now. When my oldest (who is 16 now) was 4 1/2, he was hiking 10+ mile days and going nearly every where I went. But with having 2 young ones now? Not so much. My middle son is 4 1/2 now and he hasn't hiked more than 4 miles in a hike. I just don't push him like I did the oldest. For one, I don't have that need to keep walking like I did 12 years ago. But as well, his mindset is different than his brother.
Each kid is different.
Kid 1 was tall, thin and had endless energy – but I had to watch him 24/7 when we hiked – he was on a tether system to me till he was 7. He didn't talk until he was nearly 6, so had he gotten separated from me, it would not have been good. I knew his Autism could be exploited and he made a great hiker. I often used a jogger stroller for him when we had to walk FS roads to trailheads, so he could go farther with me. But it was just him and I, and easy to plan around.
Kid 2 loves to build and do crafts. He loves plinking rocks in creeks and lakes. He has endless questions for me. He loves looking at rocks, birds, animals, flowers – and looking at guides about them. He likes hiking, but not the way his older brother did. He brings a joy to it though.
My youngest son is 2 1/2. He is super strong and big (he is nearly as big as his brother) but I have so many fears with him (he has severe food allergies). So while he would be an amazing hiker….I just don't take him far from civilization. He also has my husbands pale white-boy skin and burns awful at altitude. Sigh.
So for now I have found I feel most comfortable taking the boys to stay in cabins, on the edge of the wilderness. So my youngest can sleep safely at night. And not have to worry. I can clean them up, feed them well – and not have to spend hours setting up and tearing down gear, while trying to keep an eye on the youngest. I enjoy it, I sleep well at night, and we get out and have fun.
So do what you are comfortable with. And what the kids react well to. But don't feel like you are not doing enough, as long as everyone has fun, that is all that matters.