The advantages (besides the obvious displacement/power bump) that the big ADV bikes have over the CB500X are longer suspension travel, higher clearance and a 19" wheel in the front, neither of which you'll need if you're just doing hard-packed gravel roads. Where you'll actually need these things is if you're trying to ride over rocks bigger than the size of your fist, and the higher clearance is going to help if you're trying to get over logs and boulders.
The reason why the ADV bike blew past your cruiser is mainly because of the seating position. With your feet out in front of you, you're pretty much stuck in one seating position, so when the tires move around on the loose surface, you're basically a passenger.
If you're sitting upright with your feet beneath you, you have the option of moving around, shifting your weight forward/backwards/sideways on the bike to compensate for the loss in traction. Additionally, if the surface gets too gnarly, you can also stand up on the pegs when the bike moves beneath you. That way, the pivot point becomes your feet, not your butt, making you feel more stable, but also giving you more control. Again, if it's just hard packed gravel, standing on the pegs is not needed.
I've ridden in so many developing countries where you see the western ADV guy riding their 1200cc bike on hard packed gravel, standing up on the pegs. Then a local guy zooms past him on a 125cc Pulsar with his wife and two kids sitting behind him, and they've all got their butts planted on the seat. They always give the ADV guy a puzzled look like, "What's the heck is wrong with his seat?"
If those tiny 125cc street bikes can do it, the CB500X will do just fine.