Perhaps you took old Nipissing Rd south from Magnetewan - that is a great hard packs dirt/gravel road whichI wold always suggest if you are up for that sort of thing. Tighter and more twisty is 518 for sure. Bobcaygeon will be fine for you then. In the future, you should also look at Harburn Rd, Kennisis Lake Rd, Lower Faraday Rd as some others that are further east but fit this bill. Even Kushog Lake Rd is fun later in the season when the sand is gone from the road.but then headed back down south on the 124 and connected to 518 via some dirt roads (managed to get in both the 520 and 518)...
Sorry I have no suggestions for non camping places to stay. Air BnB?