Bigger mistake would be thinking it's too late to start. Modern bikes are so good they make it much easier and more survivable then you might imagine. Trials competitions are events that you will encounter only encouragement and support in your efforts, unlike go fast competition where the goal is to basically slaughter the other guy with stones and keep them behind you, in trials competition the other riders really do want to see you put forth your best possible performance even if it means beating them in the competition. Not unusual to see close competitors catching, minding and offering advice to each other on how to best ride a section, possibly even throw their machine on the ground and run to your assistance :/ you don't see any of that in any other form of motorcycle competition.
Try it you'll like it.