I don't ride with ear protection, wind noise has never really bothered me (maybe helmet better than some??) but even if I wanted to, my ears are weird somehow, nothing stays in them. Earplugs and earbuds alike, they all fall out seconds after putting them in.
Hearing loss is very gradual. The first sign is tinnitus (ringing in your ears that won't go away) and difficulty hearing another person in a noisy environment. I would highly recommend some type of hearing protection. Foam plugs will not fall out if they are inserted properly. Sometimes it's a little tricky to get them in.