I have done a number of firsts since turning 70 (about 2 years): moved out of the country to Peru, joined an HMO there, later bought a house in The Villages, FL after looking at it for 1 hour, Karaoke, piano lessons, guitar lessons, joined a writers club, went on a river cruise in Europe, and wrote non-technical articles.
I will not do anything dangerous, like jump out of perfectly good planes, but I am not interested in participation sports, either.
As you said, most limitations are self-restrictive.
You certainly shouldn’t let anybody else’s comments limit you.