Correct. I'm a case study on this. I learned to fly at two towered airports, D and C airspace. Once I was cut loose with my PPL, I tended to seek out less busy destination airports to avoid fees, crowded patterns and find cheaper fuel. Along with that came less need to talk to controllers. With the advent of GPS nav and computer based weather info and NOTAMs, the need to talk to anyone official decreased still more. I literally cannot remember the last time I used flight following or filed a flight plan. We have 406 MHz ELT's, APRS, ADS-B, FlightAware if anyone needs to find us. Living with my plane on a private airstrip well outside controlled airspace adds to the atrophy of comm skills. When I take friends up for a hop, one of their first questions is whether I have to talk to anyone "in charge" before or during the flight. Nope! 'Cuz 'Murica, baby. Freedom!
The end result is a guy who spends a lot of his BFR's asking the instructor to review radio protocol for getting into and out of controlled airspace. Kind of pitiful for a guy who has HF radio gear packed into his truck because he's loved talking to random strangers on the ionosphere for 4 decades. It's not mic fright, I just don't much like talking to "people with badges," as it were. I don't like ramp checks for the same reason.
Don't become me. Get comfortable in the system and stay that way. If you base your plane at a towered airport, this will take care of itself. If you do the rugged inividualist/hillbilly thing, it's a temptation to skill loss and unreasonable shyness.