This is a stupid distinction to make. There is no speech that doesn’t affect people materially.

“Free speech” is a morally neutral thing. Most leftists don’t go on about “free speech” because it’s not a value we hold. We value tolerance of people of different races, genders, sexualities, and so on. The issue is not speech in general – it’s the content of speech that matters. “Free speech” sidesteps the issue of what is actually being said.