You're right that there needs to be a distinction that goes with the word if it's to be of much use. And there are no doubt some sense of the word "special" (meaning something like rare or unique) that don't allow for all of us to be special. But there's still logical room for a sense in which we're all special. Here are two ways.

The first is simply this: it could be that each of us possesses some distinction, though not the same one for all. You may be the world's greatest kazoo player; I may make the world's best Jello salad. And so on. You'd be special in the kazoo-playing way, I in the Jello-making mold and each other person in their own (dare I say) special way. But there's another possibility.

It doesn't make sense to say that we're special, period. If any of us are special, it's because of something about us. But suppose some characteristics are intrinsically valuable -- that is, have value all by themselves, without reference to anything else. Perhaps being conscious counts; perhaps having the ability to reason does, or the capacity to participate in the moral community. Or suppose that standing in a certain relationship to something with its own intrinsic value counts. The religious might say, for example, that being creatures made in God's image gives us infinite value. We might well say: anything that is conscious, or can reason, or is created in God's image... is special exactly on that account. However, it could turn out that all of us have the special-making feature. If so, we really are special on that account, but it's not a contest; we could all be winners.

This isn't a purely idle notion. Many religious believers think we were created in God's image. Many Buddhists believe that we all have the capacity for Enlightenment ("Buddha nature"). Kant claimed that we all possess an infinite value that goes with our moral nature.

Whether there's any such trait that we all possess is quite another matter; there may well not be. But the issue here was the logic of the matter. And on that account, it does seem that even though we couldn't all be above average, we could all be special.