Privilege is in part the absense of non-discriminatory hardships. For example, members of a minority have a harder time finding others in their group to associate with. That's real but not discrimination. White Americans are a majority so don't have that hardship; they have its absence, privilege.
I think that in the American society today, both whites and non-whites have advantages over each other. Sure, white people are often born into wealthier circumstances, but they face the discrimination of affirmative action. Thus, the phrase "white privilege" doesn't carry much weight with me.
Privilege is in part the absence of discrimination that others experience. To the degree that black Americans experience discrimination for being black, white Americans do not experience that and so have white privilege.
mangala
