It is difficult and not that black and white (pun not intended). I would personally say that for external factors it is skin color. Like how other people perceive you. That's why the term 'white-passing' exists. But for internal factors it is different. Like, for example, I am a quarter black. But just like a lot of people who are a quarter black or anything, I look white. So to other people I am white because they can only see skin color, but I know that's not the whole truth because I know my family history. But I am still given privileges in society purely based on the fact that I look white that people with darker skin don't get. I have fully black friend and when we were in high school (didn't know each other back then yet) we both skipped school so much that we got in trouble with attendance laws in our country. We did the exact same thing, and got community service, but I, as a 'white' girl, got to spend it helping out a library, while he, a black young man, had to pick trash at the side of the road.

It is to some extent also influenced by ethnicity, more so in the past than now, like how Italians and the Irish weren't considered white when they immigrated to America by the people of English descent. But because of the history of colonialism and slavery by majority white countries over majority non-white countries, it has almost exclusively connotations to skin color now.

Of course, 'race', in the biological sense, doesn't actually exist in humans. We're all the same race, homo sapiens. But we still use the word race today as a shorthand because historically, when people didn't have as much knowledge about biology as we do now, people did actually think the different skin colors were different human races. So, biologically, neither white nor black nor Latino nor Asian nor whatever is an actual race, but socially, societally speaking, it is a useful shorthand for skin color because of difference in treatment, privilege, education, income, etc etc that comes with it from centuries of oppression, colonialism, slavery, exploitation and ('scientific') racism.