I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.
If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object. Even if there is some nugget of truth to the idea that there may be some implicit language bias which relates to race in there.
I think Dan does a decent job showing that there is some level of racial bias -- not on the level of "the model refuses to show asian people," but on a level of "the model conflates certain jobs with certain races," and that's fair. But I just found the lede of, "Why don't people admit there's a bug in AI" to be a little obtuse.
> that there may be some implicit language bias which relates to race in there.
I think there is some stuff in the middle. I think disadvantaged groups deal with more of these bugs by being underrepresented in the teams that design this stuff.
Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.
Especially once we keep getting adverse outcomes and don’t manage to prioritize fixing it.
> Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.
I'm struggling to extract the nuance that I feel is embedded here. Is it an ontological issue, a quantitative, or a qualitative dimension that puts it in the realm of kind-of?
There is a large amount of academic literature claiming racial groups are purely ethnocultural, and in that light it is ridiculous to claim a soap dispenser is racially biased.
> I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.
In the initial examples, it's not a slight of hand. The thing has racial biases so bad that they bugged the output. It's job was to convert a casual photo of a particular person into "professional" photo, and instead of just changing the clothes and setting, it changed the person too.
Then all kinds of apologists tried to gaslight the bug away instead of acknowledging the system is faulty and not fit for purpose.
> If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object.
And what if they just said it was an instance of "bias," like the OP describes similar bugs? I don't think you'd have grounds to object.
In the example of "upload a photo of an asian person, ask the model to convert it into a professional linkedin profile photo, race gets changed" that's the model not doing what it was asked, yes.
In the example of "asked for images of a chemistry professor many times, always a white guy wearing glasses" on the other hand the model does what it's asked, in that the images do look like chemistry professors. However the output demonstrates gender and racial bias.
"Bias in relation to what?" is the huge unanswered question I left the article with.
Are we even talking about the same thing when we discuss this topic? Are we talking about bias with respect to the training set, or bias in the training set with respect to reality, bias with respect to our expectations, or bias in reality? Each of these are completely different problems and each have different causes, importance, and consequences.
I can't help but think that at least some people are uncomfortable with reality being reflected back at them. When we generate images conditioned on occupation, should they reflect the racial proportions documented by the BLS[1]? It feels very, "I don't care what you do about them, but I don't want to see homeless people." Being confronted with reality is deeply unsettling for some people. Likewise, I'd be unsurprised to hear that some people would be uncomfortable if images generated by conditioning on occupation did accurately reflect reality because in their minds, reality is more diverse than it really is.
I’m gently optimistic on the subject. When bias is encoded in the synaptic weights of a human brain it’s extremely hard to quantify. You can’t run an ablation experiment or try different combinations of similar inputs to determine if a judge is biased, for example.
AI is materializing existing biases, perhaps amplifying some in the short term. This is object-level bad if we start hooking up important systems without building guardrails, eg I’d be worried about a “legal advice from AI” service right now.
At the meta-level this is an opportunity to run those experiments and root out some of the areas where bias does creep in. I think a lot of the coverage on the issue (not referring to OP here) fails to look past the object level and in doing so misses the big opportunity.
Of course, when you actually start having these conversations, you get to some very tricky discussions about what “fixing bias” actually means. In so many areas it’s a lot easier to throw around fuzzy rhetoric rather than a quantitative model that encodes a specific solution. But AI systems require precisely that.
I think it's difficult to conceptualize a program's behavior as "buggy" when the program has no specified behavior. The vexing thing about LLMs and image generators is that they ultimately have no intended purpose or principled construction as artifacts, being mostly discovered through trial-and-error and pressed into service due to various misconceptions about what they are — human imitators, fact retrievers, search engines, etc. But they are really just statistical regurgitators over whatever the training dataset is, and while that type of an artifact seems to sometimes prove useful, it's not something we have yet developed any kind of principled theory around.
One example is in DALL-E initially going viral due to its generated image of an astronaut riding a unicorn. Is this a "bug" because unicorns don't exist and astronauts don't ride them? One user wants facts and another wants fancy. The decision about what results are useful for which cases is still highly social and situational, so AIs should never be put in a fire-and-forget scenario or we will see the biases Dan discusses. AIs are more properly used as sources of statistical potential that can be reviewed and discarded if they aren't useful. This isn't to say that the training sets are not biased, or that work shouldn't be done to rectify that distribution in the interest of a better society. But a lot of the problem is the idea that we can or should trust the result of an LLM or image generator as some source of truth.
I feel like we need to disentangle a bunch of layers here. To take a stab at a quick categorization:
1. Policy bias, like if someone put in a system prompt to try to trigger certain outcomes.
2. Algorithmic/engineering bias, like if a vision algorithm has a harder time detecting certain details for certain skin tones under certain lighting.
3. Bias inside the data set which is attributable to biased choices made by the company doing the curation.
4. Bias in the data set which is (unlike #3) mostly attributable to biases in the external field or reality.
I fear that an awful lot of it is #4, where these models are highlighting distasteful statistical trends that already exist and would be concerning even if the technology didn't exist.
If I could channel some of my less imaginative QA colleagues, it's not a bug if it's not specified in the acceptance criteria. A bug is something contrary to the expected output. If the AI tool producers never cared about bias, then it's not a bug. And it will probably never be treated as such until they have a liability issue to contend with.
He asks for a prediction — will this still be the same state of affairs in 2033, where “same” means models encode source data bias and we don’t have in-model ways of dealing with that bias. I’d predict “yes” on that, with some caveats.
What practitioners seem to be doing now is using prompting modifications to inputs to get desired diversity spreads out of the models. I’ve written a bit about this, but I think doing this openly, with user choice, is great, and doing it secretly, without notification is evil. I think a lot of people feel this way, and it explains much of the outcry against race-shifting founding fathers.
Whatever I think about it, we’ll see a lot of that by 2033. I do think we’ll see much more sophisticated use of controlnets / LoRAs / their successors to nudge / adjust inference at the weight level, vs. prompting. These are useful right now, and super sophisticated, they’re not just for bias-related changes, almost anything you can prompt up could become a vector you adjust LLM behavior on. So, I think we’ll move out of the Stone Age and into say the Bronze Age by 2033.
That said, Dan does make a fundamental input bias error here, which is common to do when people explore and write about diffusion models, but really important to test — what does the source input image randomness look like? A diffusion model moves some number of steps away from some input. Typically random noise. This random noise has a lightness level, and also at times color tone. By default, in most inference systems, this image is on average tone-neutral (grey) and very light.
If you’re going to generate sample images without keeping track of seeds, fine, do a bunch, like he did here. But, if you’re going to determine how likely ‘whiteness’ is on a given prompt, you need to be very aware of what source image you’re giving the model to work on. Especially when we are talking about facial and race discriminators that are judged largely on skin tone. A white input image is easier to turn into a white face, and requires less deviation from the original image, and so on average, it will be preferred by most diffusion model generation stacks.
So, is PAI or Stable Diffusion biased over and above the world’s image data levels in their model? Maybe, I don’t know. Is it biased at the world’s image data levels? Maybe, probably? But, I don’t think you can pass a Gaussian noise image defined to have a fairly white lightness value and grey color tone to a thing, ask it to draw a face, and then say it’s white-face-biased a priori — you’re starting the model out with a box of very light crayons and making it ask for other colors from the cabinet vs using what’s at hand.
Anyway, I don’t think this takes away from Dan’s fundamental point that this class of ‘bug’ is not going away, especially in that it’s harder to even agree on what is a bug. But, I’d like to see someone, anyone, talk about image generation bias while aware of what’s being fed into these models at the start of inference, it would raise the level of discourse.
> a common call to action for at least the past twenty years . . . has been that we need more diverse teams
I view calls for more “diverse” teams as a sort of general platitude, basically a thought terminating cliche.
The problem is that the team has made certain assumptions about the demographics of its userbase, not that the team itself is not diverse. The real world is too long tail to represent every demographic. The 40th most popular language has 46 million speakers.
To the extent it shows up in "AI", that's just GIGO.
What surprises and disappoints me is how many people (not so much TFA, but many comments here) seem to be expecting AI to be magical pixie dust which gives "the right answer", instead of, you know, an artificial intelligence.
I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.
If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object. Even if there is some nugget of truth to the idea that there may be some implicit language bias which relates to race in there.
I think Dan does a decent job showing that there is some level of racial bias -- not on the level of "the model refuses to show asian people," but on a level of "the model conflates certain jobs with certain races," and that's fair. But I just found the lede of, "Why don't people admit there's a bug in AI" to be a little obtuse.
> that there may be some implicit language bias which relates to race in there.
I think there is some stuff in the middle. I think disadvantaged groups deal with more of these bugs by being underrepresented in the teams that design this stuff.
Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.
Especially once we keep getting adverse outcomes and don’t manage to prioritize fixing it.
> Are soap dispensers that don’t give soap to people with darker skin racially biased? Kind of.
I'm struggling to extract the nuance that I feel is embedded here. Is it an ontological issue, a quantitative, or a qualitative dimension that puts it in the realm of kind-of?
3 replies →
There is a large amount of academic literature claiming racial groups are purely ethnocultural, and in that light it is ridiculous to claim a soap dispenser is racially biased.
2 replies →
> I think Dan performs a mild sleight-of-hand trick here: he asks why we don't consider this a bug when any other software would consider it a bug. But in fairness, the question was not, "Did this prompt have a bugged output," it's "Did it have a racially biased output," and that's a more emotionally charged question.
In the initial examples, it's not a slight of hand. The thing has racial biases so bad that they bugged the output. It's job was to convert a casual photo of a particular person into "professional" photo, and instead of just changing the clothes and setting, it changed the person too.
Then all kinds of apologists tried to gaslight the bug away instead of acknowledging the system is faulty and not fit for purpose.
> If I wrote software that choked on non-ASCII inputs for say a name, and then someone said, "Hey, that's a bug," cool, yes, fair. If someone said, "This is evidence of racial bias," I mean... I'd probably object.
And what if they just said it was an instance of "bias," like the OP describes similar bugs? I don't think you'd have grounds to object.
I think the article does a nice job connecting these bugs to other historical, non-AI bugs that result in a racially biased system.
But the LLM didn't do what it was asked
In the example of "upload a photo of an asian person, ask the model to convert it into a professional linkedin profile photo, race gets changed" that's the model not doing what it was asked, yes.
In the example of "asked for images of a chemistry professor many times, always a white guy wearing glasses" on the other hand the model does what it's asked, in that the images do look like chemistry professors. However the output demonstrates gender and racial bias.
"Bias in relation to what?" is the huge unanswered question I left the article with.
Are we even talking about the same thing when we discuss this topic? Are we talking about bias with respect to the training set, or bias in the training set with respect to reality, bias with respect to our expectations, or bias in reality? Each of these are completely different problems and each have different causes, importance, and consequences.
I can't help but think that at least some people are uncomfortable with reality being reflected back at them. When we generate images conditioned on occupation, should they reflect the racial proportions documented by the BLS[1]? It feels very, "I don't care what you do about them, but I don't want to see homeless people." Being confronted with reality is deeply unsettling for some people. Likewise, I'd be unsurprised to hear that some people would be uncomfortable if images generated by conditioning on occupation did accurately reflect reality because in their minds, reality is more diverse than it really is.
1. https://www.bls.gov/cps/cpsaat11.htm
I’m gently optimistic on the subject. When bias is encoded in the synaptic weights of a human brain it’s extremely hard to quantify. You can’t run an ablation experiment or try different combinations of similar inputs to determine if a judge is biased, for example.
AI is materializing existing biases, perhaps amplifying some in the short term. This is object-level bad if we start hooking up important systems without building guardrails, eg I’d be worried about a “legal advice from AI” service right now.
At the meta-level this is an opportunity to run those experiments and root out some of the areas where bias does creep in. I think a lot of the coverage on the issue (not referring to OP here) fails to look past the object level and in doing so misses the big opportunity.
Of course, when you actually start having these conversations, you get to some very tricky discussions about what “fixing bias” actually means. In so many areas it’s a lot easier to throw around fuzzy rhetoric rather than a quantitative model that encodes a specific solution. But AI systems require precisely that.
I think it's difficult to conceptualize a program's behavior as "buggy" when the program has no specified behavior. The vexing thing about LLMs and image generators is that they ultimately have no intended purpose or principled construction as artifacts, being mostly discovered through trial-and-error and pressed into service due to various misconceptions about what they are — human imitators, fact retrievers, search engines, etc. But they are really just statistical regurgitators over whatever the training dataset is, and while that type of an artifact seems to sometimes prove useful, it's not something we have yet developed any kind of principled theory around.
One example is in DALL-E initially going viral due to its generated image of an astronaut riding a unicorn. Is this a "bug" because unicorns don't exist and astronauts don't ride them? One user wants facts and another wants fancy. The decision about what results are useful for which cases is still highly social and situational, so AIs should never be put in a fire-and-forget scenario or we will see the biases Dan discusses. AIs are more properly used as sources of statistical potential that can be reviewed and discarded if they aren't useful. This isn't to say that the training sets are not biased, or that work shouldn't be done to rectify that distribution in the interest of a better society. But a lot of the problem is the idea that we can or should trust the result of an LLM or image generator as some source of truth.
I feel like we need to disentangle a bunch of layers here. To take a stab at a quick categorization:
1. Policy bias, like if someone put in a system prompt to try to trigger certain outcomes.
2. Algorithmic/engineering bias, like if a vision algorithm has a harder time detecting certain details for certain skin tones under certain lighting.
3. Bias inside the data set which is attributable to biased choices made by the company doing the curation.
4. Bias in the data set which is (unlike #3) mostly attributable to biases in the external field or reality.
I fear that an awful lot of it is #4, where these models are highlighting distasteful statistical trends that already exist and would be concerning even if the technology didn't exist.
Absolutely. Our language is not helping here. The term “bias” is completely overloaded.
If I could channel some of my less imaginative QA colleagues, it's not a bug if it's not specified in the acceptance criteria. A bug is something contrary to the expected output. If the AI tool producers never cared about bias, then it's not a bug. And it will probably never be treated as such until they have a liability issue to contend with.
The data is correct, but bad. What do we do in a case when we can't fix the source data?
Substantive effort by Dan, as usual.
He asks for a prediction — will this still be the same state of affairs in 2033, where “same” means models encode source data bias and we don’t have in-model ways of dealing with that bias. I’d predict “yes” on that, with some caveats.
What practitioners seem to be doing now is using prompting modifications to inputs to get desired diversity spreads out of the models. I’ve written a bit about this, but I think doing this openly, with user choice, is great, and doing it secretly, without notification is evil. I think a lot of people feel this way, and it explains much of the outcry against race-shifting founding fathers.
Whatever I think about it, we’ll see a lot of that by 2033. I do think we’ll see much more sophisticated use of controlnets / LoRAs / their successors to nudge / adjust inference at the weight level, vs. prompting. These are useful right now, and super sophisticated, they’re not just for bias-related changes, almost anything you can prompt up could become a vector you adjust LLM behavior on. So, I think we’ll move out of the Stone Age and into say the Bronze Age by 2033.
That said, Dan does make a fundamental input bias error here, which is common to do when people explore and write about diffusion models, but really important to test — what does the source input image randomness look like? A diffusion model moves some number of steps away from some input. Typically random noise. This random noise has a lightness level, and also at times color tone. By default, in most inference systems, this image is on average tone-neutral (grey) and very light.
If you’re going to generate sample images without keeping track of seeds, fine, do a bunch, like he did here. But, if you’re going to determine how likely ‘whiteness’ is on a given prompt, you need to be very aware of what source image you’re giving the model to work on. Especially when we are talking about facial and race discriminators that are judged largely on skin tone. A white input image is easier to turn into a white face, and requires less deviation from the original image, and so on average, it will be preferred by most diffusion model generation stacks.
So, is PAI or Stable Diffusion biased over and above the world’s image data levels in their model? Maybe, I don’t know. Is it biased at the world’s image data levels? Maybe, probably? But, I don’t think you can pass a Gaussian noise image defined to have a fairly white lightness value and grey color tone to a thing, ask it to draw a face, and then say it’s white-face-biased a priori — you’re starting the model out with a box of very light crayons and making it ask for other colors from the cabinet vs using what’s at hand.
Anyway, I don’t think this takes away from Dan’s fundamental point that this class of ‘bug’ is not going away, especially in that it’s harder to even agree on what is a bug. But, I’d like to see someone, anyone, talk about image generation bias while aware of what’s being fed into these models at the start of inference, it would raise the level of discourse.
> a common call to action for at least the past twenty years . . . has been that we need more diverse teams
I view calls for more “diverse” teams as a sort of general platitude, basically a thought terminating cliche.
The problem is that the team has made certain assumptions about the demographics of its userbase, not that the team itself is not diverse. The real world is too long tail to represent every demographic. The 40th most popular language has 46 million speakers.
ChatGPT, in my experience, is substantially less biased than most people. I’m impressed by that, fwiw
"Racial" bias is a bug in people.
To the extent it shows up in "AI", that's just GIGO.
What surprises and disappoints me is how many people (not so much TFA, but many comments here) seem to be expecting AI to be magical pixie dust which gives "the right answer", instead of, you know, an artificial intelligence.