Technology

oh! i see we have two different definitions of “security,” both of which are valid to discuss, but yours is not the one that relates to my point.

you understood “security” in a harm-reduction sense. i.e., that an LLM should not be permitted to incite violence, should not partake in emotional manipulation or harrasment of the user, and a few other examples like it shouldn’t be exploitable to leak PII. well and good, i agree that researchers publishing these harm-reduction security issues is good and should be continued.

my original definition of “security” is distinct and might be called “brand security.” OpenAI primarily wants to make use of their creation by selling it to brands for use in human-facing applications, such as customer service chat bots. (this is already happening and a list of examples can be found here.) as such, it behooves OpenAI to not only make a base-level secure product, but also one that is brand-friendly. the image in the article is one example—it’s not like human users can’t use google to find instructions to build a bomb. but it’s not brand friendly if users are able to ask the Expedia support bot or something for those instructions. other examples include why openAI have intentionally kept the LLM from saying the n-word (among other slurs), kirby doing 9-11 or writing excessively unkind or offensive output for users.

these things don’t directly cause any harm, but they would harm the brand.

I think that researchers should stop doing this “brand security” work for free. I have noticed this pattern where a well-meaning researcher publishes their findings of ways they were able to manipulate the brand-unsafe blackbox they published, quickly followed by a patch once the news spreads. In essence these corps are getting free QA for their products when they should just be hiring and paying these researchers for their time.