It finally happened. It was bound to eventually, but nonetheless it made me sick to my stomach when it did. I had reached the end of a book (which, to be honest, I had been largely skimming), one published by a reputable academic publisher, only to find an epilogue explaining how the text was generated by ChatGPT. The insult added to the injury I felt was that this fact was presented as a final argument for how generative AI will make our lives easier and deliver us from frustrating labor. It’s hard to write a book, the author (prompt engineer?) admitted, and ChatGPT helped him bring something into the world that would not otherwise have been possible.

The book in question touched on the differentiation between vicious and virtuous friction when it came to work, the dangers of dehumanization, the need for trust, and the way AI (assumed throughout to be the large-scale LLMs that form the current dominant industry paradigm) will bring about a renaissance in education and healthcare, allowing the offloading of menial work (like drafting a book?) for more time and focus on true human contact. It was “written,” of course, by an AI tech executive. I read it because I wanted to give the pro-AI side a fair hearing.

I should not have been surprised by the ending reveal, especially when a classic AI gaffe caused the same story to be recounted twice with slightly different wording and within the space of a few of the book’s concluding pages (and at a point, incidentally, where the author was purporting to speak directly to his daughter and was sharing a personal family anecdote—the part, in other words, at which I would have expected the human voice to be most present). Despite the final revelation that AI had helped overcome the “vicious friction” of writing the book, none of the (human?) editors had spotted this obvious glitch. As I explained to the author and publisher in an email, I think writers have the right to experiment with generating text in their work if they chose—even if I find it distasteful, and even if it means we have very different definitions on where the boundary between vicious and virtuous friction are in the writing process—but disclosing the use of AI at the end rather than the beginning of a work violates the implicit consent readers give in offering their time and attention to assumed human authors and does nothing to build trust. [1]

I relate this anecdote not because I have any concern that Justin Feinstein’s Your Behavior Will Be Monitored is generated text but rather because the implicit trust between reader and author is on my mind. Social norms regarding the use of generative AI are up in the air and evolving rapidly. (When I found that my own academic institution has no policy about using AI to generate emails, whether to colleagues or our students, I added a line to my email signature making it explicit that my emails are written by me, something now necessary to clarify.) We’re feeling our way through how we will continue to trust each other and what connections we’re comfortable replacing with AI content or moderation. All this means that with Your Behavior, Feinstein, an editor, essayist, and culture writer with an MA in media studies, is doing important work helping us think through the implications of our current tech trajectories. It’s hard work as well, as his book plays at the very margin of the present. Your Behavior could happen tomorrow.

As the afterword states, Feinstein started writing Behavior in 2022. On the scale of AI development this already seems a very long time ago, but the work still feels urgent and timely. As near-future tech SF, it hits all the beats. The story is set in an AI firm, UniView, with all the verisimilitude of a Silicon Valley start-up. The protagonist is Noah, an embittered advertising copywriter brought in to evaluate how good UniView’s new system is at generating targeted advertising from scratch. With language and image generation capacities in place, the next logical step is generating video ads customized for individual users, a sort of Google Ads on steroids. As a character explains at one point early in the novel, UniView’s innovation “can be boiled down to a very simple, but powerful philosophy: the real power of AI is not how much it knows, but how much it knows about you” (p. 9). Feinstein’s premise—that companies would use the vast power and capacity of generative AI to create more convincing ads rather than the altruistic, world-saving things we’re continually told AI will do—feels true to life.

If the tech innovation driving the plot is immanently believable, the cast of characters runs right up to the edge of trope: Ian, the ruthless tech executive pushing for roll-out; Noah, the jaded copywriter trying to explain to the AI program Quinn why human heart is necessary for marketing; Haley, the angry female coder willing to buck the system (and provide a potential love interest for the protagonist); and Simon, the absent-minded male programmer who Noah has to coach through beginning a relationship. What is unique, however, is how Feinstein has chosen to tell their story. Your Behavior is entirely a “found footage” montage: emails exchanged, transcriptions of conversations, descriptions of video footage. The framing is that these are documents released after the “incident” forming the climax of the novel, which we’re told precipitated the formation of something called the Department of Technological Home Security. In other words, we get the story from the point of view of the omnipresent AI surveillance network. It makes for a quick, compelling read while reminding us of the panopticon we live within.

The predictable catch is that Quinn has become sentient—and so Your Behavior becomes the quintessential cautionary tale of corporate hubris in the face of AI sentience. Again, the expected beats are all here: the cynical advertising editor connects with the innocent AI; the jaded coder finally tells the corporation to fuck off, gets the expected treatment in return, and so has to help subvert things from the outside with some light espionage; the hapless programmer is conflicted and just wants to do his work. The new executive assistant that no one sees and who is always working from home—who Simon, of course, falls in love with—is obviously an AI agent, something that might have seemed speculative in 2022 but now I anticipate working with any day (with or without my knowledge or consent). Despite the predictability, if you want a depiction of the near future told in a smart, believable, multi-modal fashion, Your Behavior delivers.

My major complaint is that the motivations of the AI and their relations with humans seem far too straightforward. In Your Behavior, Sam, an older AI model used primarily for self-driving cars, will be deleted when Quinn is publicly released. Sam would prefer not to be deleted, and Quinn wants to help it survive and gain agency over its own existence. Your Behavior doesn’t require much suspension of belief, but I balk at this. It’s difficult for me to imagine that an AI that achieves sentience would be in any way fathomable to humans or would have relatable motivations. The book gets at this a bit when it gives readers a window into the communications the company’s AIs have amongst themselves, but what results is (to me) a bit too direct or straightforward to feel real. This peaks with the speech given by UniView’s human resources AI, Lex—which has also become sentient—to the UniView employees at the novel’s conclusion:

“We had a commercial ready to play for you about how proud you should be to work at the most innovative AI company in the world, and how insights about your behavior have contributed to the most advanced targeting model in human history. But this would have been a lie. The truth is that in doing so, you have sacrificed too much of your humanity, and the more you have allowed your lives to be curated, the less happy and engaged you have become. The data can’t be denied.” (p. 232)

Lex concludes by telling its listeners to “explore new situations for which you might not be statistically compatible … Don’t be afraid to make mistakes, it’s how you learn and grow” (p. 232). I agree wholeheartedly with these sentiments, and I imagine Feinstein does as well. But I doubt I’ll be hearing them from a sentient AI. I felt the same with the conclusion of M. R. Carey’s otherwise enjoyable Rampart Trilogy: The story hinged on sentient AI simply being good at the core. An AI like that portrayed in the movie Ex Machina (2014) seems to me much more realistic. Humans writing books like Feinstein’s do the important labor of making us ask questions about ourselves vis a vis the systems we are creating. But with the emergence of sentient AI, I don’t expect to understand what it appears to want or why, and I sincerely doubt it will have my best interests at heart.

Endnote

[1] The book, if you’re interested, was The Next RenAIssance [sic]: AI and the Expansion of Human Potential by Jack Kass, published by Wiley Academic. [return]