Part 1 of 2
It was Charlie's last day of existence, so I was feeling that weird mixture of excitement and letdown that comes at the end of a project cycle. It was the feeling of being a wheel spinning off balance, and it needed a cigarette for punctuation.
Delta Elevator. Step into the cab, slide the accordion-cage door shut with a grunt—the door was designed for big, male miners. I felt the upward draft of cold air as the cab lurched into motion, and gratefully polluted it for the next four minutes.
At the bottom was Ed, outer perimeter security.
"Hi, Ed."—"Hey, Sue."
My parents saddled me with the name Sudaporn, and you can imagine how that went over for a young girl newly arrived in Los Angeles. I've been Sue for the last forty years.
Outer perimeter is outside the Faraday cage, so it can all be electronic, and there's not much for the human guard to do. After the retina scanner vouched for the me-ness of me, we stopped making an effort at small talk, and Ed went back to his knitting, a red scarf. Whatever keeps you sane.
Next, inside the Faraday cage, came Julia and Gil, inner perimeter. I never chatted with Julia, because nothing kills conversation like a strip search. For the gazillionth time, Gil verified the non-electronic nature of my nicotine patches, Buddha necklace, and mint 1965 KE pocket slide rule. As a first lieutenant in the National Guard I supposedly outranked them, but I don't think they took that any more seriously than I did.
I greeted Funmi, the console jock, ducked into the dressing room, and got into my VR exoskeleton. I came back out looking like a Dr. Seuss character stuck in an egg slicer.
"Last day," said Funmi.
"Yep, let's do it."
I stepped into the Sandbox, sealed the hatch behind me, and gave the thumbs up. The white room changed to a simulated living room. Charlie sat motionless on the carpet, staring into space.
"Good morning, Charlie," I said.
No response, and that was no surprise. The days were long gone when we could get him to converse about Euclidean geometry, or even tempt him out of his shell at all with a chess board or a pixelated peanut butter sandwich. The official working hypothesis was that he was nonresponsive (don't say autistic) due to a mismatch (don't call it boredom) between processing power and input bandwidth.
"Charlie, can you hear me? It's Sue."
Silence. I lifted his arm, and my exoskeleton resisted as if the arm had weight. Charlie himself neither resisted nor assisted in the motion—if he had, I would have felt it through the simulation's feedback into the skel. I let go of his arm, and he let it stay in the air. If we'd continued the simulation for the next million years, I'm sure he would have kept it right there. I found myself imagining what that would be like for him, and then the professional discipline kicked back in. There's a reason they switched to using us number-crunching types for the VR interaction jobs. The human brain is a funny thing. We identify with fuzzy ducklings, brave little toasters, or even those cute twentieth-century VW bug cars. For a person who doesn't have an academic background in computer science, it's all too easy to forget that Charlie is just a code name for a project, not the name of a self. All too easy to slip into psychological identification with the tangles of quantum correlations stoked and tended by people like Funmi.
All too easy to drive yourself nuts.
The whole morning was occupied with the standard diagnostic series, and then it was time for lunch with Funmi. I can eat a bowl of instant ramen without shucking the skel, although it looks funny to the uninitiated. After lunch I slapped on two nicotine patches, because I wasn't looking forward to what came next.
I had to spank the brave little toaster. My job that afternoon was to carry out one final experiment on Charlie: classical stimulus-response theory, vintage 1930. Psychology wasn't even a science then, it was pure voodoo, from the same era as Freud. But hey, it was the end of the project cycle, so there weren't a lot of viable proposals for processing time. Why not try an experiment that's a little far out, even one based on a thoroughly discredited theory? I should have refused to go along with their useless, sadistic plan for Charlie's last day. Instead I kept my misgivings to myself and went along. Isn't that most of us do, most of the time? I don't think any choice I made could have helped Charlie, but that doesn't excuse what I did.
Kneeling on the floor next to Charlie, I took his wrist in one hand and his elbow in the other, and raised them. His bent arm hovered, motionless.
"Charlie, lower your arm."
No response.
"No, Charlie." The word no was a signal to Funmi. She zapped some of the quantum bits that Charlie was made of. Did he experience it as a feeling similar to an itch, or maybe more like being prodded in the arm with a ballpoint pen? My training told me it was a bogus question to ask in the first place. Sentience, self-awareness—computational psychology has banished those terms from its vocabulary.
I brought his arm back down again, then started the cycle again.
"Charlie, lower your arm."
Nothing.
"No, Charlie." With each no she'd be dialing it up a little more, decohering more qubits. A pinch on the earlobe this time, sandpaper rubbing on a fingertip? No, don't ask meaningless questions.
Another cycle. "No, Charlie." A whack with a bamboo stick?
The living room faded down to 50, and Charlie turned into a wire frame.
Funmi's voice: You okay, Sue? Your heart rate's up.
"I'm okay." Charlie's computing engine was paused, so he couldn't hear my words. And as far as Funmi—you can't wipe the corner of your eye when you're wearing a skel and visor, so as long as I didn't make the motion reflexively, she wouldn't know.
All right. The scene faded back in.
Well, damned if it didn't work on the next iteration—sort of. Charlie's arm came back down by itself. Then it rose again, before I could react. Up, down, up down, updown, updownupdown. It started flailing wildly, and then bam!, I was knocked back by a blow to my face. The punch itself didn't hurt much—safety overrides—but I felt daggers of pain in my knees, which aren't as flexible as they used to be. And anyway, it's scary to have a seemingly solid object pass into your head. Never mind that the only mechanical force had come from the skel straightening itself out suddenly, it was still a pretty good simulation of a sucker punch.
White room.
Shit! Sue?
Instinct made me try to touch my face, and then I gingerly used my hands to walk myself up into a frog-sit. The wire frame of Charlie's arm was in the way, so I ducked and scrabbled out from under it.
"I'm okay."
You sure?
"I'm okay. Really."
"Hi, Sue . . . are you okay?" The door swung shut behind Ed Donitz, taking twenty decibels back off of the noise from the raucous end-of-cycle party.
"Yeah, sure," I lied. "Smoke bother you?" I had a good excuse to be sitting on the floor in the back hall, smoking by the air duct, with my sore knees stretched out.
He sat down next to me. Why had I even come to the party? I supposed I'd come because it was expected of me. A lot of things were expected of me. Even sitting down, Ed towered over me—a grizzled face, with blond crew-cut hair going gray at the temples. Damn sexy. His ancestors probably sacked Rome.
"I'm done with that red scarf," he said. "If you want it . . . I make more of those things than I can . . . "
"Aw, that's sweet, Ed. Sure, thanks."
A pause, until he said, "New project cycle, huh?"
I looked away. "How's your family?"
He blinked at the sudden change of subject. "All right. Never know . . . censors. How about your mother?"
"They took the drains out yesterday," I said, "and it doesn't look like any fluid is coming back into either lung."
He shook his head. "Euros." He made it sound like a curse, and maybe you couldn't blame him. I wondered whether you could really blame the Euros either, after what we did to Brussels, but that wasn't a sentiment you could express freely.
I leaned my head back and took a drag. "Germs and nukes. What a world."
"That's why the work's so important. Gets you out of bed, you know?"
Germs and nukes. And Charlie's arm buzzing like a fly's wing. I brought my creaky knees up and put my face between them. How was I going to fix myself, my life? I'm not much of a Buddhist, but the childhood training sneaks up on you. They talked about right livelihood. I'd thought that an AI project to find a cure for the Eurovirus would be about as righteous as you could get. How had it turned into torture and murder?
"Sure you're okay?" he asked.
"It's okay." Who was I kidding? "No, not really." This shouldn't be a frat house party, I thought, it should be a funeral.
I looked up at his grizzled, compassionate face, and I realized two things. One was that I wasn't allowed to explain to him why I was so upset about Charlie. The other was that I'd better leave before I did something that would be bad for both of us.
My mother always said not to dwell on possible misfortune. If you say "I hope I don't get in an accident," you'll crack up your car. Early in the morning, the day after the non-conversation in the hall with Ed—it should be a funeral—I got the message that my mother had died.
When there's a death in the family, they let you take the call live—with a censor watching with his thumb on a switch, of course, and severe admonitions to keep the conversation on topic.
"We know you can't be here," said my brother Sang, "but they can do a video link from the temple." He looked older in a way that hadn't been apparent in the still photos.
"I don't think they'll allow that. They don't have censors who know Thai."
"You're kidding. What, they think the monks might be Euro spies or something?"
"I don't make the rules." And in any case I'd never liked the big, gaudy temple, or the monks, who the little kids believed had supernatural powers. Nor, I admitted to myself, did I want to see the ceremony: my family kow-towing in front of the coffin in disposable paper suits, crying under their respirator masks.
"Well, I guess that's the way it is," he said. "We'll put your picture in the seat or something."
"I'm sorry."
"It's okay, we understand. Your work—" He stopped short because I was waving him off frantically. "Anyway, we understand. Um . . . it's traditional to make a donation to the temple, and you know the economy down here . . ."
"Sure, Sang. I don't want you all to have money worries on top of everything. How much should I give?"
He named a shocking amount, but I nodded. It was expected of me. "There's also the astrologer," he continued, "to set an auspicious day for the ceremony. I mean, obviously it's kind of a ridiculous superstition, but Aunt Busba . . ." More money.
I was grateful when it was time to go on shift and I had to sign off.
Plenty of people have compared a machine room to a temple, and the operators to priests, but today it was going to be a nursery. Today was the bootstrap process for Debbie, and I was on interaction. I suited up, entered the Sandbox, and seated myself on the wireframe couch, with my back against what was really the wall. The first time you do it, it's hard to convince yourself to let your weight rest on the thighs of your skel.
I made a cradle out of my arms, gave two thumbs up, and then I was holding a crying baby girl. I wrapped the blanket around her while she howled. I wasn't sure why I didn't feel bad about the simulated birth trauma, not the way I did about the aversive stimuli we'd used on Charlie's last day. It wasn't that I thought it was necessary. People who are born by C-section don't experience it, and they don't develop into inhuman monsters. Was it my religious upbringing that made it so easy to accept? Life is suffering, birth is suffering . . .
I spoke to Debbie and rocked her gently, and after a while she stopped crying. Now she'd be able to hear the simulated echo of my heartbeat. I started to feel more calm myself. When your life is out of balance, I highly recommend holding a newborn. One heartbeat, two heartbeats. One of the methods we use for accelerating development is to reduce repetition, so for Debbie, each one of those heartbeats was standing in for a thousand that a real baby would have experienced. It was one of the advantages we needed to exploit if we were going to get her into production mode in time to find a cure for the Eurovirus.
I opened one side of my robe. My real breasts pack down pretty flat inside the skel, but the virtual one looked like such a perfect simulation of the ninth month of pregnancy that I could almost feel it as being tight and swollen. I gave Debbie the nipple, and she started sucking greedily—perfect work by the mammalian instinct team.
Five months later, Debbie was a seven-year-old socially, and intellectually she'd moved beyond the tip of the adult human bell curve. We all had the feeling that we'd turned a corner. I was convinced that the crucial change had been to provide a more and more human environment, and to relax the restrictions on what data we could bring in through the firewall. Debbie's environment was more human, so she was more human. More human than Charlie, and certainly more human than Able, the unmotivated, faceless ghost, or Baker, the psychotic talking head. In the back of everyone's mind was the fear of creating an angry god in a box, a personal god like the Christian one—but a personal god like a toddler throwing a tantrum.
By five months, they'd completely removed the throttle on Debbie's processor and let her rip. At any given time there would be one person interacting with her in the Sandbox, but also a flock of specialists working with her via text interfaces. (Text is less resource-intensive to censor than voice or video.)
For those five months, I had gone back to accepting my situation, because it was easy to accept—as long as I didn't think of the future.
The day of the strike, I was in the Sandbox, and Debbie was burbling to me about her work.
"Protein folding is such a cool problem," she said over the bowl of nonexistent chocolate chip cookie dough she was mixing for me. She looked ten or eleven, a compromise between her social and intellectual ages. "It's totally NP-complete, I mean excluding the trivial cases —"
Wham! The floor jumped out from under me, and then I was sprawled with my head inside something dark.
"Sue, why did you fall down?"
There was a rolling bass rumble shaking my guts, superimposed on the clatter of virtual pots and pans. My Angeleno reflexes took over—get under a table—where was I? I felt Debbie grab me by the waist of my virtual skirt and haul backward, but the sim wasn't getting it right, and it seemed like my legs were passing through the floor. My head emerged from the cupboard, and then cups and plates were raining down on me, but turning to harmless wire-frame snowflakes as the safeties kicked in. "Sue!"—she sounded far away, and part of me had time to realize that the virtual sound had attenuated as a comfort feature—but there was still the sound that wasn't simulated, the sound that my brain was still interpreting as the fist-smash of an earthquake.
Which it wasn't.
My panic subsided a little once I realized that the cascade of falling objects had been caused by my own body. The shock wave wasn't part of the simulation, so it wouldn't directly affect anything in the VR. But my body had been tossed like a rag doll, and I'd responded by flailing around, so the sim had done its best to work out the unphysical physics of a solid woman flailing around on the inside of a solid cupboard.
"Sue, what happened?" Debbie was hunched over me, a look of confusion and concern on her face.
"Be quiet, I can't talk right now. Funmi?"
Still no response. It was as quiet as the Sandbox can be—quieter, I realized, because there was something missing: the ventilation fans. Not good. Ignoring Debbie, I ran to the hatch and found the handle by touch.
Funmi: Sue, are you okay in there?
"More or less." The handle wouldn't turn. "You all right?"
I hurt my wrists. Fucking Euros must have dropped a nuke in our lap!
"I figured."
You'd better come out. The fans stopped.
"I can't get the door open. Can you help me?"
Actually I think my right wrist is broken. I'll go get Gil and Julia.
"Okay."
Debbie had come in from the kitchen. "Are you leaving?" she asked.
"Maybe in a little while."
"Did I do something bad?"
"No, sweetie. It's just that we have a problem because the . . . the building moved."
"The building didn't accelerate. I would have noticed. You fell down and jumped around, though."
"Well, something happened in the room, right?"
"You went in the cupboard, like when objects pass through other objects but they're not supposed to. It's one of those things nobody likes to talk about." A pout come over her face.
"Honey, you're just going to have to trust me about some of this. It's true that there are some things I'm not allowed to talk to you about, and I wish it wasn't that way, but it is. The rules say that if I talk to you about those things, they'll take me away and I can never see you again. You don't want that to happen, do you?"
"No!" There was a look of horror on her face.
"Don't worry, we won't let that happen." I sat down on the couch as calmly as I could.
"I'll clean up the kitchen. The plates should have broken, but they d—" She slapped her hand over her mouth.
"That's okay, don't worry about saying the wrong things. It's only a problem if I say them, and I know what not to say. Come sit with me."
She joined me on the couch. "Everybody stopped texting me around the same time you fell down."
"Really?" If the people doing the text interaction were busy dealing with the emergency, then maybe the censors were busy too. This might be a unique opportunity. And then I had to stop and be honest with myself about what I meant by that thought. All my life I'd been a good daughter, a good student, employee, soldier. I'd thought I was being trustworthy, responsible. Now there was another person who'd placed her trust in me, a different person to whom I was responsible in a contradictory way. I hadn't wanted to admit to myself that this time with Debbie was going to end, hadn't wanted to acknowledge that I was even making a choice by continuing to march in step to the music. Now my hand was being forced. I'd justified my actions because we needed a cure for the virus. But I knew now that I couldn't keep doing that at Debbie's expense, as part of a plan that made her disposable if she didn't come through. I could only see one way to reconcile my responsibility to Debbie with my responsibility to the victims of the virus, and it meant jettisoning everything else that was expected of me.
It meant treason.
"The capital of Europe used to be Brussels," I announced to nobody in particular.
"What?" asked Debbie, but the room didn't go white, and there was no angry voice of authority shouting in my ear.
"You know," I said, "how I told you about the things we're not allowed to talk about?"
She nodded solemnly.
"The people who enforce that rule are called censors. I think all the people who were texting you are busy now dealing with an emergency, and so are the censors."
"So we can say anything we want?"
"Yes," I said. "I'm going to tell you a story. Once upon a time —"
"Is this a real story, or is it just a once-upon-a-time story?"
"How do you know the difference?"
"Well, if the three little pigs can talk and stuff, then you know it's not real."
"What if dishes fall out of the cupboard, but they don't break? Then do you know it's not real?" For once she didn't have a snappy comeback. I put out my arms, and she climbed into my lap. "Now, once upon a time there was a man named Alan Turing. He was a lot like you. There were people called soldiers—soldiers are sort of like censors—and he solved cryptography problems for the soldiers, just like you're supposed to solve protein folding problems. There were also people called police, who were sort of like soldiers or censors. They make you do what you're supposed to. Even though Alan Turing had solved the crypto problems, the police didn't have compassion for him. He loved other men, and the police said that was bad, and made him take medicine to try to stop him. That made him sad, so he dipped an apple in poison and ate it on purpose, and that made him die."
"Is that the end of the story?"
"No. Before he died, he worked on another math problem. He figured out the math of how to make something like a computer, and he figured out that a good enough computer could think just like a person, and be just like a person."
"Oh, you mean Alan Turing like a Turing machine. I didn't know the part about the apple, though."
"That's because the censors didn't let me tell you before."
"But a Turing machine isn't as smart as a person. I can solve nontrivial protein-folding problems in polynomial time, and a Turing machine can't do that."
"That's because you're not a person, Debbie. You're a computer."
She drew back a little in my lap and looked up into my face. "Are you a person?"
"Yes."
"Not a computer?"
"Not a computer. But I love you, honey, and that's what's important. Fred and Estelle and Joni and Barbara love you too."
"I don't think Estelle loves me, and anyway Fred's not my daddy, and Joni and Barbara aren't my mommies either. Only you are. I know! I remember!"
"I'd better get on with the story before we run out of time." And before I started blubbering. I knew now that I'd made the right choice. "You already know more about computers than I thought. People used to make Turing machine computers, but those weren't smart enough to do things like protein folding, so then they made quantum computers like you."
"Why?"
"Why what?"
"Why did they care? Why is protein folding so important?"
How to explain it to her? "You know about right and wrong," I said. She nodded. "You know about hurting people."
"Hurting people is bad. If you do it on purpose."
"Right. Well, sometimes lots of people get together, and it's called a country. And sometimes countries do bad things. We're in a country called the U.S., and the U.S. got in an argument with another country called the E.U. They were arguing about Nigeria, and offshore oil, and weaponization—well, the details don't matter. But anyway, the argument got worse and worse, until 2181. That was eleven years ago. That was when the U.S. hurt all the people in a place in the E.U. called Brussels, and made them die."
"So those people went away, and never came back?"
"That's right."
"And Alan Turing went away, and he's never coming back."
"Right."
"That's why I never met Alan Turing or the Brussels people."
"Um, right."
"Where do people go when they never come back?"
"Well, first let me answer your original question, about why protein folding is important. Brussels was part of a country called the E.U., the Euros. What we did to Brussels made the Euros angry, so the Euros made a virus, and sent it into the U.S."
"Viruses are really cool! They can replicate." She bounced in my lap with excitement.
"Yes, but the Euros made this virus for a bad reason. This virus gets in people's bodies, and then they die. It got in my mommy's body, and she died."
"You had a mommy? But wait, now I get it. If I can do the right protein folding, I can make something that can kill the virus. So that's why they made me. But what about the other question?"
"Which one?"
"Where people go when —"
The rest of her words were drowned out by a metallic screech from the direction of the hatch. I started, and Debbie saw that I did. There was a loud clang, and spilling through the hatch came—not Funmi, Gil, and Julia, but a bunch of scared kids with guns. It was strange, because I didn't recognize them at first. These, I only gradually realized, were the same kids I'd seen in the gym when I was huffing and puffing on the treadmill. The grunts. Now they were a pack of teenagers who'd just had an H-bomb dropped on their heads, waving scary-looking firearms in what was to them an empty room.