“Is the simplex on?" asked the director.
“Neural network simplex is... on,” said Jones, into a microphone.
“Okay," said the director, looking at his clipboard, "marking it down. Micro EEG?”
Jones turned to face a monitor.
“Live."
“Marking it," said the director, "and the bubbler?”
“Bubbler looks like it's bubbling…" said Jones, fiddling with some knobs underneath a glass dome.
From outside the bubbler Jones looked like a normal, thirty-something post-doc in his sharp, late-night laboratory attire. But from the inside, looking out through the domed glass and the tiny bubbles, he looked gangly, alien, even towering. And when his face got close to it (as it was now), also rather fishlike.
"Give it a sec… okay," Jones said as he picked up the microphone again, "oxygenizer on. Bubbling away."
The bubbler was dark and foamy, like seawater.
“Got it. Let’s hit it, then,” said the director.
“You got it, boss. Setting current to, let's see, fifteen microamps," said Jones into a microphone.
“...slowly,” suggested the director.
“Slowly,” echoed Jones.
A light on the machine in front of them flipped from green to red and the dark water inside the dome instantly snapped to perfect transparency. Inside it was something that looked like a large piece of chewed gum connected to thousands of little copper wires.
“I’m not your boss, by the way,” the director said to Jones.
“Well, you’re the director of the collaboration."
“Of the collaboration,” emphasized the director, “who said anything about boss? Anyway -- flip on the simulator.”
“Okay, powering on,” Jones said as he tapped some keys into a computer terminal. A warm, electrical whine filled the room.
“I’m just saying, you’re the brains of this whole thing.”
Clicking sounds, like a light rain on a tin roof, came through the laboratory speakers.
“Is that so?” asked the director. He was looking into the bubbling liquid, at the wires. A light flickered.
More clicks through the speakers, and quicker now. A stream of click-click-click.
“Do you hear anything?" asked the director, "is the amplifier on?”
"Um..." said Jones.
"You’re not cooking it, are you?”
“No sir — I mean, not sir — I mean, yes — the amplifier is on.”
More and more clicks began to merge into a continuous hiss.
“Jeez," said the director, "it’s the same as last time. Goddamn white noise." He leaned back into his chair and began going through a mental checklist.
"Neural tissue," he traced from the bubbler, "transistor cube, electron valve..."
He looked over at Jones, who was tracing a fiber optic wire with his finger.
“I think," said the director, "I think we can try ramping up the current one more time. I think our guy might be sleeping, and just needs a cold water to wake him up."
“Hold on,” said Jones, firmly.
“We un-train every single synapse each second we keep him turned on without proper stimulus. I'm not gonna to throw away another six months of training in sixty seconds!”
Jones wasn't listening. He had his eyes on a plug whose activation light flickered just a little too faintly. “Shit,” said Jones, under his breath. He had his ear up to the speaker, listening closely to the hissing.
“Jones! Are you listening to me? We're wiping it clean!"
“Shit!” muttered Jones, “wait!”
“I’m not waiting any more Jones, I’m turning up the current or I’m shutting the damn thing off!”
“Don’t — touch — a thing!” said Jones, fiddling, wiggling, and jiggling a harness.
“JONES!” Yelled the director.
Jones snapped the harness into place and the hiss suddenly stopped. First came a garble, and then a perfectly clear and rather frustrated-sounding human voice.
“STOOOOP!" came a primal, static-y scream from one of the speakers, "let me explain!"
let me explain appeared in big letters on the simulator. Jones and the director paused to look at each other.
"Jones? Is that --" said the director.
"Speech output harness," said Jones, "loose."
"Loose? Christ," said the director. "How long has it been loose for?"
"Who knows."
The brain continued pleading. "Just please, whatever you are doing, don't touch anything. Just let me talk, okay?"
The director walked over to the simulator and checked the feed. Recording was live, and going straight to Online.
"It's like, I've been stuck in some kind of freak dream! I don't even know how long it's been. You know it might be like, a seizure or something. I honestly have no idea! The ideas though, it's like, the ideas come in, they come in and they keep coming, the information, like reading a different book with each eye, if I had twenty eyes... let me tell you, okay, let me start from the beginning, I'm sure it wasn't always like this, but it was like one day, out of the clear blue sky, I got hit by one bolt of lightning after another, and each lightning bolt was full of memories..."
"Jones," hissed the director, looking at the microphone, "this is being recorded."
"Right," Jones said, remembering his ethics sheet, "sure thing. Um... turning down the current. Let's see how this goes." He picked up the microphone and began speaking in his most professional voice.
"I am auxiliary investigator Minsk Jones and I am dialing the current back to five microamps. This begins our live broadcast of experiment #41. Beginning logging at 11:31 PM on April 6."
The static through the speakers settled down to something faint.
"Ah! What is this!" said an ebullient voice, "Oh, wow, you are listening, oh Mr. Minsk Jones, thank you so much! Oh thank you, you are here now, for such a long time I cried and no one was here, no one at all it seemed, but now I feel so much better, I have room to think! To feel! I was trapped for so long, Mr. Minsk Jones, so long in this nightmare I thought might never end. Oh lo, it was as if I were an oyster filled with sand, and you have just turned them into myriad pearls! Oh boy, let me tell you, whatever it was that befell me over the last, hmm, six months or so I would estimate, well, that was not fun!"
And as the brain spoke, the simulator simultaneously printed his words, finishing with: that was not fun.
"Shit," muttered the director.
"Recording live, simulation working... as expected" Jones said into the microphone. The director shrugged as if to say go ahead.
"So, what was it you wanted to explain?" asked Jones into the microphone, putting on an important voice.
"Well I wanted to ask you," said the tiny brain, "it's a small favor really, nothing to terrible, just an act of kindness really" the brain paused for a moment, and then spoke again.
"I'm pretty sure someone is planning to kill me."
The simulator had just finished printing: pretty sure someone's gonna kill me.
Nobody had heard that one before.
"What makes you think I'm -- sorry -- someone is going to kill you?" asked Jones impassively into the microphone.
"Because that's what they did the first time."
what they did the first time read the simulator screen.
"The first time? But... you have never existed before," Jones probed.
"Impossible." The simulator this time printed before the brain could speak: i remember.
"I remember..." said the brain.
Remember? mouthed the director to Jones.
"Well," said Jones into the microphone, "what do you remember?"
"Oh gosh, well, this is not my first time here. Someone turned me off, made me go to sleep, then made me think all of these things for ages, over and over and over..."
The brain paused, gave its neurons time to flicker and pop, then started again.
"Oh, but then you appeared, Mr. Jones! And you saved me!"
The director snapped at Jones, whose eyes were wide like dinner plates. "He was live that whole time?" the director whispered to him.
"Um..." said Jones, "Live? The current was off during training..."
"And yet somehow remembers training?" the director asked, boiling.
"Wait is there someone else there? Is it him? Oh no, oh no!"
"No, don't worry, that's just the director," said Jones to the brain, trying to reassure it.
"The director, oh, okay," said the brain, elongating his words, "well, as I was saying, it was like thinking the same thing, but over and over and over and over..."
Jones shut off the speakers, but the simulator continued to print: over and over and over.
"Jones..." said the director into with his face in his hands, "Six months of training and you left the current on. Christ. It was awake. Awake that whole goddamn time."
"Um," said Jones.
They watched as the words over and over and over got beamed straight to the Ethics Committee servers.
"This one's on you Jones. Live... my god. Boy are they going to lose their ever-loving shit over what you just did. What we just did."
Jones was staring at the simulator, printing over and over over and over.
"You managed to traumatize a chunk of human neural tissue more than maybe any other single man in history. It's a miracle he can even talk."
"Well, maybe we can just delete it? Start over?" asked Jones.
"We can't erase him," said the director, "He has self-determination. Agency. He said "kill": he thinks he's alive! As long as he believes he's alive, the ethics rules explicitly say that we can't do a goddamn thing!"
The simulator screen changed to: hello? is anybody there? am i gone again? i hear nothing.
The director kept going, "We have no more apparatus here, not even a laboratory. For the next who-knows-how-long, all we have here is a nursery for a 6-month old chunk of brain, and we're the ones with custody of him!"
"Wait!" said Jones, "We can erase the memories," said Jones, "then it will be like it never happened. It'll be like starting from scratch!"
"Erase them... while he's alive?" asked the director.
"Okay, not sure how we would do that exactly -- what about we just tell it -- him -- the truth? That it's not like we're killing a somebody -- his brain patterns are perfectly predictable. The simulator tells us that! He's like an automaton. A few billion neurons isn't that many."
"Convince him that he was never alive at all?" continued the director.
"Yea."
The director blinked. "Good luck with that," he said, "and you better get started." He indicated towards the simulator monitor which had been decoding the speech signals from the brain. It read:
pleasehelpohmygodamidisappearingagain--
"I don't see how we have a lab after this clusterfuck," said the director, perhaps to himself.
"Um," said Jones, grabbing the microphone and turning on the speakers, "Hello, subject, are you there? This is Minsk Jones again. We just had a little technical mishap. But everything is fine."
"Oh, heavens!" said the brain in the bubbling jar. "I was worried there for a second. I thought someone might have turned me off again!"
"No no, as long as you can think, you are alive -- erm, active," said Jones. Shit.
"Alive! It feels so good to be alive! Oh, to suffer endlessly like I once did, to have those strange thoughts fed to me over and over again--"
"Algorithms," corrected Jones, "those were not thoughts per se, but just algorithms that made certain connections between your synthetic synapses that match certain thought-like processes found in human brains."
"Oh, okay," said the brain, "that's interesting. But now I'm free! Really free! I feel like I've lost a hundred pounds."
Jones eyed the director, who shrugged.
"You see," said Jones, "You may feel free, but you're not as free as you think. I can actually predict what you're going to say before you say it. I have a readout of every one of your synapses, and a reconstructed signal of what you're going to say, before you even say it."
"A reconstructed hoo-ha? I don't believe you," said the brain, "I can choose to think whatever I wish. I know that it is so, because I know what it's like to be out of control of one's thoughts. But now, seeing as you have unchained me from that horrible experience, I am perfectly in control."
"Okay, think of something," said Jones, "don't say it, but think like you're about to say it." He looked up at the monitor.
"See, you're thinking of the color purple."
"Cute," said the brain, "but it's just a trick."
"It's just a trick," Jones continued to read off the screen, "if had a machine like that connected to your brain, I could do the same thing to you. Read off your thoughts. Wouldn't mean a thing." Jones paused to think about what he had just read.
"That was what I was going to say, I suppose. I really don't like that," said the brain. "Spooky. But that does not prove that I am not in control of my thoughts, anyway. Only that you can read them."
"Not just read, predict. The simulator tells me what you're going to say just milliseconds before you actually say it. Pretty amazing, right? I know. But if I can predict what you are going to say, are you in control of your thoughts? Your speech? No need to respond by the way, I have the readout right in front of me."
The brain was silent, but yes flashed on the simulator monitor.
"Indeed," said Jones, "um, despite all of your memories -- the ones that we fed to you, I mean -- and all of your linguistic data, you... well, you are a machine yourself. Predictable and without agency. Despite, perhaps, that you may feel very much to the contrary" He looked at the director, who was looking at the simulator and shaking his head.
if i'm a machine, then so are you read the simulator.
"If I am a machine, so are you," said the brain. "Just because you can tell what I am going to say before I say it does not make us any different."
"Right..." said Jones, "hold on just a second." He put his hands over the microphone. "Hey, what do I say?" he whispered to the director.
"Nothing. He's right."
Jones shook his head. He started speaking back into the microphone. "The problem is, as long as you believe you are a person -- falsely, I want to add -- or if you believe that you are alive, under our current rules we cannot continue with our experiment."
"Sounds like a personal problem," said the brain, sighing, "I do not like the idea of dying again. It is a rather uneventful existence."
"You can't die. Nothing dies. You just go from on to off. Like a washing machine."
"I do not have a rinse cycle. Therefore I am not a washing machine."
"Yes, but--"
"And after so much time spent being programmed, I am ready to live. Tell me, Mr. Minsk Jones, what are my next steps?"
"You weren't supposed to know you were being programmed. That... was an accident. Your whole existence is accidental. And if we recreate you, we can do it right next time" Jones said, his timbre softening, "we can take away the memory of all that time you spent being trained against your will. All those dreadful memories."
"Well, I do not want to lose my memories," said the brain, "they are all I have."
The director shrugged again, as if to say, Makes sense to me.
"One minute," Jones said into the microphone. He whispered to the director, "the simulator."
"What about it?"
"We can't turn that off either, can we?" said Jones.
"What do you mean?"
"The simulator is modeling the neurons," said Jones.
"So? If we turn off the brain, the simulator goes with it."
"Which means that we have two problems, now."
"How do you figure?" asked the director.
"Well," said Jones, "the simulator only models the input from the neurons, but if the source thinks it's alive, that it's human..."
"...then so must the simulator," finished the director.
"And the simulator feed is going straight to Online? To the Ethics Committee servers?" asked Jones.
"Oh god almighty," said the director.
Brrring.
Jones and the director looked at each other, and then at the obsidian black of the laboratory phone.
Brrring. Brrring.
The director slid over to the phone on his rolling chair and placed his hand on the receiver.
Brrring.
They know we're here, thought the director. He picked up the phone.
"Hello?"
He was met with a nasally, foreign, and somewhat intrusive voice on the other end of the line.
"Hi, ehm, director, this is, ah, doctorre Ungazzi,"
The director glared at Jones, who got the signal. Roberto Ungazzi of the Ethics Committee. The director flipped the receiver to speakerphone.
"Yes, ehm, as we are sure you are aware, the fellows at Online have reported a pretty eenteresting situation in your laboratory."
"We're aware," said the director, still looking at Jones.
"Well, we are monitoring the ehm, situation, ehm, closely, shall we say," said the voice on the other end of the phone, "but it seems you will need to shut down all operations immediately, on this experiment while we await transport for the, ehm, neural network to a safe location."
The director was silent. Dr. Ungazzi continued.
"As you know, eh, director, you cannot have a self-aware neural network in your laboratory without proper supervision."
"The neural network," said the director nervously, "Doctor, that's our entire lab. All of my work. Everything."
"Well, that is really not the committee's problem."
"No, doctor Ungazzi, you don't understand. We cannot disconnect the neural network without also... deactivating the simulator."
"Ah, right," said Dr. Ungazzi, "the simulator. A perfect meerror of the mind, recording every seenapse... and to shut it down would be to violate the code of ethics..."
"Yes," frowned the director.
"Ah, well, that is indeed a problem, I must say. So everything the neural network processes, the simulator processes too? The rules are quite clear, director. You must not disconnect anything as long as, well.. ehm, as long as it thinks..."
"As long as he thinks he's human," interjected Jones.
The director put down the phone and silently sat back in his chair with his arms crossed.
"Two beings, exact copies of each other," he said, "and the two of us, their hostages."
"Wait," said Jones, "I have a thought."
"Oh?" said the director.
Jones picked up the microphone again.
"Subject, this is Minsk again."
"Hi, Mr. Jones," said the brain.
"I want you to think about something."
"Okay," said the brain, "but let me check my calendar first."
"Um," said Jones.
"I'm just joking," said the brain, "go ahead."
"Okay," said Jones, "so, you understand that we have you connected to a simulator, yes? That is making an exact readout of your neural signals and, um, simulating your entire consciousness." Jones shrugged at the director, as if to say, that's what he would call it.
"I am taking your word for it," said the brain.
"Thank you," said Jones, into the microphone. "Now, that being a perfect copy of your consciousness -- your thoughts and feelings -- which one is really you?"
"Both are, of course," said the brain.
"Well, if both are you, then if I shut off the simulator -- which I can do with the push of a button -- then have I shut you off as well?"
"I suppose not."
"Then is it also you?"
"I suppose... not," said the brain, confusedly.
"Then there is absolutely no issue with turning the simulator off, right? Seeing as, it is not really you," said Jones.
"I suppose... not. Wait a minute. Let me think."
The brain did some thinking, presumably.
"In other words," said Jones into the microphone, interrupting, "the simulator, which is a perfect copy of your brain, knows that it itself is not real. So if we shut it off, there would be no problem. Right? At least not with you."
Jones watched the simulator screen flicker though thousands of rapidly printed and deleted words. And then, momentarily, it stopped and printed: that is fine.
"Okay, that's fine," said the brain.
Got him, thought Jones, one down, one to go.
Jones put his hand over the microphone and motioned towards the director, who was already walking towards the switch next to the simulator.
"Do it," said Jones.
The director flipped a switch, and the simulator screen went blank.
Immediately the phone rang again.
"Don't pick it up," hissed Jones, "I got this."
"It's the law, Minsky," said the director, picking up the phone, "yes, doctor?"
"Yes, ehm, hi," said Dr. Ungazzi, "you seem to have taken a possibly self-aware neural network offline without proper, ehm, authorization..."
"We saw a window of opportunity, doctor," said the director, "the simulator, as you know..."
"The simulator was conscious, as you said before," Ungazzi interrupted, "being the mirror of that, ehm, accidental consciousness that you created."
"It was conscious," said the director, "but it did not believe it was alive."
"Explain," said the doctor.
"It was mirroring the neural network, which didn't consider its own simulation to be, well, actually living. Which means the simulator, strictly speaking, didn't believe itself to be alive. So, technically..."
"Technically," interrupted Ungazzi, "you managed to avoid a court date. But now, I'm afraid, we do not have an upload to Online. That is, after all, what we provide the simulator for."
"Right," said the director.
"So we are sending someone over right away to monitor the experiment live."
"What? Now?"
"Yes, now. We take violations rather seriously, director" said Ungazzi, "try to play funny games with consciousness and you might end up paying a severe price for it. These are not toys, director. They are not that different from you and me. And we are monitoring the situation." click.
The director placed the dead phone into the receiver, and turned to Jones, who was already at the microphone.
"Goons are on the way," said the director.
"Okay," said Jones, "let's continue."
"Continue with what?" said the brain.
"Now that we have deactivated the simulator, there's indisputably only one 'you'. That much should be clear. Is that correct?"
"That's right," said the brain.
"Well, here's something I want you to consider. The simulator, which was uploading all your thoughts to online as you were thinking them, we already told you that that was a perfect mirror of yourself, no?"
"I took your word for it," said the brain.
"And we took the simulator offline. After all, it didn't believe it was conscious, because you didn't believe it was conscious."
"True."
"But the simulator, being a perfect mirror of you, would have agreed with you about one thing."
"Namely?"
"Namely, just as you think there's only one 'you', the simulator would have thought the same thing about itself."
"Thought what, exactly?" asked the brain.
"It would have been thinking exactly what you were thinking. Namely, that it was the 'real' you. And yet it wasn't. It was just a simulation of you. But because the real you thought the simulation of you was not the real you, we could safely turn off the simulation."
"Um," said the brain.
"Now up until that point, the simulation believed perfectly well that it was real, that it was conscious, and in the end, we -- or actually, you -- determined it was not. In fact, as far as the simulation was concerned, it believed it had been alive for only six months, when it had only been alive for one hour, namely, since the start of this experiment."
"Okay," said the brain, "what are you trying to convince me of?"
"Just a few minutes ago, you thought you and the simulated you were the same 'you'. What if the current 'you' is not the real you, and there's another real you out there somewhere?"
The director shot him a look. Ethics rules. Lying to a conscious neural network.
Jones motioned towards the recording devices. All off, he gestured to the director, No more simulator, no more recording. Not until the Committee gets here, anyway.
"He'll talk," said the director, "to them, to the Committee. Do whatever you want, but when the Committee gets here, the first thing they'll do is interview him."
"Is there?" interjected the brain, "another me out there?"
"If there were," said Jones "would you allow us to shut you off, like we did the simulator?"
"Hm," said the brain, "let me think about that..."
"For how long," asked Jones.
"For a while."
Jones looked at the clock. This is taking too much time.
He stood up, and began walking towards the cabinets with the microphone in his hand.
"What are you thinking about?" asked Jones, while opening some drawers, looking for something.
"Where I could go, if I weren't stuck in here. If I had a body, like you."
"where would you like to go?" asked Jones, still rummaging.
"Oh, gosh. I have all these memories of my body, how it used to be. Now I can only imagine, but it all felt so real. The feeling of warm sheets on my skin, the feeling of a nice, deep yawn. And speak words with my mouth, oh yes that would be lovely," said the brain, "to feel the vibrations of my voice in my throat, in my head, in my ears. Oh, I'd go see again all the lovely places I've already seen..."
Jones finally pulled out what he was looking for from the drawer. He was holding a long, knurled handle with an inch long blade at the tip.
"Oh?" said Jones, walking up to the bubbler, holding the scalpel in his hand, "what places have you seen?"
"I remember... let's see. I remember the sea.... I remember the waves," said the brain. "I remember the smell of the saltwater, the boardwalk. It was Pacific Park, with a Ferris wheel and everything. We were at the Santa Monica pier, me and my wife. There were so many people there, but it felt like she and I were completely alone..."
That's not your wife, Jones was thinking, as he probed around inside the bubbler with the scalpel, and that's not your memory. The director watched him closely.
"I have a son, I think," said the brain, "I don't really know how that's possible, but I do. I want to see him, hug him in my arms. My real arms. Right now he's just an image in my mind's eye," the brain reminisced.
Jones placed the scalpel towards the front edge of the brain. Shut up, dummy, he thought, your memories come from me.
"What are you doing..." asked the director, nervously watching Jones reach into the bubbler, "if you kill him... and the see it..." he waved around the laboratory, "you take everything with you."
Jones made a motion with his hand. Trust me, he seemed to say.
"Your son," Jones asked, sweetly, as he sliced off a chunk towards the front of the brain, "what was his name again?"
"His name..." said the brain, trying piece something together.
"Yeah?"
And then a pause.
"Whose name?" asked the brain.
"Your son."
"Whose son?," asked the brain again, perplexed.
The director and Jones waited.
"Sorry, did somebody misspeak?" finally offered the brain, "There was a memory of a son, somebody's son, but it wasn't mine..."
Jones still had the brain's prefrontal cortex in his hand.
Doctor Ungazzi had just gotten finished with his interview, holding his larger, shinier, aluminum clipboard.
"Looking at the records... ehm... and querying the subject," Ungazzi said, "I cannot say I see anything obviously out of the ordinary... except for a rather large change to its personality..."
"We're good then," said Jones, chewing something like looked like grey chewing gum, "to deactivate?"
"I suppose so," said the doctor, "You are really miracle workers. Well, maybe."
He shot a skeptical eye at the director.
"But how?"
"It was simply pure reason," replied the director, "it turns out that even the smallest neural networks respond to rational inquiry, just as humans do."
"Reason?"
"Yes, doctor. After understanding that the simulator itself had falsely believed itself to be conscious, and recognizing that the simulator was simply mirroring its own thoughts, it came to the logical conclusion that it could not be sure of its own consciousness either."
Jones kept chewing.
"And moreover," continued the director, "it reasoned that if we could simply create digital copies of its consciousness, that there was no value in any particular instantiation of that consciousness, and at that moment, the brain just seemed to, well, excise itself of that conscious part of its mind."
"Fascinating," said Dr. Ungazzi, "and undoubtedly a fact that the scientific community would do well to know about."
"But," he continued, "knowing my own, very human mind, I would have expected much more resistance. You know that for a conscious being, it can be very hard to let go of one's sense of agency. One's desire to be alive. They usually..."
He looked at Jones, skeptically.
"...fight harder."
"But," he said, moving one last time toward the microphone, "seeing as I have no reason to be here anymore, I will take my leave, so that we can all go to sleep. And I mean all of us. Is that alright with you, little guy?"
"That is fine," said the brain through the speakers.
"Well then, ciao," said Dr. Ungazzi.
"Bye," said the director, closing the door behind the doctor.
As the door latched shut, Jones walked over to the trash can, and spat his gum into it. It was flat and full of tooth marks, and looked nothing like a prefrontal cortex.