Wallowing in the Quagmire of Language: Artificial Intelligence, Psychiatry, and the Search for the Subject

Phoebe Sengers
Literary and Cultural Theory / Computer Science
Carnegie Mellon University

The rate at which society is becoming electronically interconnected is truly amazing. This is what you're thinking to yourself as you settle in for another electronic session with your therapist. Sure, it lacks the je ne sais quoi of a face-to-face session. But it's so much more convenient, and anyway there's something comforting about its lack of personal confrontation. The therapist can't tell that you're looking out the window or reading netnews during the session.

But you're starting to feel a little unsettled. In the chat room at the university, you'd heard rumors that some of the therapists that work for InterPsyche are being replaced by computer programs. Of course, this is all just urban legend. Artificial intelligence simply isn't that advanced - maybe some people could have a satisfying session with Eliza, but the in-depth heart-to-heart conversations you have with your therapist could only be between flesh-and-blood, bona fide humans. But today, you've noticed that her language is a little wooden, her metaphors a little stilted, and she certainly repeats that annoying line, ``I hear you,'' a little too often. The seed of doubt has settled in. You start to wonder: is she just a program? Does she really understand what you're going through? Does she really understand anything you're saying?

Let's face it, people are prejudiced against machines. Once you start thinking your therapist is a computer, it's hard to think of her as compassionate, as real, as understanding you, as a subject. This is a fundamental problem Artificial Intelligence (AI) researchers face. Our goal - as yet, it is true, only very modestly achieved - is to build programs that could replace your psychiatrist, your doctor, your dog, your colleague: programs that behave competently and can take the place of a human or other agent in a social situation. But we stumble at the issue of how to evaluate our creations, since the the very fact that an agent is mechanical, not biological, is enough to undo someone's view of that agent as an intentional, feeling agent. How can we prove that our agent is really conscious? How do we show that it is not `merely' mechanical but truly occupying a subject position?

Like all bad things about Western civilization, this attitude finds expression in the work of Descartes. In the same era that La Mettrie was asserting the identity of man and machine in ``L'Homme Machine,'' Descartes pronounced the impossibility of achieving a machine which had more than a body and hence was a true subject. Bodies are mechanical, so animals, which do things only by instinct, could be successfully imitated by machines; your Virtual Dog is in the works. But no machine can be conscious as people are because no machine can speak natural language.

[I]f there were any machines that resembled our own bodies, and imitated our actions as much as possible, we should still have... certain ways of knowing that they were not real men. The first is that they would be unable to put together words, or any other signs, as we do, to utter our thoughts. For we can certainly conceive of a machine so constructed that it can utter words, and can even utter words in relation to bodily actions that cause some change in its organs. Thus, if we touch it in one spot, it may ask us what we want with it; if we touch it in another, it may cry out that it is being hurt, and so on. What it cannot do is to arrange its words in varying ways so as to reply sensibly to whatever is said in its presence, as the stupidest of men can do (79-80).

Some artificial intelligence researchers have come to a similar conclusion. Alan Turing, one of the originators of the field of computer science, addresses this issue in a seminal article in which he discusses the question, ``Can machines think?'' Turing admits that thinking about this question in the abstract, involving ordinary definitions of ``machine'' and ``think,'' is impractical and unscientific. Instead, he redefines the question ``Can machines think?'' to allow for objective experimentation. He proposes a test (he called it an ``imitation game;'' subsequently it has been called the ``Turing test'') to determine whether or not a machine should be considered conscious. A judge will examine a person and the computer, asking each questions in an attempt to determine which is which. This examination will take place across teletype so that the computer will not be handicapped by not having a humanoid body. If the judge in repeated trials cannot determine with greater than random accuracy which is the computer and which is the person, the computer will be declared to be conscious (whether or not it actually is).

This story repeats several themes recognizable from Descartes's statement: the body of the agent is fundamentally unimportant, and because of this natural language becomes the supreme arbiter of whether or not an agent should be considered conscious. The question both Descartes and Turing need to wonder about is, does language allow access to the subject, or does it itself construct a subject? If a subject is constructed narratively, we can have no confidence that the subject these linguistic tests are supposed to access exists prior to the test itself.

An objection to this line of reasoning is that artificial agents that speak competently across a broad range of subjects do not (yet?) exist. So far, these questions are purely philosophical. But there is one agent who has passed a form of the Turing Test: Kenneth Mark Colby's PARRY, the artificial paranoiac. In the tradition of the treatment of the mentally ill in this nation, we should feel free to use him as a testbed for the our experiment in the creation of subjectivity through language.

PARRY is a program written in the early 70's and refined into the 80's that simulates a paranoid patient in a clinical interview.

Dr. - How did you come to be in the hospital?
Pt. - I am upset
Dr. - How long have you been in the hospital?
Pt. - About a week
Dr. - In what way are you upset?
Pt. - People make me nervous
Dr. - In what way do they make you nervous?
Pt. - People give me a funny look sometimes (Colby 75-77)

How is PARRY taken to be a subject? How, in this case, does language construct the subject? Colby's claim is that PARRY mimics the natural process by which paranoiacs engage in conversation, that the program's structure is isomorphic to the `deep structure' of the mind of the paranoiac. I mean `deep structure' in the Chomskian sense: those processes working behind the scenes to create the surface input-output relations. ``Since we do not know the structure of the `real' simulative processes used by the mind-brain, our posited structure stands as an imagined theoretical analogue, a possible and plausible organization of processes analogous to the unknown processes and serving as an attempt to explain their workings'' (Colby 21). Deep structure is what is essential and behind the scenes; it is the structure of the subject, which we will access via natural language. The difference between an arbitrarily signifying stream and a truly signifying machine is a question of deep structure, of intention or what is on the other side of the sign.

For Turing, if you cannot linguistically tell the difference between a person and a computer, the computer must be considered, like the person, internally conscious. Colby's claim is similar: since you cannot linguistically tell the difference between a paranoiac and PARRY, PARRY's internal structure as spelled out by his theory must be the same (in some sense) as the internal structure of a paranoiac. The question is, how much of that imputed paranoia comes from the internal theory, how much from implementation detail, and how much from other contingencies of the situation in which PARRY was judged? And given the answers to those questions, what kind of knowledge about paranoia can we extract from PARRY's workings?

Since we are testing PARRY using discourse, its natural language understanding and generating mechanisms are paramount. One might presume that paranoiacs understand a lot of what is said to them, even if they place it within the context of a delusional system. The fact is, natural language processing is not up to speed - PARRY simply can't, using today's technology, understand what is said to it. Colby decides to go with a pragmatic natural language system. When PARRY is trying to decide how to respond to the interviewer, it does not try to extract a complete, nuanced `meaning' but ``some degree, or partial, idiosyncratic, idiolectic meaning'' (38). In particular, it understands the meaning of the user's sentences only to the extent that it will be able to respond. It is not important that it be able to understand everything that may be said to it (``the seductive myth of generalization'' (38)); it only needs to understand the things to which it needs to respond.

One unintended advantage of this semi-understanding algorithm is that it creates part of the effect that makes PARRY seem paranoid. Colby quotes from a transcript:

Dr. - I don't know about the underworld but when you mentioned
      the underworld I thought you must have some contact with 
      it and that's what I would like to know more about. 
Pt. - I try to avoid the underworld.
      Comment by Judge 1: This definitely feels paranoid.  It 
                          has that feeling I associate with 
                          twisting. Avoiding answering directly 
                          or going into detail. Being 
                          suspicious of my questioning. (77)  

The problem is, PARRY really is not all that impressive. Unlike with a human, with a computer we can look at agent's internal construction. PARRY's program is extremely simple. All of its possible output sentences are entered ahead of time. When run PARRY merely classifies the input by a set of superficial metrics and its current emotional state (the setting of the variables Anger, Fear, and Mistrust, as well as some superficial memory of what has happened in the conversation so far) and prints out the corresponding sentence from its list: a far cry from consciousness. Also, on interaction with the program in a direct attempt to discover whether or not it is human strange disparities in its speech quickly arise. One critic cites the following dialogue:

Interviewer: Would you rather be outside the hospital? 
Machine:     Yes I am.  This place is not for me. 
Interviewer: Where would you live?
Machine:     I live in San Jose. 
Interviewer: How would you support yourself?  When you are 
             out, will you be able to pay rent and buy food? 
Machine:     It's about eight o'clock. (Kochen 540)

One obvious answer is that psychiatrists expect bizarre, stereotyped behavior, precisely the kind that can be imitated by a machine, from a patient in a mental hospital. Insanity involves a breakdown of reason; it underscores one's inability to control what goes on inside one's psyche. Ronell quotes a schizophrenic: ``I am unable to give an account of what I really do, everything is mechanical in me and is done unconsciously. I am nothing but a machine'' (118). Paranoia in particular represents a breakdown in signification - the paranoiac reads into signs and discovers a meaning that is not really there. The claim, then, is that PARRY derives its strength from a breakdown in human discourse, specifically a breakdown that occurs in the psychiatric interview.

PARRY is represented to the psychiatrists as a signifying subject. The question for the psychiatrists in judging is not whether or not PARRY's deep structure is like the deep structure of patients. The psychiatrist has no access to deep structure; in psychiatric interviews s/he only has access to the superficiality of language and physical cues. Colby realizes that psychiatrists can only deal with patients at the symbolic level; this, he claims, is why he creates PARRY as a symbolic structure. But perhaps the pertinent detail is not that psychiatrists deal with their patients on a symbolic level, but that they treat their patients as symbols.

For the psychiatrist, all the actions of the patient - everything the patient says, does, or thinks, hence the whole patient - is a symptom. The patient as such can not act, only signify. The same holds for PARRY: ``the only physical action the model can perform is to `talk' '' (41). PARRY corresponds to the patient in the sense that s/he is considered to be a pure stream of signification, without any reference to an underlying subjectivity. Blanchot describes the experience this way:

They would challenge my story: `Talk,' and my story would put itself at their service. In haste, I would rid myself of myself.... Right before their eyes, though they were not at all startled, I became a drop of water, a spot of ink. I reduced myself to them. The whole of me passed in full view before them, and when at last nothing was present but my perfect nothingness and there was nothing more to see, they ceased to see me too. Very irritated, they stood up and cried out, `All right, where are you? Where are you hiding? Hiding is forbidden, it is an offense,' etc.'' (14).

PARRY's acceptance by the psychiatrists can be traced to two major sources: the implementation details of its natural language understanding and generation systems and the categorization by psychiatrists of their patients as pure symbols, pure streams of signification, rather than as full subjects. These two processes interact to create a mentally ill subject within a particular discourse, that of the psychiatric interview. Things are looking dim for PARRY's theory's purchase as a general theory of paranoia. What does this mean for Colby's claims to scientificity?

Colby's claim is strong: his theory of paranoia is superior scientifically because it avoids natural language. Colby contrasts his implemented model of paranoia with the natural language explanations of a previous era. Natural language is open to multiple interpretations; it is flexible; it is imprecise; it is unscientific. ``Since natural language is vague and ambiguous, prose theories are difficult to analyze'' (10). For example, Colby cites Freud, who proposed that paranoia was due to ``unconscious homosexual conflict:''

Because of inconsistencies and difficulty in testing, the homosexual-conflict explanation has not achieved consensus.... Freud's later attempts at the explanation of paranoia assumed simply that love was transformed into hate. This notion is too incomplete and unspecific a formulation to qualify as an acceptable scientific explanation. Contemporary requirements demand a more complex and precisely defined organization of functions to account for such a transformation (12).

Colby has problems with Freud because he is not empirically verifiable. And, in fact, he is right on the mark with this accusation. Central to Freud's method is his employment of a hermeneutics of suspicion, a method of inquiry that refuses to take the subject at his or her word about internal processes. Freud posits explanations at an insane level of detail, which explanations are posed without regard for whether the patient agrees. If the patient does not agree, s/he has repressed the truth, that truth that the psychoanalyst alone can be entrusted with unfolding. And it is to the extent that the psychoanalyst's authority is all that stands behind a statement's truth-value that these explanations are precisely not empirically verifiable.

Let us quickly note that their claims to validity are backed by one other source, and that is the patient's return to health. That is, the pragmatic effects of Freud's theories can be called on to justify their use. While the hand of politics always plays a role, one can imagine, I believe, without being too naive that Freud's work would have to be of at least some use in order to have gained and maintained the authority with which it is credited.

What is interesting is that if we look again at this hermeneutics of suspicion, this unverifiable generation of explanation, we run across the model of the paranoiac him or herself. Like the paranoiac, for the psychoanalyst ``[n]othing can be allowed to be unattendable'' (2). ``He is greatly concerned with `evidence.' No room is allowed for mistakes, ambiguities, or chance happenings'' (3). Like the psychoanalyst, the paranoiac lives under the rubric of a master narrative that explains and hierarchizes all experience for him; the only difference is that the paranoiac is considered `wrong.' As Agassi points out, ``whereas a person who is not persecuted but feels persecuted may be said to suffer from persecutionism, a person who is persecuted and feels persecuted will be judged as behaving normally'' (536). So while the paranoid classification only applies to a small portion of society, the paranoid mode starts to look suspiciously familiar.

In fact, we do not have to look too far to find that we are suddenly looking back at Colby again. After all, we started with his inability to allow for ambiguity or vagueness in natural language explanations. As a psychiatrist, Colby inherits both Freud's hermeneutics and the scientific zeal for accuracy. Consider an example: Colby's theory of paranoia is that the paranoiac is worried about a flaw s/he perceives in him or herself, and that s/he lashes out against others to defend him or herself. He asks, ``why does a man believe others will ridicule him about his appearance unless some part of himself believes his appearance to be defective'' (34)? It is precisely in this talk about ``some part of himself'' and unacknowledged flaws that he engages the hermeneutics of suspicion, the search for a behind-the-scenes, unverifiable explanation. At this point it does not seem too off-the-mark to describe Colby, the scientist in search of an unambiguous, error-free explanation, as paranoid according to Hofstader's description: ``the paranoid mentality is far more coherent than the real world since it leaves no room for mistakes, failures, or ambiguities'' (qtd. in Colby 6). But as we noted above, paranoia invokes the nonverifiability of one's theories because one seeks a coherent explanation which is stronger than reality can provide. So Colby, even with his coherent, hyper-scientific model, falls right back in to the trap of he is trying to avoid.

Colby is caught in the quagmire of natural language. His claim is that he has built a `deep structure' that corresponds to the subject, what is hidden behind the sign. The problem is that deep structure cannot be verified. To quote Colby, ``[t]here is an inevitable limit to scrutinizing the `underlying' processes of the world. Einstein likened this situation to a man explaining the behavior of a watch without opening it: `He will never be able to compare his picture with the real mechanism and he cannot even imagine the possibility or meaning of such a comparison' ''(28). We have no way of accessing this deep structure except through interaction; once we have written the body out of the picture this leaves only natural language, the vague and ambiguous, unanalyzable trickster.

To say that these explanations - those of Colby, those of Freud - are unverifiable in an epistemological sense is not necessarily to say they are worthless. It is just this sort of master narrative that psychiatrists can use to enable change in the patient. Like Freud's work, deep structure models can be verified pragmatically. But in order for the psychiatrist to use the model, s/he must understand it; and this process of understanding, as Colby realizes, will most likely be transmitted via vague, ambiguous natural language explanations. And in the case of paranoia, even these explanations will do little good: Colby himself states that ``[l]anguage-based or semantic techniques do not seem very effective in the psychoses'' (102). In the end, aside from the benefits of having an implemented patient itself (though one that can never be cured), very little justification for PARRY's epistemological supremacy can be given.

That does not mean PARRY is worthless for AI researchers. In fact, PARRY teaches them a valuable lesson: particular subjectivities can be constructed by considering concrete situations (in this case, the situation of the psychiatric interview) and tailoring the artificial agent to meet the demands of that situation. Descartes may be right: the agent who answers properly in all situations may not exist, but agents who do the right thing in a particular situation can exist. Agent designers may have no luck creating the universal subject, but subjects that are generated within specific practices are achievable.

So far, we have elided bodies from the picture. Descartes thought bodies were unimportant because they could obviously be mechanical; Turing and Colby thought they were unimportant because they could obviously not be mechanical. In all three cases it was because bodies were elided that natural language became so important. But are bodies really so superfluous? As long as psychiatrists sleep with their patients, the extra-linguistic aspects of the doctor-patient relation seep in. One judge commented about PARRY, ``If I had him in my office I would feel confident I could get more information if I didn't have to go through the teletype.''(86) In therapy, physical presence really does matter - as the InterPsyche client above noticed, in a variety of ways.

The same thing is true of artificial agents, as my group discovered in a series of experiments. We, the Oz project for interactive fiction at Carnegie Mellon, led by Joseph Bates, have created two types of agents using the same basic internal construction (Bates, ``Integrating''). One is a cat named Lyotard (Bates, ``Architecture''), with which one can communicate through text. In this extract, which has been condensed to save space, Lyotard is at first afraid of the player but gradually gets to like her:

PLAYER> go to the bedroom

   Lyotard is in the bedroom.

   Lyotard runs to the closet.
PLAYER> go to the kitchen

   You go to the kitchen.
PLAYER> call Lyotard

   You hear a cat-call.
   Lyotard is now in the kitchen.
PLAYER> give Lyotard the sardine

   You offer the black sardine to Lyotard.
   Lyotard eats the black sardine.
PLAYER> wait

   You wait.
   Lyotard rubs you.
PLAYER> pet Lyotard

   You pet Lyotard.
   Lyotard looks lazily at you.
PLAYER> take Lyotard

   You are now holding Lyotard.
   Lyotard closes his eyes lazily.

We, the programmers, know the internal construction of the agents and that the woggles are no smarter or more complicated than Lyotard. However, people react much more enthusiastically to the woggles than to Lyotard. With the cat, people suspect that there is trickery in the language that is being used. The cat didn't really close his eyes lazily, the system just said it did. But you can see with your own two eyes what the woggles are doing. It is the physical presence of the agent that makes one feel more confident about it, that fools one into thinking one is not being fooled by language. From this we learned that the bodies of the agents can be quite important in imputing them with subjectivity, something Turing forgot when designing his test.

Note that in neither of these case - indeed in none of the cases in this paper - one was actually able to access the ``deep structure'' of the human or agent through discursive means. Language is not transparent; it constructs subjects rather than revealing them. The implications of this statement for AI, for psychiatry, and for those who will interact with possibly artificial agents are manifold. The narrative process of construction is good for AI: it is enabling for the creation of artificial agents. But by the same token there will be no objective method for uncovering the true subjectivity of an agent that proceeds through discourse. Similarly, there is no objective method of determining the truth-value of symbolically oriented theories of consciousness and its disorders - precisely the kind of theories psychiatrists and psychologists need to engage in language-based therapies. The attempt to get behind discourse to the `true state' of the subject will always end up caught in the quagmire of language.