[Originally appeared in _Hotwired_, Sept. 1995, at http://www.hotwired.com/special/pornscare/godwin.html] "JournoPorn: Dissection of the Time Scandal: The Shoddy Article" Mike Godwin picks a fight with Time, a major perpetrator of the Great Internet Sex Panic of 1995 By Mike Godwin I don't know when I first heard the name Martin Rimm. I first remember hearing it last fall, when I got involved in the censorship battle at Carnegie-Mellon University in Pittsburgh. As censorship battles at universities go, the CMU fight didn't seem terribly different - it followed the normal pattern: administrators discover that, horrors, there is sexual content on the Internet, and, in a combination of disapproval and fear of publicity, they leap into a crackdown, often cloaking their censorship motives in terms of fear of legal liability. But there were two aspects of the case that made it a bit different. The first was CMU's prominence as a networked university - in its ubiquity of connections to the Internet and its plethora of computer resources freely available to its students, CMU is second only to MIT (and many at Carnegie-Mellon would claim that it's MIT who's in second place). The second was that, in this case, the triggering event seemed to be an undergraduate research project on, of all things, pornography on the Net. Based on images he'd encountered on Usenet and a superficial understanding of the law of obscenity, Rimm, according to my sources at CMU, informed the administration that its systems were carrying material likely to be found obscene. Furthermore, he was reported to have told them, now that they knew about the material, they couldn't claim a standard defense in obscenity cases: that the administration lacked "scienter" (a legal term meaning something like "guilty knowledge"). Since CMU had been put on notice, I was told by more than one source, it had to act. And so CMU decided to announce it was cutting all alt.sex.* newsgroups and most of the alt.binaries.* newsgroups as well. The story of the CMU censorship debate has been told in many places, but only Time magazine's report of the story focused on Rimm's alleged role in triggering the abortive attempt at summary censorship. It was a role whose details, at least, he continues to dispute: in email to me, he took pains to tell me he opposed the action CMU had taken, and he urged that we meet when I visited CMU to attend a freedom-of-speech rally. He told me we had corresponded in the past - I didn't remember it, but, then, with the volume of email I handle, it is certainly possible we'd had prior contact. Although I posted email to Rimm telling him where I'd be during my visit to CMU, he made no contact with me during my visit, which centered on my meeting privately with CMU administrators, then my giving a speech at the student rally. (An excerpted version of my speech was later published in Wired.) I can't say I thought much more about Rimm at the time. There was something that smelled a bit goofy about his research project and the weird seriousness with which he was pitching it to me. His faculty adviser, Marvin Sirbu, actually wrote me independently to suggest that EFF sponsor the Rimm project in some way. But EFF doesn't normally sponsor this sort of project, and my instincts told me we should keep our distance. That instinctive reaction was only bolstered when a contact at CMU sent me an draft abstract of the Rimm study. It was the kind of scientific abstract that, for me, raised a lot of questions. These questions troubled me not because I'm a lawyer concerned with free speech on the Net so much as because - once upon a time - I had planned to be a research psychologist and had devoted serious time to studying research methodology and statistics. I'd even managed to win a graduate fellowship to pursue a doctorate in the University of Texas at Austin's experimental-psych program; it was then I discovered that, although I liked knowing science, I didn't much like doing it. So I altered my plans for graduate study - first to English literature, and then, after a few years as a journalist, computer consultant, and slacker in Austin, to law. But even with all the changes in plans, I never lost my head for math or for method - which turned out to be useful when I was reporting science stories. And it was my psych-research alarm bells, not my legal ones, that Rimm's abstract first set off. You see, even in his draft abstract Rimm was making statements that he could not possibly support. "Every time [users] log on, their transactions assist pornographers in compiling databases of information about their buying habits and sexual tastes," he'd written. It was the kind of absolute statement that no responsible researcher dealing with human behavior would ever make - given the range and unpredictability of human behavior, credible researchers of psych and social-science phenomena will qualify both their hypotheses and their conclusions. The abstract was chock-full of categorical generalizations like the one I quote here - generalizations that, given the limits on the types of data Rimm purported to be studying, were wholly inappropriate. And, as it happens, I knew that many of his statements were also flat wrong. In the course of my work, I'm regularly in contact with operators of adult BBSes (they often have questions about obscenity law, and they hope to stay on the right side of legality). Rimm claimed that BBS operators are refining their offerings of sexual material to focus on what Rimm asserts to be a more "lucrative market" in what he charmingly calls "paraphilias." This claim flew in the face of what I'd been hearing from the BBS sysops who called me for advice, or whom I met at conventions like One BBSCon. Those sysops wanted to minimize the risk of angering their communities - especially their local law-enforcement agencies - but the strategies Rimm was categorically attributing to them would increase their legal risks. There were other potential methodological problems: the reliance on verbal descriptions of the images to characterize them, the apparent conflation of Usenet and BBS data, the conflation of "download" and "consume." Sure, it was possible that in his discussion of his methodology, Rimm might put forward reasonable explanations for his peculiar approach, but even the most rigorous theoretical framework he could put forward would not leave him in the position of generalizing with the certainty to which he was prone in the abstract. And, given what I knew about Usenet and the difficulty of measuring user behavior there (I've long followed the pioneering research of Brian Reid at DEC), Rimm's implication that he might be able to determine "the percentage of all images available on the Usenet that are pornographic on any given day" was sheer fantasy. Nor were these the only problems I had with the abstract. But the biggest howler was this one: "The research team at Carnegie-Mellon University has undertaken the first systematic study of pornography on the Information Superhighway." Even from the abstract, it was apparent that the bulk of Rimm's data came from 68 "adult" BBSes - to generalize from commercial porn BBSes to "the Information Superhighway" would be like generalizing from Time Square adult bookstores to "the print medium." There were other weirdnesses that, strictly speaking, were neither factual nor methodological. Like Rimm's evident fascination with types of porn that are, uh, not mainstream. (I was about to say "off the beaten track," but I just remembered Rimm's expressed interest in material featuring sadomasochism.) It was hard to avoid the conclusion that Rimm was, at best, an odd duck, and that he had some sort of agenda. But it wasn't an agenda I was particularly worried about. Given the amateurishness of his abstract, I was certain the Rimm paper would never come to anything. I figured that, once the CMU censorship fracas died down, the Rimm research would sink, like most undergraduate research projects, into oblivion. (Look, no one can be right all the time!) So I read the occasional note I received from Rimm over the next few months with benign tolerance. In his subsequent email Rimm renewed his request that I review his legal footnotes - he even sent me the text of the footnotes for my convenience. But even if I'd had the time to check on someone else's legal research (doing the job right would require many hours), I couldn't ethically approve of legal footnotes without seeing the text of the article they were footnotes to. I pointed this out to Rimm and suggested that, if he were to send me the full article, I might be able to find the time to review the footnotes for any obvious mistakes. Rimm told me he'd get back to me on that. But he never did. And the next time I heard about the Rimm study was when Philip Elmer-DeWitt of Time called me early the week of 19 June 1995 for comment on the Rimm study and the conclusions Rimm, who by now had received his bachelor's degree, had reached. Among these conclusions, Philip told me, were that tastes for online porn were becoming more "extreme," that adult BBSes were using Usenet to market their wares, that sysops had discovered that the more "violent" the language of a description, the more popular an image was, and that Amateur Action BBS, whose Milpitas, California, sysops had recently been successfully prosecuted in Memphis, Tennessee, was "the market leader" of online porn. It was clear from the questions Philip was asking that Time was going to treat the Rimm study as a major story - perhaps even a cover story. And this insight was Part One of what I'd later think of as Philip's Triple Whammy. Given what I already knew about Rimm's research, I was appalled that Time would publicize it - I immediately tried to warn Philip of the methodological and other problems I had with the study. He told me that the study was going to be published in an article in the Georgetown Law Journal, that Time had an exclusive, and that he (that is, Philip Elmer-DeWitt) found Rimm's methodology convincing. I couldn't believe we were talking about the same study. Philip found it easy to dismiss my caveats - after all, I hadn't seen the study. So I asked to see it. I promised Philip that, if he showed it to me, I wouldn't "leak" it, but instead would use it to frame more detailed and substantive criticisms (or, perhaps, be forced to admit that the methodology and conclusions were convincing after all). That was when Philip hit me with the Second Whammy - thanks to an arrangement with the law journal and/or with Rimm (Philip was vague to me about this), no one outside of the editors of Time and the law journal would get to see the study before the Time story appeared and the law-journal issue was published. I was stunned - if there were questions about the study's reliability (and I still had every reason to believe there were), the arrangement Philip told me about practically guaranteed that those questions wouldn't be fully considered by Time's editors. Especially because Philip had already convinced himself that the doubts I tried to raise weren't serious ones. I knew Philip to be respected by Time editorial management - I'd even heard rumors of an upcoming promotion - so I was certain that, if Philip vouched for the reliability of the study, his superiors would take his word for it. At this point, I made two suggestions: first, I referred him to Donna Hoffman, a Vanderbilt University professor I knew from The Well. I knew Hoffman and her husband, Tom Novak, to be among the most knowledgeable people in the world when it came to questions of surveying Net usage or of modeling marketing strategies in this new medium. I assumed that Hoffman and Novak would raise the same methodological questions I had, plus some I'd no doubt overlooked, and perhaps that would convince Philip to look again at the reliability of the Rimm study. My second suggestion was for Philip to contact the person or parties that were insisting on nondisclosure of the article and ask them to grant me permission to see it for comment, with the proviso that I'd agree not to leak it in any way. This came to nothing - when I reminded Philip about it the following week, he professed not to remember that I'd ever proposed this arrangement. And although Philip did have one of Time's field reporters interview Hoffman, he never spoke to her himself. He did read the "file" from the reporter's interview, though. We know this because he later argued on The Well that the intensity of Hoffman's language in commenting on the Rimm study methodology (she knew about it from the abstract and - mirabile dictu! - from her own prior correspondence with Rimm, who'd solicited her advice and support months before) made her an unreliable source. After all, how could she be so critical when she hadn't seen the study? And, of course, she was barred from seeing it by the arrangement among Time, the law journal, and Rimm. The more I thought about the study's imminent publication, the more troubled I was by the secrecy and the lack of critical review. That's when it occurred to me to consider how odd it was that an article by an EE major, purporting to be a marketing study, was appearing in a law review. Although Philip took this to be an index of the study's likely reliability, I knew something that, at least at first, he did not - namely, that law reviews are unlike most other scholarly journals in that they're edited not by professors or professional editors but by third-year law students. And while I have the highest regard for the ability of student law-review editors at a school like the Georgetown University Law Center, I knew it was highly unlikely that the editorial staff had the expertise to question the claims and arguments that Rimm would be making about his computer-mediated research into the so-called information superhighway. Suddenly the legal footnotes took on a new significance - they were the thin, entering wedge that qualified Rimm's article as a fit piece for a law review. It all came together for me then. If Rimm had set out to publish an article about online porn in a way that legitimized his article yet escaped the kind of critical review the piece would have to undergo if published in a scholarly journal of computer-science, engineering, marketing, psychology, or communications, what better venue than a law journal? And a law-journal article would have an added advantage - it would be read by law professors, lawyers, and legally trained policymakers and taken seriously. It would automatically be catapulted into the center of the policy debate surrounding online censorship and freedom of speech. I tried to point this out to Philip when he called me back for a second interview, but he clearly wasn't terribly interested in hearing it - he grunted obligingly, but moved to the questions he really wanted to ask me, about the net.censorship legislation pending in Congress and about what I thought the effect of the publication of the study and its appearance in Time would be. "It will be a disaster," I told him. "It won't matter if you try to balance your presentation of the study with the questions people have about its methods and reliability. It'll be used to stoke the fires of the Great Internet Sex Panic." He noted my comments, then ended the conversation. As the days counted down to publication of the next issue of Time, I indulged in hopeful thoughts. Philip had a great track record as a reporter on cyberissues - for all that even the most balanced story would be, in my view, "a disaster," I could understand how Philip had convinced himself of the importance of the story, and, as a once and future journalist myself, I respected his commitment to tell a story even if the facts might generate the wrong kind of reaction among policymakers or the public. Not once in all my discussions with him had I ever suggested that he not do the story. And when it came to the critical issue of balance, I fancied that I could trust in his professionalism. Indeed, when rumors of the upcoming Time story had surfaced, and some Well users were ready to castigate Philip for writing it, I posted the following one-line message on Sunday morning, 25 June: "Let's hold off criticizing Time until we see what the story looks like." But all this hope left me wide open for what would turn out to be Part III of the Triple Whammy. Here's what I posted on Monday, when I had had a chance to read the piece as it appeared in Time: media.1029.86: Avant Garde A Clue (mnemonic) Mon 26 Jun 95 14:39 Philip's story is an utter disaster, and it will damage the debate about this issue because we will have to spend lots of time correcting misunderstandings that are directly attributable to the story. For example, when Philip tells us what the Carnegie Mellon researchers discovered, he begins his list with this: 'THERE'S AN AWFUL LOT OF PORN ONLINE. In an 18-month study, the team surveyed 917,410 sexually explicit pictures, descriptions, short stories and film clips. On those Usenet newsgroups where digitized images are stored, 83.5 percent of the pictures were pornographic.' Who but the most informed among us will not come away with the impression that the CMU study involved a survey of 917,410 items *on Usenet*? (Guess what -- it didn't.) And he concludes the list with this; "IT IS NOT JUST NAKED WOMEN. Perhaps because hard-core sex pictures are so widely available elsewhere, the adult BBS market seems to be driven largely by a demand for images that can't be found in the average magazine rack: pedophilia (nude photos of children), hebephilia (youths) and what the researchers call paraphilia--a grab bag of "deviant" material that includes images of bondage, sadomasochism, urination, defecation, and sex acts with a barnyard full of animals.' Problem is, this isn't the typical range of content you find in Usenet newsgroups, or on commercial services, or even on most BBSes. Instead, this is the range of content you find on the specialized subclass of commercial BBSes that focus on pornography. Just to make things worse, Philip refers to the Internet in the next two grafs (and not at all to commercial porn BBSes). This is an incredibly muddled abortion of a story, despite Philip's attempts to introduce balance. The *packaging* of the story -- a cover with an innocent child at a keyboard, the paintings of men fucking a computer or being pulled into one -- is deeply sensationalistic. And the profound problems with the study's methodology go undiscussed. Sure, we have a guy pointing the possibility of a "gaper" phenomenon, which tells us something about how to interpret the results of a correctly conducted survey. But not a hint of how methodologically flawed the study is, or about how the people doing the study were rank amateurs, or about how the legal footnotes were spiced with citations from anti-porn zealots like Catharine MacKinnon and Bruce Taylor. The Time story aims at legitimizing the study as raising important issues. What it does instead is raise serious questions about whether the lure of an exclusive eclipsed Time's professional judgment. In the course of the next few days, I questioned Philip pointedly about the writing and editorial decisions he'd made in the cyberporn cover story - decisions that both maximized the extent that the story exacerbated the Great Internet Sex Panic and actually obscured critical facts about the study. Philip occasionally responded with glib, superficial answers, which enraged me. It was as if he were deliberately ignoring the magnitude of what he'd done. Now it wasn't my researcher buttons that were being pushed - it was my journalism buttons. Philip had written the story in such a way that, in effect, he would be deceiving great numbers of his readers. With a copy of the study in hand (finally!) I began to savage Philip in the media conference on The Well: I was venting at this point. But it took a single epiphanic moment to convince me that the thing to do was not merely criticize Philip, but instead to do the kind of reporting he and Time had failed to do. What was that moment? I recounted it in a Well posting reproduced below: media.1029.197: Avant Garde A Clue (mnemonic) Tue 27 Jun 95 15:33 Well, I'm going to be on Nightline tonight, debating Ralph Reed of the Christian Coalition. It has already been taped -- we mostly debated whether the Exon legislation was a better fix for protecting your children than the software tools and filters that I advocated. But the taped lead-in focuses on the Rimm study, and stresses how the Rimm study shows how easily pornography is available to children on the Net. And not just any old pornography, but the hebephilia, urination, etc., that the Rimm study shows there is so much demand for. Before we taped, I mentioned to one of ABC's reporters, Richard Harris, that there were a number of methodological criticisms of the Rimm study. So, afterwards, Harris and his researcher arranged to have me in a conference call with Philip, Marty, and apparently one or more other people who were involved in the study. I was given a chance to raise my concerns about the study's methodology, with mixed results, so that the ABC people could hear, at least, some of the reasons for believing that the study focused not on cyberspace as a whole, but on a nonrandomly selected subset of commercial BBSs that focus on selling porn, that the study was based on descriptions, not images, and that the conclusory links between Rimm's sample and the "information highway" as a whole were not supported methodologically. I doubt it made much difference -- ABC guys aren't terribly interested in hearing nerds talk about statistical inferences. But they were kind enough to give me a hearing on the methodological and ethical problems I have with how this done. During the call, Philip noted that I'm an advocate, so it follows that I feel compelled to argue against a study that reports inconvenient facts. (I later pointed out to the ABC guys that the Levy piece has inconvenient facts, but I'm not outraged about that one, and that before I was a lawyer I was a researcher.) The implicit dishonesty of his casting doubt on my motives, of course, lies in the suggestion that Philip and Martin Rimm don't have far stronger motives of their own to pitch the study as something groundbreaking and compelling and reliable. Rimm answered a lot of the question I raised, some adeptly and others with dodges. He claimed to have no agenda. We wound down, although a few voices (mostly mine) were raised. But before we lost the connection, I heard this: Philip: "Marty, you there?" Rimm: "Yes, I'm here." Philip: [slight pause] "Good job!" To a journalist, Philip's "good job" was a revelation. At that point, I turned to Harris and mouthed (with regard to Philip): "He's on the team!" It was stunningly clear that Philip had so identified himself with the story that he believed his and Rimm's interests were essentially the same. There was only one fix, I thought - do everything I could to make sure that the truth about the Rimm study, and about Time's collusive arrangement, be made as public as possible, as widely as possible. I had already FedExed a copy of the study to Donna Hoffman. Working with EFF's legal interns, I made several copies of the article and FedExed them to people who knew enough to criticize the piece the way it should have been criticized at the beginning. I later learned that Philip characterized my labors as an "orchestrated campaign" to discredit him, conducted by a "professional lobbyist." Since I've never been a lobbyist in my life, that comment did sting, but of course in a sense he was right that I was conducting an "orchestrated campaign." It was orchestrated largely from my PowerBook, and the campaign consisted of putting copies of the study in front of independent reporters and other commentators who were capable of reading it and seeing the obvious. That's what led to Elizabeth Corcoran's insightful piece in The Washington Post, and to Peter Lewis's thorough reporting in The New York Times. It's what led to the critiques of the study that you see on Donna Hoffman's Web page and here at HotWired. It's what has led to the revelation that Rimm's own faculty adviser, named in the study's biographical footnote, doesn't think Rimm's data support his conclusions about Usenet. All the while I was getting other people to read the study, I was doing my own reading. Perhaps the single most damning discovery I made appears in this posting: media.1029.511: Avant Garde A Clue (mnemonic) Sat 1 Jul 95 18:34 Let's come back to the Footnote Quiz, which Philip declined to answer. I had written this: ********************* "9. As a result of federal legal action against a few well known 'adult' BBS operators, including Robert and Carleen Thomas (Amateur Action) and Robert Copella (Pequena Panacha), some systems have removed their paraphilic, pedophilic, and hebephilic imagery from public display. This has created a thriving underground market for 'private collections' and anonymous ftp sites on the Internent, which cannot be studied systematically. Thus, it may be difficult for researchers to repeat this study, as much valuable data is no longer publicly available. See infra notes 89-95 and accompanying text." Now, Philip, try answering this quiz: Of the many unsupported assertions in this single footnote, which one would raise the *biggest* red flag for a reader/editor working for a peer-reviewed journal? ********************* Now, this footnote is rife with candidates for "red flag status." The "some systems have removed" claim is undocumented and unsupported, as is the "thriving underground market" and the "anonymous ftp sites [market created for such sites because of porn crackdown]" comment. So's the claim that anonymous ftp sites can't be studied "systematically." (Note: it may well be true, but *it is not supported by the study*.) But the single biggest red flag is the penultimate sentence--"it may be difficult for researchers to repeat this study...." It is, in my opinion, designed to make the study unkillable, so that anti-porn activists will be able to use it forever, *regardless of subsequent studies that seem to disprove it*. Think about what would happen if subsequent studies seemed to support Rimm's conclusions: (Rimm: "See? I was right!") Now think about what would happen if subsequent studies seemed to *disprove* Rimm's conclusions. (Rimm: "See? I was right!") You begin to see why the author might have felt compelled to sidestep peer review if at all possible. The process of exposing the Rimm study, which I increasingly believe may have been a deliberately political ploy in the guise of "research," is ongoing. You can see the results here and elsewhere on the Net. And the same is true of the process of exposing the extent to which Time and Philip Elmer-DeWitt traded their responsibility to the American public in return for a questionable exclusive, written and packaged to maximize panic about the Net - it's ongoing. But there's one thing that's not ongoing, and that's any "orchestrated campaign" to discredit Philip Elmer-DeWitt. My response to that particular charge is best expressed in one last posting that I'll share with you here: media.1029.423: Avant Garde A Clue (mnemonic) Fri 30 Jun 95 13:19 Philip writes: "I can understand why you feel obliged to discredit the Rimm study." I don't think you do. Based on what you have said up to now, you think it's because it comes to some conclusions that are inconvenient for my work. You think I'm just playing out some role as an advocate for net.freedom, and therefore feel compelled to challenge the study out of a sense of loyalty to my cause. I'm sure that's what you told yourself when you decided to dismiss my comments out of hand. So long as you labor under this self-delusion as to my motives, you won't have a clue about why I'm doing what I'm doing now. "But I'm having a hard time understanding why you felt obliged to discredit me at the same time." Philip, you should be clear on this: I have never, ever had it in my power to discredit you, nor have I ever thought I did. You discredited yourself. Copyright 1995 Mike Godwin