In the 1970s, when I first started writing about science fiction, I was one of those who argued strenuously that we should tear down the ghetto walls. I believed we had a strong case: science fiction would certainly benefit from all that the mainstream had to offer, and the mainstream, then in a rather moribund phase, would equally benefit from the vigor and the storytelling and the invention of science fiction. I also believed that we were winning. Hadn't the new wave injected literary modernism into British science fiction? Wasn't Thomas Pynchon shortlisted for the Nebula Award and didn't Michael Moorcock win the Guardian Fiction prize and hadn't Christopher Priest been included in the first Granta Young Writers Promotion, and weren't serious writers like Doris Lessing and P.D. James and Paul Theroux experimenting with the genre (however incompetently in some instances), and weren't those members of the literary establishment who looked down their noses at sci-fi fewer and more foolish? But we lost the battle: Star Wars came out. It may have taken a while to notice, but from that moment SF became primarily a visual medium, a form distinguished by big effects and simple stories. The literary establishment found it easy to continue disdaining such a simple narrative form, while within the genre the literature would quickly start to follow the visual. All those reinventions of the old forms, the new space opera, the new hard SF and their like, are direct descendants of Star Wars and its ilk.

Writing in 1998, Jonathan Lethem argued that it was the moment that Arthur C. Clarke's Rendezvous With Rama beat Pynchon's Gravity's Rainbow to the Nebula Award that science fiction gave up the unequal struggle to be a serious literary form, a part of the mainstream. This anthology is John Kessel's and James Patrick Kelly's counter-argument. Here they amass evidence to show that Lethem is wrong, that the loss of a battle is not the end of a war, that slowly, steadily, we are gaining ground. Of course I want to believe them, I want to imagine that though the ghetto walls still stand they are less secure than they once were, that traffic now flows more freely in both directions across the boundaries. But how strong a case do they present?

Let's face it, nowadays science fiction on film and television receives far more attention that the literature, and there it is inevitably judged more on visual impact than literary niceties. What's more, the space opera and the hard SF story are enjoying a popularity within the genre that they haven't known since the 1950s, and there are writers pursuing these subgenres whose literary skills are greater than any who have gone this route before. It's exciting back inside the ghetto walls, so what's the incentive to leave?

This, then, is what Kelly and Kessel have to contend with. Their armament consists of 19 stories ranging in date from 1971 (two years before Lethem's "tombstone marking the death of the hope that science fiction was about to merge with the mainstream") to 2008. Many of the stories will be very familiar (Ursula K. Le Guin's "The Ones Who Walk Away From Omelas," Lucius Shepard's "Salvador" and Connie Willis's "Schwarzschild Radius," for instance, have all been anthologized numerous times), and all of the authors are well known names. It is a comprehensive and wide-ranging attack, though it has to be said, right from the outset, that it is an attack launched on one front only.

Just as the twentieth century may well have been the American Century, so science fiction may well have been a predominantly American literature; but in neither case was that ever the entire story, and as we come to the end of the first decade of the twenty-first century it is emphatically not the case. It is a disappointment, therefore, that the editors of this anthology have not turned outside of North America for a single contributor. This means, for example, that British and British-based writers who clearly belong within this secret history (Christopher Priest, Alasdair Gray, Angela Carter, Kazuo Ishiguro) and South American writers whose magic realism did so much to initiate this secret history (Gabriel Garcia Marquez, Isabel Allende) are omitted from the story. We are left with an inevitably partial view of the whole enterprise.

There is another problem that is not actually addressed within the book: what are we talking about here? What makes (or does not make) these stories science fiction? Because only by answering that can we know what this is a secret history of (or even if it is a secret history of anything). The closest we get to an answer comes in one of the pairings of well-chosen quotations that introduce each story, when Maureen McHugh says: "I don't want to define science fiction . . . I'm not saying that genres don't exist. I'm just saying that they don't have defined edges." In other words, the secret history works best if we assume that the definition of science fiction is as porous as the ghetto walls we are now being asked to imagine. If we want a tightly delineated genre, if we are looking for the technological fix that we mostly associate with hard SF, then much of what we find here is not science fiction. But if we are comfortable with a looser, more generous fit, if we are looking for something more along the lines of what John Clute has taken to calling "fantastika" (and which I, more prosaically, prefer to term "the fantastic"), then what we have here is a form of SF that we are all familiar with, but that doesn't call attention to itself because it is not part of any self-proclaimed sub-genre.

To be more specific, any argument that the ghetto walls that separate science fiction off from the mainstream are crumbling has to show two things: that mainstream authors continue to move in and out of genre as the needs of story dictate, and that genre authors have benefitted from an infusion of influences from the mainstream. To do this, Kelly and Kessel have brought together stories by seven authors generally considered to belong to the mainstream but that all dip enthusiastically into genre, alongside 11 stories by genre authors that clearly show mainstream influences. Molly Gloss doesn't fit easily into this binary, since her novels tend towards the mainstream and her short stories tend towards genre. But then, the more you examine it, the less clear-cut the division is throughout the book. Steven Millhauser would presumably be identified as a mainstream author, yet practically everything he has written has belonged to the fantastic. Michael Chabon began as a mainstream author but has moved increasingly into genre, to the extent that The Yiddish Policemen's Union won both the Hugo and Nebula Awards (achieving what Thomas Pynchon so signally failed to do). Jonathan Lethem's earliest novels were all science fiction, and there is still an element of the fantastic in such supposedly mainstream novels as The Fortress of Solitude. Margaret Atwood won the inaugural Arthur C. Clarke Award for The Handmaid's Tale, Don DeLillo wrote an SF novel, Ratner's Star, and George Saunders and T.C. Boyle have both written a number of stories that merge into genre. These are not exactly representatives of a mainstream we might normally expect to turn its nose up at genre, though, of course, that's the point. In this history of science fiction there is no way of telling where genre ends and the mainstream begins.

Nevertheless, there are differences between the two groups. Although Chabon's "The Martian Agent, A Planetary Romance" is a steampunk alternate history jammed full of unashamed genre clichés, he is the only member of the mainstream group to use genre devices purely for the fun of it. In the main, as represented by T.C. Boyle's story of a woman having a love affair with an ape, "Descent of Man," and George Saunders's masterful "93990," in which human observers fail to recognize the increasingly human behaviour of an ape responding to the torment and death of its fellows during chemical experiments, the devices of science fiction are being used as tools with which to question what it is that makes us human. It is in this light, for instance, that Don DeLillo has two astronauts looking down on an embattled Earth in "Human Moments In World War III," Jonathan Lethem has prisoners literally built into the walls of a jail in "The Hardened Criminals," and Steven Millhauser has Edison experimenting with artificially simulated touch which creates sensations never known before in "The Wizard of West Orange."

Among the science fiction writers that same questioning of what it is to be human is at the heart of Ursula K. Le Guin's "The Ones Who Walk Away From Omelas," but in the main this group of writers is questioning what it is that makes science fiction. In other words, for one group science fiction is a set of tools to be put at the service of the story, for the other group science fiction is a way of telling story, and this difference, though subtle, is significant.

Some stories specifically refer to other science fictions, most obviously "The Nine Billion Names of God" by Carter Scholz in which, Pierre Menard-like, a writer called Carter Scholz recreates word-for-word Arthur C. Clarke's famous story. More subtly, the very title, "Frankenstein's Daughter," shapes our reading of Maureen F. McHugh's story of a family struggling to cope with a brain-damaged cloned daughter. Or again, in "10¹⁶ to 1" by James Patrick Kelly, the young protagonist's collection of Galaxy is of vital importance in shaping expectations on both sides when he meets a genuine time traveller at the time of the Cuban Missile Crisis.

Others are considerably less overt, even when they are exploring much the same territory. When, in "Standing Room Only" by Karen Joy Fowler, Mary Surratt's daughter encounters strange, possibly foreign, people gathering outside Ford's Theatre on the night we know Lincoln will be shot, we assume but are not told that they are time travelers. Then we wonder whether we are making this assumption only because the story was first published in Asimov's and is here collected in a science fiction anthology. Something similar happens in our reading of "The Ziggurat" by Gene Wolfe when a solitary (and decidedly misogynistic) man in an isolated, snow-covered landscape meets strange women who may be from the future, who may be fairies, or who may even be aliens (in passing, I can't help feeling that this story could be read as a masculine response to James Tiptree's "The Women Men Don't See"). We can never be entirely sure, however, how much of our interpretation of these women is based on our knowledge that Wolfe is an acclaimed author of the fantastic and this story is appearing in a science fiction anthology. Right at the beginning of this book, Wolfe is quoted as saying "Realistic fiction leaves out far, far too much," a notion that is perfectly in keeping with the spirit of this anthology, but that is particularly ironic given that what is most vivid and compelling in Wolfe's story is the realist account of a man isolated in a landscape and alienated from his family, while the fantastic elements are melodramatic and work at all mostly because of what is left out for our imaginations to add.

In a celebrated essay, "POSTcyberMODERNpunkISM," the critic Brian McHale identified a feedback loop between the postmodernism of Thomas Pynchon and the cyberpunk of William Gibson (two authors notable for their absence from this collection). This sort of feedback comes when there is a self-awareness within the genre, a loss of innocent storytelling replaced by a more knowing story construction. Such self-awareness seems to be part of what we see in these stories by authors more commonly associated with genre, a consciousness of the expectations that genre conventions can arouse. (A surreal fantasy such as "Buddha Nostril Bird" by John Kessel consists of little but genre conventions piled upon genre conventions, and we make sense of the story only by measuring what is presented within the story against what we would normally expect such devices to achieve.) Is "Schwarzschild Radius" by Connie Willis science fiction because it uses physics as a metaphor for front line experience during the First World War, or because it is written for an audience who have become experienced in recognizing such metaphors?

Whatever, the stories written by the science fiction writers are almost invariably stories written in the knowledge of science fiction, written as a way of exploring what it is that makes the story science fiction. The exceptions, and perhaps the purest expressions of the ideal that the ghetto walls must tumble, are the earliest in the collection, written when such cross-currents of affiliation were first being espoused. Le Guin's "The Ones Who Walk Away From Omelas" is a humanist thought experiment that just happens to wear the clothing of the fantastic. "Ladies and Gentlemen, This Is Your Crisis" by Kate Wilhelm is a chilling anticipation of our addiction to what we now call reality tv, but it is primarily a humane examination of the relationship between two devoted members of the audience. Perhaps most tellingly of all, "Angouleme" by Thomas M. Disch is a searing tale of what we assume is the near future written entirely in the mainstream realist mode. It is science fiction written without any consideration for genre expectations, science fiction that has fully absorbed the lessons of character and manner from the mainstream. It is, also, brilliant, perhaps the best thing in an excellent collection. One wonders what science fiction would have been like over the last 40 years if it had taken this as its model rather than Star Wars, a different and not so secret history.

In a lengthy passage that concludes this volume, Ursula K. Le Guin says "it will take not only courage for SF to join the community of literature, but strength, self-respect, the will not to settle for the second rate. It will take genuine self-criticism. And it will include genuine praise." I want to give genuine praise to this book. It is full of stories you should read, there isn't one you will regret reading, and together they may help to change your expectations about the genre. But I must hesitate also. As I said earlier, it is only part of the story. And while we hover in the doorway, I wonder whether SF has the will to step through and join the "community of literature," or conversely that the community has any interest in stepping through to take a look at genre. At its best, though, this anthology reassures me that there are still some of us holding on to the dream.

Paul Kincaid is the coeditor of The Arthur C. Clarke Award: A Critical Anthology, and the author of the Hugo-nominated collection of reviews and essays, What It Is We Do When We Read Science Fiction.