Charlie and the Chocolate Factory is 50

In many ways, the most astonishing thing about Charlie and the Chocolate Factory – a bestseller since it was first published 50 years ago, beloved of generations of children, multiply adapted for stage and screen and a furnisher of images and phrases that have entered the cultural lexicon forevermore – is that it ever got written at all.

At the end of the 1950s, Roald Dahl was riding high. He had had a "good" war, as first a flying ace and then an intelligence agent in Washington funnelling information back to MI6, and was leading a glamorous life in the US as a successful writer of dark short stories for adults and the husband of the film star Patricia Neal. By 1957, he had started making up bedtime stories for the first two of the couple's five children, Olivia and Tessa. One of the tales was about a small boy who lived near an enormous chocolate factory – inspired, perhaps, by the one flattened by the eponymous fruit in James and the Giant Peach, his first book for children, which he had just finished and which would be published in 1961.

He wrote a first draft of the chocolate story but, as he wrote to young readers many years later: "I got everything wrong. I wrote a story about a little boy who was going round a chocolate factory and he accidentally fell into a big tub of melted chocolate and got sucked into the machine that made chocolate figures and couldn't get out." He is taken to a shop and discovered by a little girl when she starts to eat her treat. Dahl's young nephew read it, told him it was rubbish and so – we must assume, as it has never been found among Dahl's otherwise meticulously kept papers and correspondence – the writer threw it away.

But shortly after his son Theo was born in 1960, Dahl sent a revised version of the story, entitled "Charlie's Chocolate Boy" to his agent Mike Watkins, in which the eponymous hero visits the factory with nine other children and is accidentally made into a chocolate figure and delivered to Mr Wonka's house, where he foils a burglary and is rewarded with a sweet shop of his own "nine storeys high".

The others include Augustus Pottle (Gloop's precursor), Miranda Grope (disappears up the pipe with Augustus), Wilbur Rice, Tommy Troutbeck (whose fates you will learn in the never-before-published chapter cut from the draft that accompanies this piece); Violet Strabismus (had been Glockenberry, would become Beauregarde, always ends up violet); Clarence Crump, Bertie Upside, Trevor Roper (who all overheat after ingesting an unwise number of Warming Candies), and Elvira Entwhistle (gets booted down a rubbish chute but would eventually be known as Veruca Salt as she went) – an unwieldy group and it is obvious that some of them need to go, but it is still great fun while they're around.

But then Theo was almost killed when a cab hit his pram in New York. He survived, but developed hydrocephalus. The shunt put in his head to drain the fluid kept clogging, nearly killing him each time. Dahl mined Theo's neurosurgeon Kenneth Till for every ounce of his knowledge then took the problem to his friend Stanley Wade who, in a twist you wouldn't dare write, was an engineer whose hobby was making miniature engines for toy aeroplanes and whose job was running a factory that produced precision hydraulic pumps.

Together, the three men invented the Wade-Dahl-Till valve for Theo and the thousands of other children in his situation. In June 1962, the first one was inserted into the head of a patient in London's Great Ormond Street hospital. It worked beautifully. Theo was well enough not to need it by then, but over the years it was used to treat thousands of children all over the world.

Dahl went back to his book.

The publishers Knopf were reading a second draft when Olivia and Tessa arrived home from school with a note warning parents about a measles outbreak. Vaccination programmes would be introduced a year later in America, but that was too late for the Dahls. Seven-year-old Olivia caught the bug and died. It is perhaps both impossible and unwise to try to describe the depths of anyone's grief at losing a child: recalling it 20 years later in her autobiography As I Am, Neal still struggled to articulate hers; of her husband, she says simply that he "all but lost his mind".

Eventually, however – and it's not quite clear how long after, because Dahl did not date his drafts – the need, both financial and personal, to work reasserted itself and another draft of Charlie took shape. And another, and another.

Reading them now is like watching a familiar landscape slowly emerge out of the mist, or the coloured chips of glass in a kaleidoscope before a final turn of the lens aligns them in the proper pattern. The chocolate river is there, but there's no waterfall or minty grass-meadow setting (though the latter has its precursor in a garden Wonka makes for a rich woman, full of trees with barley sugar branches, fudge trunks and mint crisp leaves). The inventor is the central figure, not Charlie. There are uniformed workers in the factory and disembodied voices whispering the songs that accompany each child's departure instead of a musical tribe of tiny, cacao-loving Loompaland natives.

What we think of as the "real" Dahl is there, moving underneath the story like a shark but only occasionally breaking the surface to show his grinning teeth (one mother objects to her child being made into fudge on the grounds that "we've spent far too much on his education already"). But it is only after a letter from his former agent and confidante Sheila St Lawrence that you can see him start to really trust his instincts. Although she says now that "he was going to get there anyway … If someone else hadn't alerted him, I'm quite sure he would have alerted himself", she made a variety of specific suggestions – including making the uniformed assistants "something more surprising than they are" – but also encouraged him more generally to let rip. "I'd like to see more humour, more light, Dahlesque touches throughout," ends the letter. "I hope some of my remarks will produce counter remarks in you that will stir you to flights of fancy to make the book take off and really fly, as it undoubtedly will."

And it did. It was published in the US in 1964 and sold 10,000 copies in the first week (and was acclaimed as "fertile in invention, rich in humour, acutely observant … he lets his imagination rip in fairyland" in the New York Times), and has been pretty much flying off the shelves ever since.

Dahl was writing at an interesting time in the history of children's books. Following the war, publishers had become properly interested in the field for almost the first time. Penguin had established the world's first major children's paperback imprint, Puffin, under the mighty Eleanor Graham's leadership. Other publishers, too, began appointing specialist editors with ambition, building quality lists and paying more money to those who could supply the books they needed. The combination of increased pay and prestige started to attract writers who, like Dahl, had previously written for adults. Educators started to take a greater interest in children's literature, and support (including financial, particularly in the more affluent US) in schools and for children's experts in public libraries grew. In that sense, the conditions were right for Dahl.

In other ways, they were not. From its earliest days, children's literature has been associated with moral improvement. Even now the idea persists that every book should be dedicated to shoring up parental efforts to mould the "half-civilised little buggers", as Dahl once called the juvenile population. US librarians – increasingly powerful gatekeepers standing between a book and its access to its intended audience – were often vociferously committed to this view.

Among educators there was a belief that children should not be exposed to anything in their books that they had not come across in real life, in case they became disconcerted or upset and were put off reading – hence the existence of official word lists for different age groups that authors were advised to stick to. Theodor "Dr Seuss" Geisel famously made these limitations work for him when he produced The Cat in the Hat using only the 200-odd words with which first-graders were supposed to be familiar, but both the practicalities and the attitude enshrined were for most far from a liberation. In addition, the American world of children's books was fairly solidly anglophile, and the Anglos they philed in the book world were a conservative lot.

Dahl's anarchic spirit and spikily, gruesomely satisfying resolutions to his problems and plots ran counter to all of this. From the beginning there were objections (the Library Journal's review of James and the Giant Peach had damned it for its "violent exaggeration of language and almost grotesque characterisation … Not recommended") and they would increase as Dahl became more prominent.

But his main problem at this point was getting published in the UK. All the "top-tier" firms turned him down. In a 1964 letter to Alfred Knopf's wife Blanche, Dahl blamed the literary establishment's "priggish, obtuse stuffiness". A former publisher at Bodley Head, Judy Taylor, told Dahl's first biographer, Jeremy Treglown: "I could see that Dahl would be popular with children, but publishing for them has to involve more than that somehow." Another editor told Donald Sturrock, Dahl's most recent biographer, that she remained proud to have turned the book down twice.

Then Tessa's schoolfriend Camilla Unwin brought home US editions of James and the Giant Peach and Charlie and the Chocolate Factory that Tessa had given her. Her father was Rayner Unwin, co‑founder of Allen & Unwin publishers. The books were published by his firm in 1967 and as in the US became immediate bestsellers. Over the next decade, Dahl became more and more popular with children and less and less popular with the powers that be.

Most of us remember our first Dahl book as a revelatory experience. The tone – both confident and confiding from the moment Charlie is introduced ("'How d'you do, how d'you do, and how d'you do again?' He is pleased to meet you") – and the brio, exhilarating and infectious. It is fascinating to see that whatever else came and went in the drafting process, that most vital and inimitable feature remained constant. "If you look now, you can see Mr Wonka quite easily," runs the opening paragraph of "Charlie's Chocolate Boy", followed by a breathless description of all that goes on inside the factory before finishing with "I am telling you all these things simply to show you what a fantastically clever chocolate maker Mr Willy Wonka was – and still is." And there your eight-year-old self is, right where Dahl wants you, hushed, hooked and hardly breathing for the duration.

And then there was the content. Wonka's outspokenness and flip-to-brutal asides are almost always a new and thrilling development for a child reader. This is not what he or she expects and, as Alison Lurie points out in her collection of essays about childhood books Don't Tell the Grown Ups, young readers are instinctively attracted to subversive stories. They are, after all, people too. The stories "suggest that there are other views of human life besides those of the shopping centre and the corporation. They mock current assumptions and express the imaginative, unconventional, non-commercial view of the world in its simplest and purest form. They appeal to the questioning, rebellious child within all of us, renew our instinctive energy, and act as a force for change. This is why such literature is worthy of our attention and will endure long after more conventional tales have been forgotten."

Parents and others were less uniformly enamoured than their offspring (or, we might infer, Lurie). The discomfort expressed by the UK publishers who rejected him found further expression elsewhere as Dahl's fame grew.

But first he had to deal with a more specific charge, that of racism. News in 1970 that there was to be a film of the book (Willy Wonka and the Chocolate Factory, directed by Mel Stuart) drew the National Association for the Advancement of Coloured People's attention to the fact that the Oompa Loompas were black pygmies from Africa and their importation to the factory had objectionable overtones of slavery. Although Dahl didn't say so at the time, it seems to me from reading the early drafts that their original incarnation is left over from a long, eventually cut section (though it was used in Tim Burton's 2005 film, in which Johnny Depp's Wonka hacks his way through the jungle to find the Oompa Loompas) about Wonka's efforts to save some tribespeople from being thrown off their land by a vicious new plantation owner and starving to death. It is obviously a result of and a showcase for the knowledge and love of the country he developed when working in Africa for Shell and being stationed there during the war as a young man. He insisted that the Oompa Loompas represented no racist intent of any kind and willingly rewrote them in time for the second US edition as the tiny, hippyish figures from an invented land that we know today.

But his head was now above the parapet and, after the film was released, a highly regarded children's writer and critic called Eleanor Cameron came out swinging against the book, claiming in an article in the influential journal the Horn Book that it was a "phoney presentation of poverty and its phoney humour, which is based on punishment with overtones of sadism" that left readers "poorly nourished with [their] taste dulled for better fare".

Similar accusations pursued Dahl through the 70s and 80s and are still – very occasionally – heard today. Dahl rebutted most of them to his own satisfaction by reiterating his belief – last stated in an interview in the journal Children's Literature in Education in 1990, the year he died – that "a good children's book teaches the uses of words, the joy of playing with language. Above all it helps children learn not to be frightened of books … If my books can help children become readers, then I feel I have accomplished something important." His primary task, he felt, was to entertain rather than inform.

The argument that reading itself is a sufficiently improving act almost regardless of content has perhaps gained greater acceptance in recent years, as adult concern about the colonisation of young, malleable minds by the internet, and the prospect of future generations in effect being deskilled in the arts of immersion, concentration and appreciation of anything longer than an Instagram caption, has grown.

The subtly increasing prioritisation of form over content is an interesting phenomenon, but it only comes into play here if you think the criticisms of the contents of Charlie (and Dahl's other books) are valid.

One of the great benefits of writing a fairytale is that it lends itself almost infinitely well to adaptation. Archetypes are archetypes for a reason. The heroic/greedy/spoiled/pathologically irritating/ill-mannered will always be with us. Add a chocolate factory, a roomful of trained squirrels and another containing an edible riverside idyll and a boiled sweet boat, and you have catnip to every generation of young readers, some of whom grow up to be film, opera and stage directors, producers, writers, set designers – and to the generation after that who slice, dice and repackage the results (like the Oompa Loompas with Tommy Troutbeck in the Pounding and Cutting Room for the internet age).

Stuart's 1971 film Willy Wonka and the Chocolate Factory wasn't initially a hit. But a few years later it began playing on US network television at Christmas and Thanksgiving. And in the 1980s cable television arrived, then video. Burton's 2005 film didn't really take off, even though by that time technology had enabled that roomful of trained squirrels to be accurately rendered. Stuart had had to give in and use puppet geese and golden eggs instead. But his film has become a classic, endlessly (lovingly) parodied by everyone from The Simpsons to 30 Rock, referenced in rock and rap songs, and has lent itself particularly well to meme-sized culture. The web is awash with jpegged and Gifed Wilder/Wonka-LOLZ.

Dahl's biographer Sturrock and composer Peter Ash transformed Charlie into an opera, delighted to find that Augustus Gloop was a natural fat Italian tenor given to Puccini-esque arias at moments of high drama, shrill show-off Violet a coloratura soprano (traditionally the most high-maintenance of performers), Veruca the deeper-voiced, more dictatorial mezzo, and that Mike Teavee's hyperactivity mapped perfectly on to a baroque-style counter-tenor, full of Monteverdian agitation.

And of course, a stage musical of the book, directed by Sam Mendes and written by David Greig, is currently playing in the West End. Augustus is as large as ever (I suspect I shall live to see the day when Dahl is vilified for fat-shaming), Violet is a child popstar, Mike a fan of violent video games and Veruca simply appalling. All of which is tremendous fun and has doubtless helped both spread the word (literally – there can be barely anyone among us who has not at some point used the phrase "Golden Ticket" to mean a free pass for something, or "Willy Wonka" as shorthand for any kind of innovator or genius) and keep the book in the collective consciousness.

But Charlie was a bestseller for years before even the first film came out, and it's hard to imagine that the potent blend of imagination, wit, energy and eternal appeal to all the senses it contains would not have assured its constant place on our shelves for half a century without any other aid.

In 1972, Watkins wrote to Dahl to tell him they had sold more than 200,000 copies of Charlie, James and the Giant Peach and his third book Fantastic Mr Fox that year. "Imagining that each copy must have a minimum of five readers," he said, given that children share and many copies belonged to school and public libraries, "you have given an enormous amount of pleasure." 

 Lucy Mangan