Speech is what we all do, however literate, and we pause for effect. Writing is related, but different, set down and fixed to be picked up later, and the construction of writing attempts clarity. Grammar and punctuation is different regarding the two. The two have often been confused and competed. Partly this is a problem of grammar and punctuation itself, because it was never fixed or rule laden in classical languages in their times, and was breath based, nor was it rule bound as in the later development of English (just like spelling). Grammar and punctuation was about reading better, then it was about clarity of the written word, then it was about house-style in the new printing (and later on newspapers produced lots of small paragraphs), and then it was about rules of presentation.
Today we should write like this for speech.
He said, "Where am I going today?"
There is the rule of the quotation marks which surround the speech including the full stop or other end punctuation, which replaces that of the wider sentence starting with "Notice how those quotes have the full stop afterwards, sometimes different quotation makrs may be used. Oh, and brackets can be in sentences or outside them!) The speech shown begins with its own capital letter. The next sentence for the next speech should start on a new line (but does it always?) and the comma is contained within the quotation marks.
Formal education led to formal grammar and punctuation, and embodied notions of propriety, correctness and even morality. So grammar and punctuation took on rules. However, these rules have never quite had universal application, and differed between aiding speech from writing (and reflecting speech in writing) and reducing ambiguity in the meaning of writing.
Aristotle and Locke had classical views of the compartmental function of the mind which was realised through the later study of the classics and classical languages. It is almost as if the subject studied did not matter, so long as it exercised the faculties of reasoning, observation and attention (see Baron, 2000, 145, who discusses the early American experience). Of course, high art and high literature were more demanding and enriching than something more common. Only when one had learnt to think did application of thinking take place in real life.
Against this is the view that we think in our minds and the speech comes out in the way that we think (and then we can write what we say). Yet the classical view held that knowledge was a given, high and fixed (to be discovered, certainly not creatively invented) and therefore such direct use of mental faculties was slovenly and common.
Latin was once punctuated for reading out (Baron, 2000, 167), but its word order and its place and purpose as brain fodder had the effect of making grammar and punctuation more formal and rigid in English too.
This did rely on having a developed education system and plenty of books: a lot of basic education has been oral and repetitive chanting. Those who could afford literate education received the formal rules. Professions adopted these rules too, populated by products of formal education.
The effect of all this classical education out of time was to maintain the separation of writing and speech, so that writing was constructed and punctuated one way and speech another. Conversely, another effect, through the continuation of Latin long beyond its practical use, was to give less clarity to English grammar and punctuation. Beautiful English could be learnt, but it was always second best to the purity of Latin. Therefore speech continued to be important in writing grammar and punctuation.
English of course was always second best in the eyes of the Norman French and the establishment which followed, which used Latin and spoke in French, and which maintained the superiority of the quality of French long after English came back into the government system. English (as it evolved) was always a language of the people; and whilst government and aristocracy needed to speak to the wider population in English, and thus English was successful (as it had been over the Danes), it was nevertheless left to be a fluid and informal language.
The King James Bible was an exercise in literate communication with ordinary people by keeping to relatively few words, whereas Shakespeare was the expansion of English into high art by invention, using the crossroads of dialects where he lived, the growing influence of the monarchy/ government located London-Oxford dialect as semi-official, and the effects of English nationalism and the renaissance in general. Nevertheless, even then, Latin and French had its class based status, and so did proper written texts and grammar and punctuation.
These classical and class based notions were built in to the academic based school system, also reflected in the type of education available for those who passed the eleven plus after the Second World War. This was an education for those seeking to go to university and public service, not industry, and thus it imported all these historical notions of "training the mind". Training the mind was still a concept about using the remote and unapplied. Formal gramatical rules mattered.
When this type of education began to loosen, and when education became a relationship with personal experience (again see Baron, 2000, 146-147, for some comparisons), then grammar and punctuation was bound to become more individual, informal and more related to speech.
Progressive education (with its philosophical base in Rousseau) emphasised further the experience of the student. Much was about being child centred, encouraging creative self-expression, doing rather than listening, and being socialised (Baron, 2000, 148). Seeing as we speak of our experiences, the writing is going to be more speech like. So now the underlying theory of the mind rejected the view that education was just building up the brain for later life, but was about surrounding life and experience. We all know how erratically stream of consciousness novels are punctuated (if at all - example James Joyce's Ullysses). This shift in pedagogy is therefore a shift in grammar and punctuation.
Meaning is derived from experience. It leaves English unclear if there are any rules about grammar and punctuation.
English now exists in so many different forms. There is a world business English, almost a second language with added commercial and technological and scientific terms. It combines American, British and some world regional Englishes. The Internet is full of this kind of English. Then there is American English and British English, the former which went around the world with the British Empire and the latter with American influence and its communications. Spelling is very confused again. Then there are dialects. Some dialects in the New World are continent wide, or nearly, and based on informality (for example the cockney that spread across Australia), and dialects in the home country are many and dense, though being eroded through the American media, national media and youth culture (often American derived).
If English is conversational throughout the world, and this is after all its world function, then grammar and punctuation will reflect this variety. So Captain Kirk the American said "to boldly go" whereas Latin derived rules demand that he should have said "to go boldly". His crime was that he had split the infinitive. Nothing should go in between "to" and "where to". But he was speaking!
In terms of English as a world uniting and translation medium (the aim of Esperanto), the practioners simply haven't the unified grounding in a home base to make common puntuation decisions. Although English was always going to be more successful than a constructed Esperanto, it is losing this base now.
Now at one time writing was an individual affair. Much of it still is. But more recently documents have become passed around. This is the effect of technology, where one networked computer user, or an Internet linked user, collaborates in the production of written material with others. So the original written piece is rather a dialogue, and enters into change through dialogue. The text ceases to be therefore the property of one author. Rather than using strong rules to unify the final output, the input by experience is a process that grows, and a layering process of many grammar and punctuations styles (or lack of style) take place.
It is bizarre, therefore, that a principal word processing software package, Microsoft Word, should have a grammar check which insists that "which" needs a comma before it whereas "that" does not. This is strictly true in an age where the strictly true has been undermined by the uses made possible by technology. "Which" is used for non-restrictive clauses and "that" for restrictive ones, but who knows this?
The author was once a key figure with a history and intentions. This once skillful author had to know the art of writing and grammar and punctuation. However, we now say that the readers are their own authors. So even if it was produced by one person, poststructural theory suggests that it is the reader who makes a text meaningful, and we need know no history of the author nor the intentions of the author. We now know that a text does not have to be coherent around the one person's thoughts or access to knowledge, but can be plural, and we see that clarity is a construct usually requiring artificial devices (again thanks to poststructuralism). Text will flow in different directions and, like a soap opera, have no distinct ending (another device). (See Baron, 2000, 159) So the essayist, who once produced a beautiful rounded piece of work, cannot, after all, produce a perfect reflection of knowledge, but has been involved in some kind of hoodwinking to end up with the rounded artefact. Grammar and punctuation along with narrative was the method of hoodwinking.
It follows to ask why a student should give different arguments and come up with a balance of opinion, to one side or another, according to the argument only presented there? When an essay is instead fluid, what is grammar and punctuation to do, other than to allow breathing? The student is too often trained in the hoodwinking devices. The studnt learns to argue a hind leg off a donkey.
Nowadays an online document often has links that allow the readers to go off in different directions (er, mind don't usually). An argument on the page is not therefore followed through, but received in a fragmentary manner, before the reader hops off elsewhere. In fact the argument may be joined down the text at a bookmark linked to from elsewhere. Clearly grammar and punctuation in this situation is losing its point.
Then someone decided that the business letter should be grammar and punctuation sparse. There should be none except in the body of the letter. The text should run down the left hand side of the page. Speed is the reason for these changes. Authors of books now tend to use less punctuation than before. A text should flow and if its meaning isn't clear sentences are reduced in length. Commas should be the minimum necessary.
The latest developments in communication seem to be assassinating grammar and punctuation altogether. Emails written rapidly and text messaging with a minimum of letters for meaning are losing even capital letters, never mind comments. Words are constantly subverted, and text icons give emotive expression where tone of voice was used by telephone, or grammar in the written word. The language has become inventive through reduction (it seems). The rapidity and repetition of messages makes up for lack of clarity in the first place.
My own approach is to dislike the scrappiness and lack of clarity in email conversations. I prefer some rules. I also think the author is attempting to make a point, and there is at least a dialogue involved; the reader is restricted somewhat. The rules, though, are flexible. I hate all lower case letters when some upper case is more elegant. Grammar and punctuation has to be for effect, to separate out (for example when placing a comma before "and"), and make emphasis. A lot of my output is to be readable, and I write in a way that can be spoken. I punctuate sermons to read out.
My history is finding writing difficult. I am often unclear even now. I tackled this by reducing sentence size and reducing paragraph sizes. I still puzzle people. I like a few archaisms.
Grammar and punctuation is an evolved social agreement. It is indeed social, reflecting a class and education system. Its apparent rules have only been in existence for a short time and in some places. Variety has returned, and meltdown is threatening.
Baron, Naomi S. (2000), Alphabet to email: how written english evolved and where it's heading [title case as given], London: Routledge.