Writing is like any other art form in that it has its accepted styles, its rules of composition, and, among all those rules, the emphasis to push beyond those boundaries to stake a claim on a unique style all one’s own. This can leave an author in a difficult situation, tossed between the stylistic conventions of old, codified for a reason, and the desire to be revolutionary in his/her writing. So what do you do? Is there a point where your press against the boundaries of style and formatting cease being unique and start being simply obfuscating?
There are no hard and fast rules when it comes to stretching the conventions of style and formatting. After all, there are critically-acclaimed volumes that throw almost all of those conventions out of the window. Such stylistic choices as stream-of-consciousness have gone so far beyond the realm of normal styles to essentially create new creative schools in the writing world. However, it is important to note that it is just as easy to run such a story completely off the rails as to ride them to critical success. Why is that?
Well, as I said above, the styles and formatting guidelines that exist exist for a reason. In communication and language, these rules are there to encourage clear communication of thoughts and ideas. The further you move beyond those guidelines, the more you cloud the true meaning of your words. It might fit the themes of your book to introduce some of this confusion, but it is important to measure it in the proper dollops.
To me, the guide posts, the line that I don’t want to venture beyond, is the point where the third-person voice, the voice of the narrator/God if you will, begins to lie to the reader. Use whatever extreme stylistic tools you like when the lens is focused through the eye of a character, essentially using the style and format to add to the unreliable narrator technique, but when it comes to that direct communication between writer and reader, trust must be complete.
Another thing to keep in mind is the ease of the read. Is the book you are creating still able to be read fairly easily by your intended audience? Yes, it is not an irrevocable sin to create a work that requires time and study to comprehend. However, keep in mind every ounce of effort required to crack the literary code you have created is an ounce of the reader’s patience you burn away. Wear too hard on the reader’s patience and you run the risk of having the reader decide to put your book down, deciding the fun of the story doesn’t hold up against the effort required to read it.
So be experimental! Try out new styles and new techniques! However, make sure you expose these new things to a variety of beta readers and an editor or two. They can help you find the fine line between innovation and obfuscation!