Writing The Great American Novel(s)

I’ve been doing a little writing lately. Specifically, I’ve written a legal thriller, and I’m in the middle of a second. I’ve got one more to go, and then I plan to switch to science fiction for a while.
It seems that every American is working on their own novel. Indeed, it’s the rare person I’ve met who hasn’t got at least a basic idea of what they’d like to write, and interestingly, everyone has a different reason for wanting to write. Some have a story they want to tell. Some want to understand themselves. Some want to strike it rich as writers. Some just have the writing bug, or they love the idea of creating something.
I started writing the first novel because I wanted to see if I could do it. I’ve read a lot of books and seen a lot of films in the past, and several things kept bothering me. Too many authors cheated. They let their readers into a character’s mind right until they needed to generate tension, then they cut the reader off. Others created characters whose sole purpose was to push the hero through the plot. Ron Weasley falls into this category in Harry Potter; he acts according to the demands of the plot rather than according to his own interests and beliefs. Other authors routinely rely on deus ex machina or incredible coincidences to make their plots work. This bothered me.
But criticism is one thing and creation is another, so I drew up my complaints as a list of rules and I set out to see if I could write a book without violating these rules. Yep, I did that thing.
By now I’m writing because it’s addictive to create worlds and immerse characters into them, though I find I still need to set challenges for myself. Besides my list of rules, I challenged myself to include some pretty big philosophical points in the first book, but not so that casual readers would notice. The second book needs to give a realistic portrayal of how much attorneys work in the dark about the true motives and intents of their clients and of the other side, and how little they can be sure of about what really happened. Plus, I’m trying to build a genuine mystery where all the clues are constantly there, but the reader just won't assemble them until the end. Another challenge I’m looking forward too will be making one of the science fiction stories into a comedy, without it being a wacky comedy or slapstick (which I also find to be cheating).
The more I've written, the more I’ve discovered some interesting things. I found, for example, that my characters became independent, sometimes annoyingly so. As the stories progressed, they took on real personalities, and I found they wouldn’t always do what I wanted them to do. That meant new plot twists and actually kept me guessing how things would turn out. Indeed, strangely, I found my opinions about several characters changing. Some became more sympathetic, some less so. Some I even felt probably deserved their own books.
I also discovered that the more time I spent building these characters, the easier it became to see how other authors built theirs. Did they just toss them into the story to satisfy the plot (JK Rowling)? Were they designed to hit certain plot points (Stephen King)? Were they all the same character, presumably a projection of the author (Grisham)? Or were they real people with independence. At the same time, I started to hear film dialog a lot more clearly, and I realized I could almost see the dialog printed on the page. This showed me some of the real beauty in the writing in certain films, and the utter lack of it in others, and it gave me total disdain for formulaic pabulum.
So what I’m curious about today is how many of you have written something or want to write something, whether you finished it or not. What was it? What was it about? What did you learn or what did you find fascinating? And most importantly, why did you start? Do tell!

(Let’s put the issue of publishing aside for now, that’s a discussion for another day.)