User:Cscott/Ideas/Wikitext Composition and Extension

I'm drifting increasingly off topic here, but note the separation between "parsing"/tokenization and evaluation/expansion. The contents of the  are opaque to the tokenizer while the contents of the top level template arguments are not, but that turns out to be mostly a syntax issue regarding how delimiters are found and braces matched up. The "value" of  can be a linked list data structure, it doesn't "have" to be transmitted as a string. Similarly  doesn't have to have   tags in it's argument when the parser function gets it. Tokenizing the arguments helps identify delimiters in "human friendly" ways in corner cases -- aka forgetting to add a close  or close   usually gives much nicer results than forgetting a closing   which will happily suck in everything until it finds the next   -- but fundamentally the arguments should be thought of as raw strings from the original source /not/ tokens, at least until the...