User:Oshanis/SCFG Carl deMarcken

The main article for SCFG could be found at. Please consider this as an addendum to that article.

This article is based on the work of Carl deMarcken.

Representational problems
Unsupervised learning techniques such as the inside-outside algorithm produces grammars that structure text in ways contrary to our linguistic intuitions. For example, when semantically interpreting the phrase “listening to music”, it is more likely that a human will group the prepositional phrase “to music” rather than group the verb phrase “listening to”, because “to music” can move and delete as one unit better than “listening to”. Consider the examples: “it is to music that I listened” or “it is music that I listened to”.

However, when simple techniques for estimating CFGs by minimizing cross entropy is used, “to music” will have a higher entropy associated with it than “listening to”. This is because in English “V,P” (verb followed by a pronoun) will have higher mutual information than any other combination; i.e. I(V,P) > I(P,N) > I(V,N) Therefore, when considering the following parse tree structures (A-H), structures (A-D) will have a higher entropy than structures (E-H), and thus the SCFG will favor a structure in the group (E-H) although structure A is the most accurate at a syntactic level.

In another study by Olivier (1968) in acquiring a lexicon form unsegmented  character sequences by treating each word as a stochastic context free rule mapping a common nonterminal (say W) to a sequence of letters; the following representational problem occurs. Although it creates rules such as W => the and W => tobe, it also hypothesizes words like W => edby, since “-ed by” is a common character sequence that occurs in passive constructions such as “walked by” or “cooked by” etc. Here “edby” occur together not because they are part of a common word, but because English syntax and semantics places these two morphemes side by side.

Convergence to a sub-optimal grammar
Inside-Outside algorithm is attracted to grammars whose terminals concentrate probability on small numbers of rules that it is incapable of performing real search. As a consequence of such rule interaction the search space becomes discontinuous. It converges on the nearest such grammar only biased by its relative merits.

Multiple expansions of a nonterminal
Multiple expansions of the same nonterminal will lower the probability associated with the nonterminal. Therefore the algorithm selects grammars which may have a non-optimal phrase structure. Since it fails to model multiple ordered adjunction without increasing the number of nonterminals, it is also incapable of converging when there are terminals with recursive expansions.

Head driven grammatical formalisms:
Grammars are represented in terms of head relations like in link grammars. With this kind of representation many details of phrase structure that are unimportant as far as minimizing entropy is concerned are factored out. This simplifies the search space and is less likely to encounter local optima. Also with this method a Viterbi learning scheme is more likely to estimate accurate counts as it narrows down the parse hypotheses. This is important because unbracketed corpora has very high computational complexity in estimating long-distance word-pair probabilities.

Fringe rules and learning:
By permitting non-binary production rules, much of the problems in statistical induction can be avoided. It makes sense to provide rules in a more natural representation of the words in a sentence which spans all the words in the sentence, rather than treating all these words as a chain of pairwise relationships. Although there is a risk of over-parameterization and the production of enormous rules, using minimum description length (MDL) criterion those effects could be minimized. Also to address the problem of long parses containing very little internal structure, the right hand side of a rule can be considered as a partial derivation tree with terminals and non-terminals mixed together.