User:PerfektesChaos/js/WikiSyntaxTextMod/flow/char

WikiSyntaxTextMod → Syntax polishing → Step 1

The first step on the syntax polishing lane standardizes on single character level. This is prerequisite for later analysis by this script, by bots and other gadgets, in order to detect strings in expected manner.

1–31 ASCII control codes

 * 10 – newline
 * The only appreciated control code is newline.
 * All other control codes should be removed.


 * 9 – horizontal tab
 * The horizontal tabulator is replaced by exactly one space character.
 * This has the same effect when rendering wikitext.
 * Tabs are often introduced by copy&paste, when existing rendered texts (mainly enumerations and tables) are inserted into Wikipedia. Strange effects occur when editing in TEXTAREA.
 * The only eception is within  where the tabs remain. Therefore the occurrence of a tab is only prebooked. If any tab was seen, cleanup is postponed until finished tag analysis and   areas have been protected.


 * 11 – ver tab, 12 – form feed
 * It seems that the database rejects these codes if attempting to save. It is not possible to enter them by normal keystrokes, but they might result from copy&paste.
 * These characters indicate a separation between two sections. If really found in edited text, they are replaced by newline.


 * Any other code ≤ 31
 * As before; neither insertable, nor stored in database.
 * If occurring nevertheless they will be removed.

Unicode control codes

 * Writing left-to-right
 * The invisible l-t-r and r-t-l (200E16 and 200F16) might be very meaningful for interlanguage and pages with content in other languages. Basically they are kept.
 * All are consistently made visible as  and   entities.
 * If these codes are found
 * around the target of a wikilink or
 * around a template name,
 * they will be removed programmatically.
 * Templates for support of foreign languages are supposed to handle the bi-directional issues internally. In the plain article text the entities may be removed without any consequence then.


 * Zero width characters
 * These are currently 200B16 (ZERO WIDTH SPACE), 200C16 (ZERO WIDTH NON-JOINER) and 200D16 (ZERO WIDTH JOINER) and they are made visible as,   and   homogeneously.
 * In non latin scripts they have a certain meaning. In latin text they are pointless; usually resulting from copy&paste insertions, perhaps database queries, and may be deleted. They are regularly occurring in interlanguage links and are mandatory there; within links they remain unchanged. For the sake of undisturbed diffpage all entities are turned into character codes within link targets later.


 * Line Separator
 * In Unicode there is a “line separator” 202816, which is not visible in normal browser displays. This particular line break is not meaningful for a wiki and will be replaced by a regular newline character. Within link targets the code is ignored by the parser anyway and will be removed. The character is generated by copy&paste insertions, when a mostly “soft” line break has been inserted into the displayed text by the rendering system.

Unicode typography

 * Space
 * Simple spaces (ASCII) are replacing:
 * 200216 – N-SPACE
 * 200316 – M-SPACE


 * Non-breaking hyphen
 * This character (201116 = 820910 – NON-BREAKING HYPHEN) might be helpful for future rendering of web sites with improved automatic syllabification sinnvoll sein; but searching for strings is obstructed.
 * Directly before or after a (regular) space or a line break ( as well as  ) it is always pointless. There it is always turned into by ASCII hyphen.
 * In some cases where no syllabification is possible the non-breaking hyphen is removed, if never coming into effect.
 * A user defined change into ASCII hyphen or removal is left to the user.

Exotic characters

 * Instead of regular characters sometimes decorative variants from dingbats are used. Typically the author is not aware of the difference and was not heading for that. Those codes will break search processes and scripts/bots, and they are not defined in many fonts yet and won’t be rendered.
 * Instead of dagger † (822410,, might mean “died”) one of three representations 10013–1001510 (271D–271F16); those will be replaced.
 * Instead of times cross × (21510, ) one of 10005/1000610 (2715/271616); those will be replaced.
 * The (male) number sign º \xBA &amp;ordm; looks very similar to the degree sign ° \xB0 &amp;deg;. Digits ahead indicate more a degree sign, and a temperature (°C) clearly requires a degree sign. If unambiguous the character ist corrected automatically.
 * Some authors use roman numbers 8544–857510 (216016 –217F16) rather than a sequence of ASCII letters. However, this is often unavailable to user fonts and not expected on string search. Out of protected regions this will be replaced by common ASCII letters. In context of CJK that Unicode solution gives better typographic results; if such CJK character is next replacement is inhibited.
 * CJK glyphs with latin equivalent between latin characters (mostly ASCII) are reduced to ASCII characters. This affects the ranges U+3000-300D and U+FF01-U+FF5E.

Invisible characters

 * Some source texts contain hidden characters from copy&paste activities in normal TEXTAREA. They are made visible as entities by the script as  or   and could be deleted now manually (and should be discarded – see also 29794). If the inserted entity remains in source code, that does no harm since the source in effect has not been changed and rendering keeps the same appearance. If that character is placed at end or begin of alink target and known to be redundant, it will be removed.
 * In general invisible characters for control of writing direction (left/right) are made visible as hexadecimal entities, in order to remove superfluous ones. The same goes for the “zero width space” U+200B (ZERO WIDTH SPACE) etc.:

Zero width characters
Some non-latin scripts use invisible “zero width” characters. Within link targets (mostly interlanguage) or recognized text sequences (especially the entire page) they areencoded invisible; else visualized as entity.

Spaces at line end
This avoids that searching expressions for  fail.
 * 1) If a line consists of spaces only, all these spaces are removed always.
 * 1) If for any other reason it happens to change the text, on termination of automatic processing all spaces invisible at end of line will be removed.
 * 2) * If this would be the only change, that is dispensed if control diffpage is active as common, otherwise the user would be confused by solely invisible modifications.
 * 3) * If the line end is terminated by one single equal sign, exactly one from existing space characters is kept. Some people prefer this for not yet assigned template parameters.

Character entities

 * Since a character entity could be protected by  or    or may be used within a comment as plain text, entities are replaced by UCS not during this first step, but after identification of protected areas (syntax readability).

Kept characters
It may be discussed whether some character codes are meaningful for a wiki project. The script does not replace them by default, but they might be changed or removed by user defined rules.


 * U+00A0 – non-breaking space
 * Since a few years this code (not only as entity ) is possible in wikitext. In the first years the database changed it into a normal space.
 * Currently they are made visible as  and might be judged for intended usage.


 * U+00AD – soft hyphen
 * Sometimes AD16 = 17310 is present in wikitext, mainly if entire text has been imported from other text systems. In future it might improve the layout of websites with automatic syllabification.
 * Searching for strings is obstructed. If no wikilink adheres, the character can be removed without any consequences by user defined rules or replaced by  entity.


 * U+2010 – hyphen
 * The Unicode hyphen 820810 is generated by text systems when automatic syllabification breaks a line. It is unlikely that they result from manual input, probably they result from copy&paste of an entire word which has been separated. In wikitext it might be assumed that the parts are to be joined again into the single word. Unicode hyphen and an adjacent space are to remove or replace by soft hyphen, but this requires human control.

127–159 (Windows code pages)

 * This range is not defined in ANSI/UCS.
 * Such character codes appear sometimes in wikitext, but are invisible for many authors.
 * The script makes them visible as decimal entities.
 * In latin based wikipedia it might be assumed that codes 128–159 shall be interpreted according to codepage 1252. User defined replacements might help if a frequent problem.

Remarks
&#91; German page &#93;