console: consume APC, DM, DCS

commit 3a2b2eb55681158d3e3ef464fbf47574cf0c517c upstream.

The Linux console's VT102 implementation already consumes OSC
("Operating System Command") sequences, probably because that's how
palette changes are transmitted.

In addition to OSC, there are three other major clases of ANSI control
strings: APC ("Application Program Command"), PM ("Privacy Message"),
and DCS ("Device Control String").  They are handled similarly to OSC in
terms of termination.

Source: vt100.net

Add three new enumerated states, one for each of these types.  All three
are handled the same way right now--they simply consume input until
terminated.  I hope to expand upon this firmament in the future.  Add
new predicate ansi_control_string(), returning true for any of these
states.  Replace explicit checks against ESosc with calls to this
function.  Transition to these states appropriately from the escape
initiation (ESesc) state.

This was motivated by the following Notcurses bugs:

 https://github.com/dankamongmen/notcurses/issues/2050
 https://github.com/dankamongmen/notcurses/issues/1828
 https://github.com/dankamongmen/notcurses/issues/2069

where standard VT sequences are not consumed by the Linux console.  It's
not necessary that the Linux console *support* these sequences, but it
ought *consume* these well-specified classes of sequences.

Tested by sending a variety of escape sequences to the console, and
verifying that they still worked, or were now properly consumed.
Verified that the escapes were properly terminated at a generic level.
Verified that the Notcurses tools continued to show expected output on
the Linux console, except now without escape bleedthrough.

Link: https://lore.kernel.org/lkml/YSydL0q8iaUfkphg@schwarzgerat.orthanc/
Signed-off-by: nick black <dankamongmen@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Jiri Slaby <jirislaby@kernel.org>
Cc: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
1 file changed