piss

<p></p><p></p><p><img alt="Terminal app icon" class="icon" height="112" src="https://cdn.jim-nielsen.com/macos/512/terminal-2021-06-03.png?rf=1024" width="112" />&nbsp;<img height="74" src="https://github.com/gdamore/tcell/raw/main/logos/tcell.png" width="107" /></p><p></p>(Note: I'm resurrecting this dormant blog after a <i>long</i>&nbsp;period of inactivity. &nbsp;But this thing bothered me so much that I felt I needed to write a blog about it.)<div><br /></div><div style="text-align: left;"><i>TLDR: The macOS Terminal.app is still buggy and not recommended. Instead consider switching to a 3rd party emulator. &nbsp;The rest of this post describes the faults in far more detail.</i></div><div><p style="text-align: left;"></p><p style="text-align: left;">One of the more lauded features in macOS 26 is the updated <i>Terminal.app</i>.</p> <p>As some of you may know, I'm the author of a very popular library for interacting with terminals and terminal emulators, at least within the Go ecosystem.</p><p>I was excited to hear about the new Terminal.app. Finally, we would get support for 24-bit color, years after everyone else has had it. (Even <i>Microsoft</i> beat Apple to the punch here, leading by something like half a decade.)</p><p>So yes, the good news is that macOS Terminal.app in <i>does</i>&nbsp;support 24-bit color.</p><p>But in nearly every other interesting case it is far inferior.</p><p>The rest of this is probably only interesting to terminal nerds, but for those of you who care about these things, the new Terminal.app is a major disappointment.</p> <h3 style="text-align: left;">Identification Woes</h3> <p>The first and most egregious failing is that there is no good way to remotely identify the macOS terminal, or even what features it might support.</p> <p></p> <p>Normally we would like to be able to query and get useful feedback via one of the following mechanisms:</p> <ol> <li> <p>Primary Device Attributes (obtained via <code>CSI c</code>): <i>Terminal.app</i> reports that it is a vanilla VT100 with no extra features, it does not claim to be a VT220 or similar.</p></li> <li><p>2. Extended Device Attributes (obained via <code>CSI &gt; q</code>): <i>Terminal.app</i> reports <em><b>nothing</b></em> here. Most modern terminal emulators store their application name and version here. While I dislike hardcoding capabilities for a specific terminal emulator (so-called "quirks"), this isn't even an option because we can't know that this is Terminal.app.</p> </li> <li><p>3. <code>DECRQM</code> - DEC mode queries. &nbsp;This is supported officially starting with VT320 and up, but pretty much every other software emulator supports it. This can be used to query whether a given DEC mode (including things like mouse support and features, automatic margin wrap, etc.) are present and modifiable. &nbsp;It has been used for new things like <a href="https://github.com/contour-terminal/vt-extensions/blob/master/synchronized-output.md">synchronized output</a> (to fix tearing during a resize) and to identify support for <a href="https://mitchellh.com/writing/grapheme-clusters-in-terminals">grapheme clusters</a> (very important in a Unicode world). &nbsp;<i>Terminal.app</i> does not support these queries, but it even gets it wrong in a worse way. This is a CSI sequence, so the reasonable behavior would be to parse to the end of the sequence, and discard it if you don't support it. <i>Terminal.app</i> instead aborts the parsing early, emitting the final character of the sequence on the screen. So during early startup we see these mysterious "p" characters show up as Tcell tries to query for support.</p> </li> </ol> <p>Ok, this is just about identification. &nbsp;What about <code>TERM</code>? &nbsp;Well Terminal.app identifies as <code>xterm-256color</code> - although it clearly is not compatible with genuine <i>xterm</i>. &nbsp;What about <code>$TERM_PROGRAM</code>? &nbsp;Well, yes, we <i>could</i>&nbsp;use that. &nbsp;But note that this environment variable is <i>not</i>&nbsp;passed through <i>ssh</i> by default, which makes it useful only for locally running applications.</p><h3 style="text-align: left;">Unicode Woes</h3><p>Modern applications and users expect to be able to use things like skin-tone modifiers for emoji, national flags, and similar "characters" (well graphemes technically, which is just another way of saying a user-perceived character). In Unicode, these graphemes are composed by combining multiple Unicode "characters" together.&nbsp;</p><p>This approach is also used with certain languages, such as Arabic, or even in Western languages where a character might be modified to include the diaeresis using two code points joined by a special character called the "zero-width-joiner". (For example, in Bengali, র্য&nbsp;(No, I do not read or speak Bengali.)</p><p>How does <i>Terminal.app</i>&nbsp;fare here? &nbsp;Well its mixed results. For some, such as 👨‍🚒 and the Bengali example, it works reasonably well. &nbsp;But for others, like the flag emoji&nbsp;🇨🇭 it miscalculates the width. &nbsp;(In this case it calculates the width of the flag as just one cell, instead of two, even though it renders it using two cells!).</p><p>This behavior could be detected by an application emitting the composed sequence, then querying the resulting position using <code>CSI 6 n</code>.</p> <p>It would be easier, and better if Apple would implement the DECRQM and advertise mode 2027. &nbsp;It could be hard coded to 3 (forced on) in this case, unless Apple were prepared to add the complexity to support both legacy and modern grapheme support.</p> <p>Hashimoto also has something to say about <i>Terminal.app</i> and grapheme clustering:&nbsp;</p><blockquote><p>🤡 Living in its own cursed little world</p></blockquote><p></p>(Apparently it miscalculated the width a joined sequence as 6, probably because it considered the ZWJ as occupying two cells.)<div><br /></div><h3 style="text-align: left;">Broken State Machine</h3><div><br /></div><div>So modern terminals, which includes everything starting from VT100 or claiming to support ANSI X3.64 or <a href="https://ecma-international.org/publications-and-standards/standards/ecma-48/">ECMA 48</a>, are expected to fully parse (but may not implement) certain classes of escape sequences. &nbsp;For example, &nbsp;consider the sequence emitted by this command:</div><div><br /></div> <blockquote><code>printf "\x1b]something\x1b\\"</code></blockquote><div><br /></div><div>This is a OSC (operating system command). &nbsp;The "\x1b" values are escape codes, and the command is "something". &nbsp;Now no terminal (probably) has a command "something". &nbsp;(Usually OSC sequences begin with a number followed by a semicolon, followed by arguments, but this isn't required.)</div><div><br /></div><div>Every reasonable implementation should parse the entire string, using a simple state machine that recognizes "\x1b]" is the start of OSC sequence, which is terminated by either "\x1b\\" (formally the ST sequence in ECMA-48), or "\x07" (due to historical accident.) &nbsp;Note that ST can also be encoded in a single eight bit character as "\x9c", but 8-bit encodings are uncommon due to potential confusion with other national encodings (but not UTF-8, or ISO-8859, or any EUC or ISO-2022 encoding).</div><div><br /></div>So what does <i>Terminal.app</i> do when it sees this sequence? It prints "omething" on screen. &nbsp;(It consumed the leading "s".) &nbsp;One possible interpretation is that they implement all OSC commands by printing the entirety of the command minus the first character, to the screen. &nbsp;Is this a legal interpretation? &nbsp;Technically, yes. &nbsp;Is it a reasonable interpretation?<div><br /></div><div>I believe this is an egregious violation of <a href="https://www.laws-of-software.com/laws/postel/">Postel's Law</a>. (By the way, if you work in software, you should be well familiar with Postel's Law, which was first invoked by the late Jon Postel, whose name is listed as an author of most of the foundational RFCs that define the Internet.)<br /><div><br /></div><div>More importantly, this interpretation means that you cannot safely use OSC sequences at all on <i>Terminal.app</i>, although you can reasonably do this for real <i>xterm</i>. But Terminal.app is not a real <i>xterm</i>, so that should be fine -- <b>except</b> that Terminal.app does nothing to make itself discoverable. It doesn't define its own value of <code>$TERM</code>, it doesn't answer any of the standard queries to test for functionality, nor does it answer the extended attributes to help a user program identify that it should avoid any of the normal things it might expect to be able to do with a real implementation of <i>xterm</i>.</div><div><br /></div><div>Btw, this is not limited to just the OSC sequences. &nbsp; A similar, but slightly different behavior occurs with DCS sequences (device control sequences). &nbsp;It just emits the entire string (including the first character).</div><div><br /></div><div>What about <code>APC</code>, <code>SOS</code> and <code>PM</code> (which are generally unused by modern terminals)? They behave as <code>DCS</code>. The content goes straight to screen.</div><div><br /></div><div>Then there is that strange case of CSI sequences that it does not understand. Remember <code>DECRQM</code> above? That is a valid CSI sequence:</div><div><br /></div> <blockquote><div><code>CSI ? </code><i>Pm</i><code> $ p</code></div></blockquote> <div><br /></div><div>(The final character here is "p" and the <i>Pm</i> is a numeric mode number such as 7 for automatic margin wrapping.) As we have seen, the parser in Terminal.app does not swallow these, but acts as if "$" were the legal character. &nbsp;(Technically "$" is considered an "intermediate byte" in the ECMA specification.)</div></div><div><br /></div><h3 style="text-align: left;">Modern Key Events</h3><div><br /></div><div>Modern terminal applications are looking for ways to make better use of the keyboard, for things like control-keys for short cuts. &nbsp;Historically, it was impossible to discriminate between certain keys. &nbsp;For example, <kbd>Ctrl</kbd>-<kbd>I</kbd> is the same as <kbd>TAB</kbd>, and <kbd>Control</kbd>-<kbd>M</kbd> is the same as <kbd>Enter</kbd>. (This goes all the way back to decisions made in the 1970s.)</div><div><br /></div><div>In answer to this, there are a few standards available. &nbsp;The most widely popular one is a <a href="https://sw.kovidgoyal.net/kitty/keyboard-protocol/">protocol</a> designed by the author of <i><a href="https://sw.kovidgoyal.net/kitty/">kitty</a></i>, and lets terminal applications fully discriminate different key sequences and can even report key repeat and release events. &nbsp;(Xterm has its own way to do this, which is a subset of the functionality, and Windows Terminal invented its own scheme called <a href="https://dev.to/andylbrummer/taming-windows-terminals-win32-input-mode-in-go-conpty-applications-7gg">win32-input-mode</a>, which offers the full richness but has some drawbacks in efficiency and safety -- you cannot recover from this mode if you turn it on by mistake in a shell.)</div><div><br /></div><div>Of course, <i>Terminal.app</i> has none of these, and is stuck in the 1970s for key reporting.</div><div><br /></div><h3 style="text-align: left;">So What To Do?</h3><div><br /></div><div>With all these problems with Terminal.app, one might wonder what recourse there is. Fortunately, I have good news.</div><div><br /></div><div>There are a plethora of excellent options available which are all far superior to <i>Terminal.app</i>, and which are free. (There are non-free options as well.)</div><div><br /></div><div>Personally, I use the excellent <a href="https://ghostty.org"><i>Ghostty</i></a>, but in the past I had great results with the old stand by&nbsp;<i><a href="https://iterm2.com">iTerm2</a></i> as well as&nbsp;<i>kitty</i>. &nbsp;The up and coming <i><a href="https://rioterm.com">Rio</a></i> is also pretty excellent. The&nbsp;<i>Alacritty </i>program is popular but I found it inferior to these other options (no grapheme clustering, and no advanced keyboard support). I found <i>WezTerm</i> to be brittle (it would occasionally hang), so I don't recommend it personally, but it has many adherents.</div></div><div><br /></div> <h3 style="text-align: left;">What About Windows Users?</h3> <div><br /></div><div>For Windows users, <i>Rio</i>&nbsp;might be the best option, although for advanced keyboard handling the modern Windows 11 Terminal is quite excellent. &nbsp;It still lacks some things I'd like to have, like support for <code>CSI &gt; q</code> identification, and better support for grapheme clusters, but for most applications it behaves quite well. One I specifically do <i>not</i>&nbsp;recommend is <i>ConEmu</i>, as it has many different ways in which it is broken. &nbsp;It also appears to be abandonware, in spite of its once great popularity.</div>