Replacing TCT backend with Notcurses

[dankamongmen]

To be clear, I am interested in implementing this feature myself. I just wanted to get feedback and direction before running off and doing so.

Expected behavior of the wanted feature

mpv currently supports a TCT blitter using Unicode halfblocks, and a Sixel blitter. Both can be used in certain terminals and setups, but they are distinct backends. Many (most, at least on Linux) terminals don't support Sixel, and TCT halfblocks are noticeably inferior to the Quadblitter and Sexblitter of Notcurses (take a look at dankamongmen/notcurses#1071).

Notcurses is committed to best-of-breed multimedia rendering on the terminal, and unified these various blitters in a single backend, with intelligent backoff and degradation. notcurses-view takes a fraction of the CPU that mpv -vo tct requires for the same media (admittedly, it's doing less work, and needn't sync audio to video), despite producing superior output (for all but pathological inputs).

I'd thus like to replace the TCT and new Sixel backends with Notcurses.

I'd appreciate any feedback the mpv team might have. Thanks!

[avih]

First of all, nothing prevents you from starting to implement it. If the result is good, it could be interesting, but we can't guarantee to merge before understanding what kind of complexities it adds.

Second, you didn't mention why is it superior to half blocks and sixel.

For how, I would not think about replacing existing backends. First you need to show something better.

Specifically, I could not find, during my short link zapping, meaningful info on how it works. I did see it uses unicode quadrants and some symbols for 2x3 (if you go further, you can even do 2x4 with breil), however, once you try to push more than two "pixels" into a character cell, then except when using sixels, you run into the limit that a cell can have only two colors - foreground and background - which is quite limiting when you try to convert an arbitrary image.

So, first, for now don't think about replacing - concentrate on showing something which works well. Second, I'd appreciate some explanation on why it's better than halfblocks and sixel, and an answer to the number-of-colors-per-cell.

[dankamongmen]

First of all, nothing prevents you from starting to implement it. If the result is good, it could be interesting, but we can't guarantee to merge before understanding what kind of complexities it adds.

Absolutely. I just wanted to see if it would be inapplicable outright.

Second, you didn't mention why is it superior to half blocks and sixel.

Well, Sixel is always going to give you the best fidelity, obviously, but it's not implemented on most terminals available on Linux (xterm is the primary one that does; kitty, alacritty, and VTE do not). So this provides a fallback where Sixel isn't available on the terminal, without having to manage fallback yourself -- just tell ncvisual_render() to blit with the best solution it can find.

As for halfblocks vs quadrants and sextants, the benefit is only present for media that has greater resolution than {cols x 2xrows}. Media smaller than that can be represented perfectly by half blocks. Of course, most video etc. is at a greater resolution than this. In such cases, rather than scaling the image to the {1x, 2x} of halfblocks, you can get {2x, 2x} or {3x, 2x} with quads and sextants, respectively. Obviously, if there are more than two colors in the [2x2] or [3x2] area, some lerping goes on, but Notcurses handles this well, and the resulting visual output is cleaner. An example can be found here: https://nick-black.com/dankwiki/images/1/13/Worldmap-sexblitter..png. On the right is ASCII spaces. The third position is halfblocks. Quadblitter and finally sexblitter fill out to the left. The difference in quality is pretty apparent IMHO.

For how, I would not think about replacing existing backends. First you need to show something better.

Certainly =]. Sorry if I came off as dismissive; I did not intend to. mpv is awesome.

unicode quadrants and some symbols for 2x3 (if you go further, you can even do 2x4 with breil), however, once you try to push more than two "pixels" into a character cell, then except when using sixels, you run into the limit

Notcurses actually has a Braille blitter as well, but due to how it's actually represented in font glyphs, it tends not to work nearly as well as block-oriented glyphs. They're all available through notcurses-view if you'd like to check them out.

Thanks for your reply. I'm hearing that the team isn't outright opposed to this proposal, but that I'd need bring data and evidence to any kind of code checkin. I assure you I wouldn't dream of doing less.

[dankamongmen]

https://user-images.githubusercontent.com/143473/97769637-19678a80-1b03-11eb-8ac1-2a696a11fd9c.png would be another good example of the various unicode blitters. here we have (clockwise from upper left) ASCII, halfblocks, sextants, and quads. Look at how much sharper the sextant render is.

[avih]

rather than scaling the image to the {1x, 2x} of halfblocks, you can get {2x, 2x} or {3x, 2x} with quads and sextants, respectively.

Wouldn't it be far more efficient and easy to just reduce the font size - to increase the resolution which can fit at the terminal?

[dankamongmen]

rather than scaling the image to the {1x, 2x} of halfblocks, you can get {2x, 2x} or {3x, 2x} with quads and sextants, respectively.

Wouldn't it be far more efficient and easy to just reduce the font size - to increase the resolution which can fit at the terminal?

sure, if you don't care about other glyphs (used for diagnostics, info, subtitles etc.).

[avih]

Well, the way I see it, I expect the main issue to be the limit of two colors per cell. I just don't think it would end up not-ugly for arbitrary video content, because the only thing which can be done about it is workarounds.

If you have a demonstration of a video using notcurses - post a link. Personally I think it would not be worth the effort, but I can't stop you from trying to prove me wrong :)

[CounterPillow]

Notcurses is committed to best-of-breed multimedia rendering on the terminal

I'm glad they're committed to coming first in the special olympics.

[haasn]

Does anybody actually use these VOs for anything other than getting a few novelty giggles out of them?

[dankamongmen]

Nevermind, have fun, folks! Back to the special olympics with me. I am left wondering why you have the TCT halfblock solution given the apparent stupidity of the whole approach, but I'll leave that question for the ages.

[haasn]

I am left wondering why you have the TCT halfblock solution given the apparent stupidity of the whole approach, but I'll leave that question for the ages.

You're not alone in wondering this.

[CounterPillow]

I am left wondering why you have the TCT halfblock solution given the apparent stupidity of the whole approach

Because it's a funny haha VO that also serves a role as a convenient minimal example of how the mpv video output stuff works. Hell, the similar caca video output right above it explicitly has a note stating that it's a joke.

The arguments I see for shoehorning graphical output into a terminal as coloured text are precisely those used by the abstraction violating sickos that gave us the abominable webpage-bundled-with-browser-as-an-application known as Electron™, and similar confirmations of Ted Kaczynski's beliefs about the evils of technology. We do not need to repeat history. If you want to remotely view graphical output, you should still use a protocol that operates on graphics, not text. If you want to view a video in a VT, use the drm API. If the drm API does not align with your use cases (e.g. having a terminal and a video output on the same VT), reconsider whether you really want to be in a VT or whether you're just reinventing compositors.

[dankamongmen]

Because it's a funny haha VO that also serves a role as a convenient minimal example of how the mpv video output stuff works. Hell, the similar caca video output right above it explicitly has a note stating that it's a joke.

got it.

The arguments I see for shoehorning graphical output into a terminal as coloured text are precisely those used by the abstraction violating sickos that gave us the abominable webpage-bundled-with-browser-as-an-application known as Electron™, and similar confirmations of Ted Kaczynski's beliefs about the evils of technology. We do not need to repeat history. If you want to remotely view graphical output, you should still use a protocol that operates on graphics, not text. If you want to view a video in a VT, use the drm API. If the drm API does not align with your use cases (e.g. having a terminal and a video output on the same VT), reconsider whether you really want to be in a VT or whether you're just reinventing compositors.

certainly a proper pixel-based solution is superior in just about every way. it cannot, however, run over ssh, in a screen, or on console.

in any case, i grok your responses, and understand that this will not be a good fit for mpv. thanks for giving it some consideration, and heading me off before i started work -- that's exactly why i posted here.

[Hrxn]

So, should we hold a vote to remove TCT, or something?

Because if it stays, it might be as well replaced by something better.

[haasn]

I vote in favor of expending zero energy.

[clort81]

Nice to meet dangamongmen who had the same basic idea a few years after I did.

I was more interested in using braille u2800-28FF.
A sensible terminal font renders braille as contiguous blocks.

This is because braille gives us 2x4 vs 2x3. Divides more evenly into most bitmap fonts (not as often divisible by three).
Yes, a sensible terminal font uses contiguous blocks for braille because [this might surprise you] -- blind people do not read braille on computer screens; these glyphs are more useful as pixels.
The more universally useful place to implement is in sdl2.

[CounterPillow]

@clort81 If you absolutely just need your video to be in a terminal window, then instead of using glyphs for weird perversions you could also either use the sixel VO (iirc based on escape sequences, so works through SSH et al I believe) or the new kitty VO (shared memory, so local only). Or god forbid, use a graphical environment for graphics.