Unicode failure

[N7DR]

If I have an ordinary gnuplot script file that contains, say, the line:
set xlabel "β(°)"
Then, when executed, gnuplot does the expected and labels the x axis with the text "β(°)".

If I put exactly the same text into a cell that is run with the gnuplot jupyter kernel, then both of the non-ASCII characters appear in the label as garbled.

The same failure is apparent in titles.

[has2k1]

I think it is an issue with your installation. Trying out your unicode string with this notebook gives me unicode in the output.

[N7DR]

Hassan Kibirige wrote on 5/10/20 1:46 PM:

I think it is a issue with your installation. Trying out your unicode string with [this notebook](https://github.com/has2k1/gnuplot_kernel/blob/master/examples/gnuplot-kernel.ipynb) gives me unicode in the output. 

Attached is what I see. FWIW, Unicode works fine in all the other kernels I've tried. Doc

…

-- Web: http://enginehousebooks.com/drevans

[has2k1]

I don't see anything attached!

What version of python is it?

[N7DR]

Hassan Kibirige wrote on 5/10/20 6:07 PM:

I don't see anything attached!

I guess that the reflector usefully removes attachments. I'll put it somewhere publicly accessible and send you the URL as soon as I get a chance to do so. Later today.

What version of python is it?

[ZB:gnuplot] python --version Python 3.7.3 [ZB:gnuplot] Doc

…

-- Web: http://enginehousebooks.com/drevans

[N7DR]

D. R. Evans wrote on 5/11/20 7:59 AM:

I'll put it somewhere publicly accessible and send you the URL as soon as I get a chance to do so. Later today.

https://www.adrive.com/public/yFJpnM/Screenshot_20200510_162329.png Doc

…

-- Web: http://enginehousebooks.com/drevans

[has2k1]

Okay, I have seen the same issue at binder.

[N7DR]

Hassan Kibirige wrote on 5/12/20 7:27 AM:

Okay, I have seen the same issue at [binder](https://notebooks.gesis.org/binder/jupyter/user/has2k1-gnuplot_kernel-946avlym/notebooks/examples/gnuplot-kernel.ipynb).

OK. I get a 403 Forbidden at that URL, but I'm glad someone else is seeing the problem and it isn't just me :-) Doc

…

-- Web: http://enginehousebooks.com/drevans

[has2k1]

That link did work, but I tried it https://mybinder.org/v2/gh/has2k1/gnuplot_kernel/master?filepath=examples and I got the exact garbled output as in https://www.adrive.com/public/yFJpnM/Screenshot_20200510_162329.png.

After some debugging on the containers at binder , I have tracked the failure to pass on unicode to somewhere between the pexpect library (that is part of the requirements) and the gnuplot installation. For my system with the same versions of pexpect and gnuplot, unicode works well. For the binder system, pexpect with sh can pass between themselves unicode strings, while pexpect with gnuplot mess up the unicode handling

A snapshot of the debugging

Code

# Send some unicode string to the shell and see what it prints
import pexpect
spawn_cmd = '/bin/sh'
child = pexpect.spawnu(spawn_cmd, echo=False)
cmd = 'echo "β(°)"'
child.sendline(cmd)
child.readline()
res = child.before
res

# Send some unicode string to gnuplot and see what it prints
import pexpect
spawn_cmd = '/usr/bin/gnuplot'
#child = pexpect.spawn(spawn_cmd, encoding='utf-8')
child = pexpect.spawnu(spawn_cmd, encoding='utf-8')
cmd = 'print "β(°)"'
child.sendline(cmd)
child.readline()
res = child.buffer.split()[-2]
res

[N7DR]

Hassan Kibirige wrote on 5/12/20 2:56 PM:

That link did work, but I tried it https://mybinder.org/v2/gh/has2k1/gnuplot_kernel/master?filepath=examples and I got the exact garbled output as in https://www.adrive.com/public/yFJpnM/Screenshot_20200510_162329.png. After some debugging on the containers at [binder](https://mybinder.org/v2/gh/has2k1/gnuplot_kernel/master?filepath=examples) , I have tracked the failure to pass on unicode to somewhere between the `pexpect` library (that is part of the requirements) and the gnuplot installation. For my system with the same versions of `pexpect` and gnuplot, unicode works well. For the binder system, *`pexpect` with `sh`* can pass between themselves unicode strings, while *`pexpect` which gnuplot* mess up the unicode handling

I am using vanilla up-to-date debian stable, and it reports that python3-pexpect is version 4.6.0-1 (which I think really means 4.6.0 --- the -1 suffix is a debian packaging number, I believe). Doc

…

-- Web: http://enginehousebooks.com/drevans

[has2k1]

python3-pexpect is version 4.6.0-1

I have ruled out pexpect and its only dependency ptyprocess by testing the minimum required versions pexpect-4.2.0 and ptyprocess-0.5.0 and my system. With those versions unicode strings are handled correctly.

That effectively rules out the python packages, but not the installed CPython (esp. it's compilation environment) and the general OS environment. With that I am out of moves.

[N7DR]

Hassan Kibirige wrote on 5/13/20 7:19 PM:

> python3-pexpect is version 4.6.0-1 I have ruled out pexpect and its only dependency ptyprocess by testing the minimum required versions pexpect-4.2.0 and ptyprocess-0.5.0 and my system. With those versions unicode strings are handled correctly. That effectively rules out the python packages, *but not* the installed CPython (esp. it's compilation environment) and the general OS environment. With that I am out of moves.

I don't know what CPython is, but in any case it's not installed here. Maybe it's a debian issue of some kind, but I'd be very surprised if so. I routinely use Unicode in all kinds of places on this system, and this is the first time I've seen a failure (excepting when using unicode variables in C++ programs, and that's because the version of g++ currently packaged with debian doesn't fully support unicode characters in variable names, so not really an OS issue. I don't know if there's any other information I could give you to help further. It sounds like I'll just have to stick with not using the gunplot kernel in notebooks :-( I can work around that by using bash and then, in the notebook cell, piping the gnuplot code into a script, and then executing that script. That works fine, but is obviously much uglier than using a gnuplot kernel. Doc

…

-- Web: http://enginehousebooks.com/drevans