4

problem arises when using mosestokenizer (https://github.com/luismsgomes/mosestokenizer, installed through pypi https://pypi.org/project/mosestokenizer/) on windows platform (tried multiple windows machines).

Although there was no new releases during last month, before it used to work fine. I believe it has something to do with communication between python and perl.

>> MosesSentenceSplitter('en')
stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "...\.venv\lib\site-packages\mosestokenizer\sentsplitter.py",
    line 82, in __init__
        super().__init__(argv)
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 64, in __init__
        self.start()
    File "...\.venv\lib\site-packages\toolwrapper.py",
    line 102, in start
    self.proc = subprocess.Popen(
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
    File "C:\Users\MyUser\AppData\Local\Programs\Python\Python38-32\lib\subprocess.py",
    line 1307, in _execute_child
        hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

In order to reproduce it all it takes is to create some moses* class object:

from mosestokenizer import MosesSentenceSplitter
splitsents = MosesSentenceSplitter('en')  # and here error appear.
ikegami
  • 367,544
  • 15
  • 269
  • 518
Simon
  • 55
  • 3
  • How did you install the package on Windows ? Can you show a complete Python script? See [mcve] for more information – Håkon Hægland Jul 27 '20 at 10:45
  • 1
    Yes I can reproduce the error message `stdbuf was not found; communication with perl may hang due to stdio buffering.` on Windows 10, Python version 3.8.5. I am looking into it. – Håkon Hægland Jul 27 '20 at 11:36
  • 1
    The warning comes from [line 84](https://github.com/luismsgomes/toolwrapper/blob/master/src/toolwrapper.py#L84) in `toolwrapper.py` when it cannot find the `stdbuf` executable – Håkon Hægland Jul 27 '20 at 11:52
  • 1
    I think you can install `stdbuf` from MSYS2 (the package name is `coreutils`) – Håkon Hægland Jul 27 '20 at 12:00
  • @Håkon Hægland identified the source of the "stdbuf was not found" warning, but I think there's a second program that can't be found resulting in the fatal error, and I think [that program is `perl`](https://github.com/bkj/mosestokenizer/blob/master/src/mosestokenizer/sentsplitter.py#L64). – ikegami Jul 27 '20 at 12:28
  • 1
    ([This](https://github.com/bkj/mosestokenizer/blob/master/src/mosestokenizer/split-sentences.perl) should have `$| = 1;` to avoid the need for `stdbuf`.) – ikegami Jul 27 '20 at 12:30
  • Apparently, perl was not installed. It is weird, because one month I was using the very same thing on the same envirnoment, which did not changed. – Simon Jul 27 '20 at 13:21

1 Answers1

0

Installing perl fixed the error.

Also, as @ikegami mentioned, This should have $| = 1, otherwise you will get stdbuf warnings.

Simon
  • 55
  • 3