0

I use an open source project called ChatScript for natural language processing app development.

When you execute a build operation with ChatScript, it scans all the script files that comprise your chat-bot. In my case, that's hundreds of files. This process takes nearly 30 times longer on Windows 8.1 than it does on Ubuntu 16.04. Therefore I do use Linux for much of my work, but there is a part of my work that I have to do on Windows because of certain associated tools, so I would like to modify the code base so that Windows ChatScript compiles are as fast as on Linux.

Can anyone think of a reason the code would run so much slower on Windows vs. Linux? Are there some C++ file operation codes (read/write/etc.) that are known to be much slower on Windows compared to Linux due to variances in the C++ run-time libraries running on each platform?

Robert Oschler
  • 14,153
  • 18
  • 94
  • 227
  • 1
    Are you sure you are using an *optimized* build of the code on both platforms? – Jesper Juhl Jan 20 '18 at 12:38
  • @JesperJuhl Pretty sure. I compile from source on both platforms. I am not using a pre-compiled binary. – Robert Oschler Jan 20 '18 at 12:52
  • 2
    just to be sure, go back and check your compiler options on Windows :) The only thing I can think of that could cause such a massive slowdown by just changing platform would be accidentally running an unoptimized debug build of the code (unless the hardware is also massively different or if the compiler used on Windows is ancient). – Jesper Juhl Jan 20 '18 at 13:08
  • @JesperJuhl Not a bad idea. – Robert Oschler Jan 20 '18 at 13:11
  • 1
    Are you using the same compiler on both systems? Same compiler settings, too? One suggestion: `std::ios_base::sync_with_stdio(false)`. This shouldn't be necessary in modern programs. On MS Windows, it might make more of a difference, because there you have CR/LF lineendings by default and the conversion is yet another step that needs to be done. – Ulrich Eckhardt Jan 20 '18 at 13:47
  • Stack Overflow is a site for programming and development questions. This question appears to be off-topic because it is not about programming or development. See [What topics can I ask about here](http://stackoverflow.com/help/on-topic) in the Help Center. Perhaps [Super User](http://superuser.com/) or [Unix & Linux Stack Exchange](http://unix.stackexchange.com/) would be a better place to ask. – jww Jan 20 '18 at 14:08
  • 1
    Be sure all your programs on all the platforms are in a Release configuration, including `-DNDEBUG`, `-O3` (Linux) and `/Oi /Oy /O2` (Windows). Maybe you can start with [windows terminal slower than linux](https://www.google.com/search?q=windows+terminal+slower+than+linux) and come back when you have something closer to programming and development. Also see [C++ cout printing slowly](https://stackoverflow.com/q/1736267/608639). When you run your command, redirect all output to a file to avoid the "slow terminal" print problem. – jww Jan 20 '18 at 14:11
  • @jww This question couldn't be more about programming and development. – Robert Oschler Jan 20 '18 at 14:21
  • @jww Thanks for the search link, I'll have a look. – Robert Oschler Jan 20 '18 at 14:22
  • @UlrichEckhardt Your comment about the CRLF is extremely interesting. – Robert Oschler Jan 20 '18 at 14:22
  • Anti-virus maybe? – James Jan 20 '18 at 14:31
  • @james Unlikely. I don't see the anti-malware service active. – Robert Oschler Jan 20 '18 at 14:33
  • 1
    I've also noticed this with ChatScript. Are you sure that it's to do with C/C++ and not the optimisation of CS itself? I've found that my Mac runs `:build` and `:verify` in a manner of seconds, and Windows (and even older Macs) can take minutes. Possibly due to CPU vs GPU processing? – Matt Fletcher Jan 22 '18 at 08:54
  • @MattFletcher I'm not sure. I am going to experiment further with the optimization settings as you and others have suggested. I didn't have much luck with that in the past but perhaps I'll find something new. – Robert Oschler Jan 22 '18 at 15:21

3 Answers3

3

By "code running slow" in your last paragraph I'm assuming from context that you're referring to the compiler???

I've encountered frequently and consistently over many years a general, significant performance difference between linux and Windows in disk I/O. NTFS (Windows file system) and the linux file systems handle the situation of lots of files differently, and linux is always quicker in the circumstances I've been in.

You may benefit from some of the pointers in answers to questions like How do I get Windows to go as fast as Linux for compiling C++?, like defragmenting your windows drive, and checking how the compiler optimisations are configured; some of them can slow down the compiler (although an aggressive compiler optimisation setting can slow the compiler down, you produce a faster executable at the end, but that might be something you switch to after most of your development is done).

But doing all those things for me has never got the Windows compile to be quicker than on linux using equivalent disk hardware, not once. If your code is on the one disk and sourced for both compiles, any improvement you'll see in the Windows build (e.g. because the code's put on an SSD) will likely be replicated in an improvement in the linux build as well.

greenbutterfly
  • 323
  • 2
  • 9
  • No I'm not. I'm referring to the disk heavy operation ChatScript does when *it* compiles the script files. During that operation, it opens and reads tons of script files. There is something about the code on the Windows build that interacts with files that is much slower than the same code on Linux. – Robert Oschler Jan 20 '18 at 14:24
  • OK, that's not clear from your question. It mentions (and tags) C++ and compiles, and comes across as asking re compiling ChatScript, rather than the compile that Chatscript is doing. Many of the options in the link to the other answer have nothing to do with C++ but are OS-related and hardware-related options which you could just as easily apply to your situation as for a C++ build, so I'd still recommend you check it out. Especially since it discusses large numbers of files on Windows in one of the proposed answers. – greenbutterfly Jan 21 '18 at 09:03
  • Just for reference: Since it appears to be about the performance of Chatscript, not about a language compiler, I agree with jww's comment on your question that this isn't a programming and development question. Yes, you might have encountered this situation during programming and development work, but the question itself is not about programming and development: it's about OS disk I/O and performance. You could remove the C++ tag without affecting the question at all, and IMO that's key as to if and how it fits within the SO framework. – greenbutterfly Jan 21 '18 at 09:09
  • Title "C++ *program* runs slower". I'm not sure how it could be clearer than that. "It's about OS disk I/O and performance". Possibly, but it could also be something that could be solved via the C++ library (e.g. - Full block read of file combined with CRLF delimited byte grabs from the loaded buffer, instead of using console line reads, etc.) There is no way for me to know that up front and with luck, someone could still provide a creative answer to this question. – Robert Oschler Jan 22 '18 at 15:07
  • 1
    Linux GCC is ridiculously fast for building C++ code. I don't know about these scripts but in my former company we had build servers for linux, OSX, and Windows builds. Linux and OSX used GCC (OSX with XCode against GCC). Windows used MSVC, and on the same hardware. Between the three, Linux was hands-down the fastest, building the code in about 10 minutes. Windows took about 35 minutes. OSX took about 2 hours. That said, we did the majority of our development on windows so I'm not sure if it's a fair comparison (something we did might have made the OSX build choke), but Linux was so fast. –  Jan 23 '18 at 15:09
1

Just to confirm I found the same thing. Ran the same Chatscript scripts on an average Mac and on a fast XPS 15. The Mac compiled the code 30-50 times faster than Windows. What's weird is ChatScript was originally developed for Windows. I also have not worked out why such a gigantic lag, in spite of the Windows PC hardware being much more powerful than the Mac running the script.

Leamsi
  • 51
  • 1
  • 6
0

So I have come across both an explanation and a (partial) solution. There are two areas of lag in Windows compared to Linux:

  • Networking behaviour. According to the creator of ChatScript, Bruce Wilcox, the Windows server code is worse under the hood, and is also implemented worse in ChatScript for Windows vs ChatScript for Linux. The lag here is however minor.

  • Build times. This is where building the bot in Linux takes 10-20 seconds and on Windows 4-5 minutes. Turns out that the reason is the antivirus:

"Curious about the huge discrepancy, one of our hardware engineers
did some profiling and found the real culprit to be the anti-virus
software. Disabling the real-time virus protection feature of
Windows Defender brought the 4 minutes down to 14 seconds! Even
keeping Windows Defender active, but excluding the
ChatScript-master folder solved much of the slowdown problem,
resulting in about 20 s for :build 0 to complete." (https://www.chatbots.org/ai_zone/viewthread/3575/)

So for OP, if you exclude CS from Windows Defender, or switch it off, the build differential will largely disappear.

Leamsi
  • 51
  • 1
  • 6