0

I'm trying Python for the first time and am having a slowness issue. I wrote the program using Python 3.7 on Windows 10 64-bit. The program reads a 2.5GB text file on a server and splits it into 3 separate files on the local hard drive. It splits the file line by line based on the first character of each line.

On my Windows 10 machine it takes 25 minutes to complete, which I thought was long but it is a very large file.

I used pyinstaller to create an .exe file so users can run the program without installing Python (although I don't think this is part of the problem as it takes 25 minutes just running the .py from VS2017).

Two different users on Windows 7 machines can run the program in under 5 minutes, which is a vast difference from the 25 minutes on my Windows 10 box (and now verified on a 2nd Win10 box). I've done extensive searching online but haven't found anything that helps me determine the cause yet. Some people say to exclude Python directories from Windows Defender scans but my Defender is turned off (we use McAfee).

Could someone maybe point me in the right direction to look why it is taking so much longer on Windows 10? I'm happy to post the code but I do not think it is a coding issue as it still works perfectly on both OS. I've been developing professionally for ~15 years now in a multitude of languages but never had a discrepancy this big.

EDIT:

  • I have tested on two Win7 machines and two Win10 machines now. Both Win7's run the program in 5 minutes. Both Win10 machines run the program in 25 minutes.
  • The Win7 PC has an Intel i5-3470 CPU @ 3.20GHz with 8GB RAM
  • My Win10 PC has an Intel i7-6700 CPU @ 3.40GHz with 16GB RAM
  • I can't check Win7 PC at this time, but my Win10 HD is Solid State Drive Non-Encrypted 256 GB FLASH - NAND (TLC) 120 TBW SATA-3.2 6 Gb/s 2.5 Inch
  • I have run the same exe on both machines but it takes 5x longer on Win10
  • We are both on the same network and I ran AT&T's online network speed test. The Win7 and Win10 machines both registered ~100MB down.
  • I also turned the Win10 to High Performance power option and no improvement.
  • The Python code is not multithreaded as I'm only iterating through one file once.

Thanks.

Michael Bruesch
  • 632
  • 7
  • 23
  • 1
    Gonna _guess_ it's related to the issue I had [here](https://stackoverflow.com/questions/38554495/function-appears-to-run-faster-each-time-it-is-called-within-a-for-loop). Basically, the power management system is brutal in throttling the CPU even on a desktop. I bet they stepped it up a notch again now they're reshaping the OS for more portable devices. – roganjosh Dec 05 '18 at 19:41
  • Did you run the same exe file on both windows 7 and windows 10 machines? – Steven Dec 05 '18 at 19:41
  • Also, is the hardware the same? Are both using the same type of hard storage? What about CPU? Is the code multithreaded? – juanpa.arrivillaga Dec 05 '18 at 19:44
  • Did you test your internet connection on your computers? You said it read the files from the server. Do your windows 7 and windows 10 machines have comparable internet speed? – Steven Dec 05 '18 at 19:46
  • @Steven I updated my question with answers to yours – Michael Bruesch Dec 05 '18 at 21:08
  • 1
    @MichaelBruesch what about the hard drives? – juanpa.arrivillaga Dec 05 '18 at 21:09
  • Hard drive throughput is important and RAM. Maybe swapping is involved. – Thomas Weller Dec 05 '18 at 21:15
  • @juanpa.arrivillaga What exactly about the HD do you want to know? They are both NTFS Fixed hard disks. The Win7 is twice as large but the Win10 has 125GB free (out of 250GB). – Michael Bruesch Dec 05 '18 at 21:23
  • @juanpa.arrivillaga I have now tested in on 2 Win7's and 2 Win10's. The execution times are 5 minutes on both Win7's and 25 minutes on both Win10's. – Michael Bruesch Dec 05 '18 at 21:25
  • I don't know what your reply to me refers to? – roganjosh Dec 05 '18 at 21:30
  • Sorry @roganjosh, I mentioned you after updating the question in case you had more input. Just desperate for help lol. – Michael Bruesch Dec 05 '18 at 21:32

0 Answers0