Poor performance with LTSpice + Wine on multicore Linux machine

Question

I'm trying to build an ML model using LTSpice simulation data. I'm running 64 concurrent simulations of LTSpice using Wine on a 64-core Centos linux machine to acquire ideally several million data points. A typical simulation takes about 2-4 seconds. However I'm finding that after a few hundred concurrent simulations, the server slows down significantly and simulation times increase to 120+ seconds.

I'm looking for some advice on how to improve performance and speed up the simulation times. From what I can tell, performance degrades after several hundred simulations of LTSpice. However, individual memory usage for each simulation is not massive, but perhaps in total it's hosing up the system. I ran top to see what the process memory usage is and I don't see CPU or memory usage too far out of whack. My thought is that perhaps Wine or LTSpice are leaking memory, but I can't pinpoint the issue.

Here is what I've tried so far:

1. LTSpice simulations are run in batch mode, each simulation in a separate directory. I delete all intermediate files (both LTSpice and Wine) after completing each simulation. Here is the command I'm running:

WINEDLLOVERRIDES=winemenubuilder.exe=d WINEDEBUG=-all wine-preloader /usr/local/bin/wine ~/.wine/drive_c/Program\\ Files/LTC/LTspiceXVII/XVIIx64.exe -Run -b [ltspice_file.asc]

2. I use a virtual frame buffer (XVFB) to output any display windows. This had some significant impact on performance.

export DISPLAY=:1
Xvfb :1 -noreset &

3. I was using python initially, but found that performance was terrible, so I switched to C++ multi-threading, which improved performance significantly. I'm using OMP and dynamic scheduling to allocate simulations. Note that each loop is based on a fixed chunk size of 500 simulation configurations, running across 64 threads.

        #pragma omp parallel for schedule(dynamic,1)
        for ( int i = 0; i < (int)sim_config.size(); i++ )
        {
            #pragma omp critical
            {
                ++pid;
            }

            // run the LTspice simulation using configuration settings
            run_simulation(pid, sim_config[i]);
        }

4. Every 500 simulations, I kill the wineserver (and related windows exe files) and reset the Xvfb application to clear memory:

system("wineserver -k; pkill .+\\.exe; pkill Xvfb; Xvfb :1 -noreset &")

5. I've also tried flushing the linux cache memory:

sync; echo 3 > sudo /proc/sys/vm/drop_caches && sudo swapoff -a && sudo swapon -a

6. I tried running less threads and smaller chunk sizes, but that doesn't seem to improve performance.

I appreciate any feedback or thoughts on how / where I can reduce the memory leakage.

Thanks!

Answer 1

Since the LTSpice simulation is fairly short, and assuming the memory issues are caused by many LTSpice invocations+destructions, you should try adding many independent simulations in the same netlist file by duplicating your circuit with the different parameters.

Probably best to define your original circuit as a sub-circuit, depending how you're generating the netlist.

As MicroservicesOnDDD has mentioned in their answer, if you're not relying on LTSpice-only features you should consider using ngspice (PySpice is a nice Python interface for it) as ngspice can run as a server/daemon controlled by pipes rather than outputting to disk like LTSpice does.

Answer 2

Here's a few thoughts for you. I'm an old-school Senior Dev.

First major step is to remove WINE from the equation, along with Linux. Temporarily spin up a Windows machine in the Cloud and reproduce your problem there. If the problem still exists, you know it's probably LTSpice, but for sure that it is probably not WINE and probably not Linux. If it is WINE or Linux, then play with the WINE settings and see if anything changes. If Windows performs admirably, then you can consider spinning up a bunch of machines in the cloud to implement your necessary concurrency, and I'll bet that it won't be too expensive for you, but do it in stages so that you're not surprised when the charges appear on your credit card. Unfortunately, I'm really a Windows developer, so I'm not as knowledgeable about WINE and Linux, and therefore I'll have to leave further work on those layers up to you and others here that you can ask.

If the problem still exhibits on Windows, there are a lot of metrics you can look at right away without doing much special -- in Task Manager, you just have to display the columns you want to see. Do that and hopefully something will jump out at you. (I'm answering this question somewhat generically for all who arrive here.)

Here is what my Task Manager window looks like on my Windows 10 Pro machine. By right-clicking on the column header labels, you can choose what operating system metrics you want to see:

And I've pasted together screen shots from the dialog (which, frustratingly, won't resize) so you can see the entire list of metrics. The complete list is here:

My off-the-cuff list: PID, CPU Time, File Handles, Threads, Memory (active private working set), Page Faults, Page Fault delta (PF Delta), Image Name. My other settings you can see check-marked in the above pic.

If all else fails, consider using a different Spice or other circuit simulator, because there are a lot of them out there: List of Free Electronics Circuit Simulators. I'm not specifically familiar with it, but based solely on the above Wikipedia page, the following looks good: Qucs-S (Qucs with Spice) works on Windows, MacOS, and Linux, and the Wikipedia page states that Qucs-S also runs both VHDL and Verilog, which aren't just for FPGA's, but are also for mass testing of large systems of electronics. You'll have to also install NgSpice or some other engine, and possibly other packages. The advantage of going the NgSpice way is that it looks like it scales more naturally. (See https://ngspice.sourceforge.io/extras.html) under "Programming ngspice", especially the last item "The ASCO optimizer". In addition, you will of course have total control over the source code, so you might even be able to make a minimal-sized Linux app that does only what you want, and hopefully scales even better.