How to get Parallel solving to work?

[sudsy]

I have managed to get some solutions working in serial mode and I'm happy to say that we have seen a boost in performance over the serial implementation of Xyce that we are currently utilizing.

We are hoping that we will be able to take advantage of the parallel solving capabilities of SpiceSharp but we are struggling to get an answer when enabling localSolver for our subcircuit. While enabling parallel without localsolver doesn't give any performance boost.

I have created a test below that demonstrates the problem. There are 2 places I have been experimenting with adding localsolver and I get "Could not determine the operating point." when I uncomment the line:

// newSubCircuit.Parameters.LocalSolver = true;

If you have any suggestions or guidance on how we might get some parallel solving working on a circuit like this, we would really appreciate it. I guess any other suggestions on how we could solve such a circuit more rapidly would be welcome too.

        [Fact]
        public void TestParallelSolve(){
            int childCircuits = 72;
            int parentCircuits = 28;
            double I01 = 0.00000000001;
            int m1 = 1;
            int ILbase = 8;
            double ILvary = 0.1;
            double Rs = 0.0025;
            int Rsh = 150;

            Random r = new Random(); // Should be declared at the topmost level

            List<IEntity> parallelComponents = new List<IEntity>();

            for (int parentCircuitIndex = 0; parentCircuitIndex < parentCircuits; parentCircuitIndex++)
            {
                string parentCircuitID = "ParentCircuit_" + parentCircuitIndex.ToString();
                string parentN1 = parentCircuitIndex.ToString() + "_" + "0";
                string parentN2 = parentCircuitIndex.ToString() + "_" + "1";
                string parentN3 = (parentCircuitIndex + 1).ToString() + "_" + "0";

                List<IEntity> parentEntities = new List<IEntity>();

                for (int childIndex = 0; childIndex < childCircuits; childIndex++)
                {
                    string childCircuitID = "Child_" + childIndex.ToString();
                    string Node1 = "CN_" + childIndex.ToString();
                    string Node2 = "CN_" + (childIndex + 1).ToString();

                    SubcircuitDefinition scDef = new SubcircuitDefinition(new EntityCollection(), new string[] { "posTerm", "negTerm" });

                    DiodeModel diodeModel = new DiodeModel("J1Diode");
                    diodeModel.Parameters.SaturationCurrent = I01;
                    diodeModel.Parameters.EmissionCoefficient = m1;
                    scDef.Entities.Add(diodeModel);
                    Diode newDiode = new Diode("D1", "1", "negTerm", "J1Diode");
                    scDef.Entities.Add(newDiode);
                    double IL = ILbase + (ILvary * r.NextDouble()) - (ILvary * r.NextDouble());
                    scDef.Entities.Add(new CurrentSource("iL", "negTerm", "1", IL));
                    scDef.Entities.Add(new Resistor("RS", "1", "posTerm", Rs));
                    scDef.Entities.Add(new Resistor("RSH", "1", "negTerm", Rsh));

                    Subcircuit scCircuit = new Subcircuit(childCircuitID, scDef, new string[] { Node1, Node2 });
                    parentEntities.Add(scCircuit);
                }

                parentEntities.Add(new VoltageSource("Vterm1", "CN_0", "parent_terminal_pos", 0));
                parentEntities.Add(new VoltageSource("Vterm2", "CN_" + (childCircuits).ToString(), "parent_terminal_neg", 0));

                SubcircuitDefinition newSubCircuitParentDef = new SubcircuitDefinition(new EntityCollection(), new string[] { "parent_terminal_pos", "parent_terminal_neg" });

                foreach (var item in parentEntities)
                {
                    newSubCircuitParentDef.Entities.Add(item);
                }

                Subcircuit newSubCircuit = new Subcircuit(parentCircuitID, newSubCircuitParentDef, new string[] { parentN2, parentN3 });
                // newSubCircuit.Parameters.LocalSolver = true;
                parallelComponents.Add(newSubCircuit);
                parallelComponents.Add(new Resistor("R" + parentCircuitIndex.ToString(), parentN1, parentN2, 1));
            }

            string lastNodeName = parentCircuits.ToString() + "_" + "0";
            parallelComponents.Add(new VoltageSource("Vterm1", "0_0", "parallel_terminal_pos", 0));
            parallelComponents.Add(new VoltageSource("Vterm2", lastNodeName, "parallel_terminal_neg", 0));

            Circuit solveCirc = new Circuit();
            SpiceSharp.Components.Parallel newParallel = new SpiceSharp.Components.Parallel("PL1", parallelComponents);
            TPLWorkDistributor workDistributor = new TPLWorkDistributor();
            newParallel.SetParameter("workdistributor", new KeyValuePair<Type, IWorkDistributor>(typeof(IBiasingBehavior), workDistributor));
            solveCirc.Add(newParallel);

            // Ground the negative lead (will not work without something grounded)
            solveCirc.Add(new Resistor("Rgnd", "parallel_terminal_neg", "0", 0));
            // Define the vterm for solving
            solveCirc.Add(new VoltageSource("vterm", "parallel_terminal_pos", "parallel_terminal_neg", 0));

            DC dc = new DC("DC1", "vterm", 0, parentCircuits * 55, parentCircuits * 55 / 100);

            RealCurrentExport currentExport = new RealCurrentExport(dc, "vterm");

            double maxPower = 0.0;

            dc.ExportSimulationData += (sender, args) =>
            {
                double input = args.GetVoltage("parallel_terminal_pos");
                double output = currentExport.Value;

                double power = input * output;
                if (power > maxPower)
                {
                    maxPower = power;
                }

            };

            System.Diagnostics.Stopwatch watch = System.Diagnostics.Stopwatch.StartNew();
            dc.Run(solveCirc);
            watch.Stop();
            Assert.NotEqual(0.0, maxPower);
        }

[svenboulanger]

I would have to take a closer look when I find the time. Thanks for pointing to this.

In the meantime, here what's supposed to happen:

Without local solver, only the methods for computing the model equations are parallelized. If most of the simulation time is spent on solving the matrix then there will not be a large benefit. This type of parallel computation is only something that benefits if you have many instances using complex model equations that take a long time. There is some overhead involved to avoid that instances write to the same matrix elements at the same time.

If you use a local solver, the simulator will go one step further and tries to make a smaller "local" matrix that can be solved in a separate thread. After the "local" matrix is solved, the computed contributions are passed to the "global" solver which then gets solved in turn. This "passing of contributions" is overhead, so this type of parallel computation is only something that benefits to situations where there are many internal nodes to the "local" solver circuit. If there are many nodes or variables shared with the outside of that local circuit, then the global solver is still forced to solve a rather large matrix and the speed benefit would be limited.

I have only been able to limited testing on the actual performance of parallel computations in Spice#, so I appreciate you sharing your findings.

[sudsy]

Thanks for your continued investigations into this. We tried a few things at our end and haven't progressed much at this stage either.

[svenboulanger]

No problem! 👍

I'm happy to report that I've managed to isolate the issue and I believe I have tracked what is going wrong with the local solver scheme. I made a bug report (#227) to track progress. I'll probably add some extra info to the docs if it gets resolved as well.

[svenboulanger]

The local solver works now in the development branch, but only for "regular" LU forward/backward substitution. There is still work left to do involving solving the adjoint matrix (used in noise simulations).

You can give it a spin if you don't mind cloning the development branch, knowing noise simulations will not work yet with local solvers. If you want to wait for an updated NuGet package I'll have you wait for a while longer...

[sudsy]

Thanks!

I finally had a chance to test this in our own environment and can confirm that i s working well. I can confirm that we see a good performance increase with parallel local solvers. We haven't run a detailed comparison but can see that it is important to find a balance between the amount of work each solver does and the total number of solvers to keep the performance good. We did see that in one situation where we broke the circuit down into too many small sub circuits that the performance was worse than serial solving. This is as you have described and completely understandable.

We need to do a little more work to put this into production so it will liekly be a little while. I will keep an eye out for your next nuget package, but it is not essential for our release.

[svenboulanger]

Thanks for the confirmation. That's great news!