This answer is going to take advantage of a few advanced concepts with Biml. The first will be tiering so we will generate our 150 packages in tier 1. Then in tier 2 (or any number greater than the previous tier), we will be able to reference the fruits of our labor in tier 0 to (tier max -1).
Tier 0 is static/flat Biml (which we don't have any in this example). Since we'll be looping to generate the child packages, it will automatically be in tier 1 but I chose to be explicit here in case you have precursor, yet dynamic, tasks to solve
<#* .... #> is a super powerful comment construct that is ignored by the Biml compiler
<#+ .... #> is construct for adding explicit methods and classes into your Biml
We will also use an Extension method to approximately split our packages into groups to then stuff into Sequence Containers.
Tier 1
This code generates blank SSIS packages with names ranging from so_54773502_000 to so_54773502_149 (150 in total)
<#@ template tier="1" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<#foreach (int index in Enumerable.Range(0, 150)){#>
<Package Name="so_54773502_<#= string.Format("{0:000}", index) #>" />
<#}#>
</Packages>
</Biml>
Tier 2
In here we specify how many parallel containers we want. The results of the Split method is a list of lists. For each element in the outer container list, we need to add a Sequence Container. For each element within that popped list, we need to enumerate through it and execute a package task.
<#@ template tier="2" #>
<#
int taskCountPerContainer = 10;
int currentContainerNumber = 0;
#>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<Packages>
<Package Name="master">
<Tasks>
<#*
This is such a fun bit of LINQ. I used the Split extension so handily defined on the linked question
I pass in a List of package names and how many buckets I want and it returns a list of lists
From there, we enumerate through the list bucket and for each element we find, we create a sequence
container. Then, for each element in the bucket, we add an Execute Package Task.
I was unable to Split an instance of AstPackageNodes as it resulted in the error below but the only reason
your sample needed the full object was to provide both Name and PackageFileName properties. We can derive
the second given the first
// foreach(var outerlist in Split(this.RootNode.Packages.ToList<AstPackageNode>(),taskCountPerContainer)){
// results in this error.
// Destination array was not long enough. Check destIndex and length, and the array's lower bounds.
// TODO: Check with Varigence or run in C# project
*#>
<#foreach(var listOfPackages in Split(this.RootNode.Packages.Select(x => x.Name).ToList<string>(), taskCountPerContainer)){#>
<Container Name="SEQC_<#=currentContainerNumber++#>" ConstraintMode="Linear">
<Tasks>
<#foreach(var packageName in listOfPackages){#>
<ExecutePackage Name="EPT <#=packageName#>"><ExternalProjectPackage Package="<#=packageName#>.dtsx"/></ExecutePackage>
<#}#>
</Tasks>
</Container>
<#}#>
</Tasks>
</Package>
</Packages>
</Biml>
<#+
// https://stackoverflow.com/questions/419019/split-list-into-sublists-with-linq
public static IList<List<T>> Split<T>(IList<T> source, int buckets)
{
return source
.Select((x, i) => new { Index = i, Value = x })
.GroupBy(x => x.Index / buckets)
.Select(x => x.Select(v => v.Value).ToList())
.ToList();
}
#>
End result
Look ma, lots of packages getting executed!
