SharePoint Solution Package Deploy Random Failures

Question

I keep seeing this error sporadically during solution deployment and/or feature activation...

Copying of this file failed . This operation uses the SharePoint Administration service (spadmin), which could not be contacted. If the service is stopped or disabled, start it and try the operation again.

The following is the PowerShell script used to deploy:

$programfiles_path = $null
$stsadm_path = $null
$url = $null
$solution = $null
$feature = $null
$appPoolName = $null



$url = "http://" + ${env:COMPUTERNAME}
$programfiles_path = ${env:ProgramW6432}

if ($programfiles_path -eq $null)
{
    $programfiles_path = ${env:ProgramFiles}
}

$stsadm_path = $programfiles_path + "\common files\microsoft shared\web server extensions\12\bin\stsadm.exe"

if(!(test-path $stsadm_path))
{
    echo "stsadm.exe was not found: $stsadm_path"
    exit -1
}


ECHO "***********************************"
ECHO "*** Contoso.Common.SolutionPackage ***"
ECHO "***********************************"

$solution = "Contoso.Common.SolutionPackage.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.Common.Infrastructure"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.CustomCAS"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.Common.WebParts"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.ExcelExtender"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }



ECHO "*********************************"
ECHO "*** Contoso.Something ***"
ECHO "*********************************"

$solution = "Contoso.Something.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.Something"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }




ECHO "*********************************"
ECHO "*** Contoso.Nothing ***"
ECHO "*********************************"

$solution = "Contoso.Nothing.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = $null










ECHO "*****************************************"
ECHO "*** Contoso.Everything ***"
ECHO "*****************************************"

$solution = "Contoso.Everything.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.Everything"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }




ECHO "*****************************"
ECHO "*** Contoso.CashMoneyMillionaires ***"
ECHO "*****************************"

$solution = "Contoso.CashMoneyMillionaires.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.CashMoneyMillionaires"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }





ECHO "************************************"
ECHO "*** Contoso.DollaDollaBill ***"
ECHO "************************************"

$solution = "Contoso.DollaDollaBill.wsp"

ECHO "*** add the solution ***"
& $stsadm_path -o addsolution -filename $solution 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

ECHO "*** deploy the solution ***"
& $stsadm_path -o deploysolution -name $solution -immediate -allowgacdeployment -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }

$feature = "Contoso.DollaDollaBill"

ECHO "*** activate feature(s) ***"
& $stsadm_path -o activatefeature -name $feature -url $url -force 
if ($? -eq $false) { exit }

ECHO "*** execute admin jobs ***"
& $stsadm_path -o execadmsvcjobs        
if ($? -eq $false) { exit }



ECHO "************************"
ECHO "*** IIS APPPOOL MGMT ***"
ECHO "************************"

ECHO "*** query wmi for apppool ***"
$appPoolName = "SharedServices1"
$appPool = get-wmiobject -namespace "root\MicrosoftIISv2" -class "IIsApplicationPool" | Where-Object {$_.Name -eq "W3SVC/APPPOOLS/$appPoolName"}
if ($? -eq $false) { exit }

ECHO "*** recycle apppool ***"
$appPool.Recycle()
if ($? -eq $false) { exit }

I have an equivalent batch file which also randomly fails. It fails on different servers. I am quite baffled as to the cause and solution.

Any help would be appreciated.

Based on comments here, I am opening a support ticket with MSFT to be certain of the nature and resolution. I will post the result. — Daniel P. Bullington, May 13 '09 at 16:20
So, weeks later, Microsoft has no clue. What do large SharePoint farms do for deployment? MS is recommending making the farm account local admin or deploy manually or IISRESET every step of the way... — Daniel P. Bullington, Jun 12 '09 at 18:07
We use the default setup batch file generate by VseWSS to deploy on each server and IISRESET. This is the recommended deployment method for quite a while. We have automated the deployment by stored in the files in a central repository and have a dedicated setup login for each SharePoint server. — BinaryMisfit, Jul 01 '09 at 07:53
In my experience, the issue with spadmin service being unavailable seems to be connected with farm account being not a domain account and not local admin on all the farm servers, as you noted. — axk, Jul 27 '09 at 17:21

Chris Dibbs · Answer 1 · 2009-05-07T17:59:02.013

7

The best thing that I have found to do is have a Power Shell that will wait for a pending timer job to complete before moving on to the next task, you can but a pause in however that requires you to check to make sure the time job is finished.

The trick is to poll the server using stsadm -o displaysolution -name . if the wsp is installed on the farm the command returns some xml, if there is a timer job for deployment there is a "SolutionDeploymentJob" tag in the xml. With a simple loop you can wait for a time job to finish

Function WaitForJobOn($name){
    [xml]$val = stsadm -o displaysolution -name $name
    while($val.Solution.SolutionDeploymentJob) {
        $log.Info("waiting for a job on $name")
        sleep 5
        [xml]$val = stsadm -o displaysolution -name $name
    }
    $log.Info("Job Compleated")
}

usage: WaitForJobOn somefeature.wsp

it should be noted that the function will fail if the wsp is not added to the farm.

edited May 07 '09 at 17:59

answered May 06 '09 at 18:11

Chris Dibbs

81
3

This seems so heavy handed. Is SharePoint this brittle when it comes to deployment? I mean, there has to be a way to serialize deployment tasks in an enterprise manner...? – Daniel P. Bullington May 07 '09 at 18:52
That all depends on how far out side the box of SharePoint that you get. If all that you want to do is add new things to the farm - all is well. However the night that you need to retract and redeploy a solution there can be issues. The problem is that is the only way that I have found a page can be removed is to retract and redeploy the solution. stsadm -o execadmsvcjobs only flushes the queue of jobs, so if you have other jobs that depend on the time jobs being finished (ie a delete solution after a retract solution) this issues comes up. – Chris Dibbs May 26 '09 at 14:10
1

Nice observation that stsadm return xml and this can be easily parsed with powersell's built in xml functionality. – axk Jul 27 '09 at 17:17

Trent · Answer 2 · 2009-09-03T14:47:45.783

I am also running into this problem. The only way I've gotten around it is to restart the Windows SharePoint Services Administration service each time before a deployment/retraction.

UPDATE: I rebooted the WFE server and this problem has disappeared. I wish I had more info why, but the classic Microsoft reboot solution prevails again.

It is very annoying.

ANOTHER UPDATE: I just released an stsadm command called smartexecjobdefs that addresses this issue. In this blog post, I talk about a Job Definition Executor that monitors for the completion of running jobs before returning control. This stsadm command simply wraps this class.

score 0 · Answer 3 · answered Jun 30 '09 at 08:21

I've also had this problem. Every time I tried to deploy the solution some of the files were copied then it stopped with this error before copying all of them. I don't have any evidence but I believe the anti-virus software on the server was locking the files and interfering.

It eventually worked after I attempted to deploy the solution several times (each time copying a few more files than the time before).

score 0 · Answer 4 · answered Jul 01 '09 at 07:34

I find Windows SharePoint Administration Service (SPAdmin) fails sporadically in sp1 when used in a SharePoint farm with multiple web front end servers.

I suggest disabling spadmim and run stsadm -o execadmsvcjobs using remote execution tool such as psexec.

Example:

psexec.exe @serverlist.txt "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm.exe" -o execadmsvcjobs

Greg Hurlman · Answer 5 · 2009-05-07T13:38:30.437

0

You need to pause between deploying your solution and trying to activate your features - it's a bit of a race condition.

More: If there are several web front ends, and connectivity is slow, or the servers are busy, or any number of other things that cause this sort of thing to take some time - then you run into trouble. Batch files can't deal with it - a Powershell script could use the object model to check if a solution is truly deployed before moving on.

edited May 07 '09 at 13:38

answered May 05 '09 at 20:20

Greg Hurlman

17,666
6
54
86

I thought the who purpose of execadmsvcjobs was to sync up the execution of admin functions...? – Daniel P. Bullington May 05 '09 at 20:43

SharePoint Solution Package Deploy Random Failures

5 Answers5