42

I am using TeamCity on ubuntu (TeamCity Professional 9.1.7 (build 37573)). And having some problems with automatic merges lately (git). I haven't changed anything in the configuration from when it used to work. I tried to remove the work directory. it only helped for a couple of merges.

Symptoms: The build log looks clean, and I am getting this on the build overview:

Failed to merge sources in VCS root foo. Merge error 'git fetch'     command failed.
stderr: git@bitbucket.org:foo/foo.git: channel is not opened.
exit code: 1.

Although, in the agent running this i am getting a totally different error in teamcity-vcs.log:

[2016-03-30 15:14:31,722]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git show-ref refs/remotes/origin/foo 
[2016-03-30 15:14:31,775]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git log -n1 --pretty=format:%H%x20%s 193f46d88205c5e419a8a7458e742ce9b598cca8 
-- 
[2016-03-30 15:14:31,797]   WARN -      jetbrains.buildServer.VCS - '/usr/bin/git log -n1 --pretty=format:%H%x20%s 193f46d88205c5e419a8a7458e742ce9b598cca8 --' command failed.
stderr: fatal: bad object 193f46d88205c5e419a8a7458e742ce9b598cca8 
[2016-03-30 15:14:31,798]   INFO -      jetbrains.buildServer.VCS - [/home/ubuntu/BuildAgent/work/7676127c0a691f42]: /usr/bin/git fetch --progress origin +refs/heads/foo:refs/remotes/origin/foo 
[2016-03-30 15:14:35,832]   WARN -      jetbrains.buildServer.VCS - Error output produced by: /usr/bin/git fetch --progress origin +refs/heads/foo:refs/remotes/origin/foo 
[2016-03-30 15:14:35,832]   WARN -      jetbrains.buildServer.VCS - remote: Counting objects: 2, done.ESC[K
remote: Compressing objects:  50% (1/2)   ESC[K
remote: Compressing objects: 100% (2/2)   ESC[K
remote: Compressing objects: 100% (2/2), done.ESC[K
remote: Total 2 (delta 1), reused 0 (delta 0)ESC[K
From bitbucket.org:bar/bar
   62ba378..193f46d  foo  -> origin/foo

As if the log command failed..

Thanks.

Update: I got an update that there is anecdotal evidence that this is due to changes specifically in BitBucket's git over SSH servers.

John
  • 29,788
  • 18
  • 89
  • 130
Oren
  • 589
  • 4
  • 10
  • 6
    We are seeing the same issue, only started today – John Jul 11 '16 at 13:28
  • 3
    +1 Just started seeing this in the last couple of days after I upgraded Ubuntu. I am running TeamCity on EC2 and using Git on BitBucket. Are you also running on EC2 or is this in your own data center? – Sohail Jul 11 '16 at 15:13
  • 1
    Getting this as well since a few days. I thought some sort of rate limiting BitBucket side. The thing I notice is that the errors are more frequent right after a push. My interval time is 30 seconds, usually the first check after a push fails while the following works. Self hosted on OVH. – namero999 Jul 11 '16 at 15:18
  • 2
    We started getting the same thing around Friday, July 8 (2016). Any idea how to address??? – TruncatedCoDr Jul 11 '16 at 15:20
  • I guess one way is to shift to alternate hosting and see if the error still occurs. It will be annoying. – Sohail Jul 11 '16 at 15:24
  • This is also reported on JetBrains System - https://youtrack.jetbrains.com/issue/TW-46052 – John Jul 12 '16 at 06:30

4 Answers4

23

Per https://youtrack.jetbrains.com/issue/TW-46052, the problem has been fixed. The fix will be shipped with the next release, either 9.1.8 or 10. If you don't want to wait for the release, you can manually update the git plugin by downloading it from here (logging in as a guest, or creating an account on JetBrains' build server) and replacing it in your installation.

  • Shut down TC
  • Put the downloaded file jetbrains.git.zip in %TEAM_CITY%/webapps/ROOT/WEB-INF/plugins replacing the existing one
  • Restart TC

This fixed the problem for me and others.

namero999
  • 2,882
  • 18
  • 24
22

TeamCity used ssh multiplexing which was recently disabled in Bitbucket Cloud. If you are affected by this problem, please watch the https://youtrack.jetbrains.com/issue/TW-46052, fixes will be posted there (at the moment of writing there is a fix for TeamCity 9.1.x and 9.0.x). A workaround without installing a newer git-plugin is to use https or anonymous protocol instead of ssh.

Update

The initial fix caused changes collection to fail with timeout errors in TeamCity 8.1.x and 9.0.x, this is fixed now.

neverov
  • 1,283
  • 8
  • 11
  • Falling back to use https instead of ssh, quick and easy fix. The plugin addition only works for some build configurations, one worked for me and one kept on getting stucked in Collecting changes in vcs root – Joshscorp Jul 13 '16 at 02:23
  • 1
    No, falling back to HTTPS does not work for those of us with large, large bitbucket repos that are several years old. SSH worked fine for this. Shame on Bitbucket and Atlassian. – danieljimenez Jul 13 '16 at 13:00
  • [This link](https://youtrack.jetbrains.com/issue/TW-46052#comment=27-1515350) is the right place to download (in my case it was 8.1.4). And you just need to drop it into the `/plugins` directory (despite the fact it is already bundled, this will override the bundled copy). – NightOwl888 Aug 30 '18 at 12:26
10

Updated: JetBrains has released a patch for this issue - see https://youtrack.jetbrains.com/issue/TW-46052 for details.

KlimczakM
  • 12,576
  • 11
  • 64
  • 83
Benjamin Echols
  • 109
  • 3
1

I've ran into this problem twice already and it got annoying.

For linux users which properly installed TeamCity (e.g. have a separate user and a teamcity service), this script should do all the heavy lifting (given you have sudo):

function patch_teamcity_plugin() {
  # Find teamcity plugin dir.
  sudo updatedb
  path_part="webapps/ROOT/WEB-INF/plugins"
  teamcity_plugin_dir=$(locate $path_part | grep -o -E ".*$path_part" | head -n 1)

  if [ "$teamcity_plugin_dir" == "" ]; then
    echo "Cannot find teamcity plugins directory." 1>&2
    return 1
  fi

  login_url="https://teamcity.jetbrains.com/guestLogin.html?guest=1"
  cookie_file=$(mktemp)

  # Log in as guest.
  # Wget login - thanks to: http://stackoverflow.com/a/1432161/2041634
  wget --save-cookies $cookie_file --keep-session-cookies $login_url

  # Download the plugin to the teamcity plugin directory.
  plugin_url="https://teamcity.jetbrains.com/repository/download/TeamCityPluginsByJetBrains_Git_JetBrainsGitPluginTeamCity91x/843194:id/jetbrains.git.zip"
  plugin_dest_filename="$teamcity_plugin_dir/jetbrains.git.zip"
  sudo wget --load-cookies $cookie_file $plugin_url -O $plugin_dest_filename

  if [ "$?" != "0" ]; then
    echo "Failed download of plugin." 1>&2
    return 1
  fi

  # Copy permissions - thanks to: http://unix.stackexchange.com/a/20646
  sudo chown --reference="$teamcity_plugin_dir" "$plugin_dest_filename"
  sudo chmod 0755 "$plugin_dest_filename"

  # Restart TeamCity server.
  sudo service teamcity restart
}

patch_teamcity_plugin()
kburnik
  • 359
  • 3
  • 6