Issues with upgrades

mturchi's Avatar

mturchi

17 Apr, 2017 12:41 PM

We have noticed with the last couple of recent upgrades that our remote tentacle are not upgrading cleanly. We are seeing the following with large percentage of servers. We are manually restarting the agents to resolve the issues at this time.

Removing the 'D:\Octopus\Upgrade\20170417122809-W7NDQ' directory
April 17th 2017 08:28:22
Info
The process cannot access the file because it is being used by another process.
April 17th 2017 08:28:22
Fatal
The remote script failed with exit code 65535

  1. Support Staff 1 Posted by Shane Gill on 17 Apr, 2017 11:23 PM

    Shane Gill's Avatar

    Hi,

    Thanks for getting in touch. Could you please send the raw task log for your Tentacle upgrade? (https://octopus.com/docs/how-to/get-the-raw-output-from-a-task)

    Cheers,
    Shane

  2. 2 Posted by rodolfograve on 20 Apr, 2017 07:38 AM

    rodolfograve's Avatar

    We have also experienced this issue when upgrading to 3.12.5.0. Several (not all) of our machines went into this strange mode, marked as "Unhealthy".

  3. Support Staff 3 Posted by Shane Gill on 20 Apr, 2017 11:12 PM

    Shane Gill's Avatar

    Hi,

    We made some improvements to Tentacle upgrades in 3.12.6: https://github.com/OctopusDeploy/Issues/issues/3413

    Please let me know if it continues to happen.

    Cheers,
    Shane

  4. 4 Posted by mturchi on 21 Apr, 2017 05:59 PM

    mturchi's Avatar

    With the recent upgrade to 3.2.14 we are getting the following error with the tentacle upgrade.

    This server does not have version 3.7.53 of the Calamari package. It will be pushed automatically.

    April 21st 2017 13:55:49

    Info
    Calamari upgrade successful

    April 21st 2017 13:55:56

    Info
    Running Tentacle version 3.12.4

    April 21st 2017 13:55:56

    Info
    Beginning upgrade

    April 21st 2017 13:56:04

    Info
    Waiting for the upgrade to complete

    April 21st 2017 13:56:06

    Fatal
    The remote script failed with exit code 65532

  5. 5 Posted by tmboyd1 on 27 Apr, 2017 08:31 PM

    tmboyd1's Avatar

    Getting this on 3.12.8 for a handful of machines. Happened on a lot at first, but clicking "Try Again" fixed most of them.

  6. Support Staff 6 Posted by Shane Gill on 28 Apr, 2017 04:57 AM

    Shane Gill's Avatar

    Hi,

    Thanks for reporting. Would it be possible to get the raw log from the Tentacle upgrades that are failing?

    Thanks
    Shane

  7. 7 Posted by sbethea on 28 Apr, 2017 11:22 AM

    sbethea's Avatar

    Attached a snippet of the raw log output for two machines - one successful and one failed (some info redacted). In the UI, the Connectivity tab for these failed machines initially shows a Current Version of "0.0.0 (upgrade available)".

    It looks like the failure occurs after the upgrade when EnsureSuccessful() tries to find an exit code log. So far, it looks like all of our upgrades were actually successful. After running a health check, the Current Version is correct and the machine status is healthy.

  8. Support Staff 8 Posted by Shane Gill on 03 May, 2017 11:23 AM

    Shane Gill's Avatar

    Thank you.

    It seems the process that performs the Tentacle upgrade is returning exit code 65532 but I haven't figure out why that is happening. Unfortunately I am finding it very difficult to reproduce.

    I have added a few tweaks and some extra logging to the upgrade process. I expect the error will be encountered again but the Tentacle upgrade should succeed.

    Cheers,
    Shane

  9. 9 Posted by mturchi on 03 May, 2017 01:37 PM

    mturchi's Avatar

    I will let you know the outcome after we update in the near future.

    Mike

  10. 10 Posted by Mayank Kumar on 07 May, 2017 09:02 AM

    Mayank Kumar's Avatar

    Hello,

    I have upgraded today to 3.13.0 and encountered the same error while upgrading the tentacles.

  11. 11 Posted by dpoland on 08 May, 2017 05:48 PM

    dpoland's Avatar

    I just performed an upgrade from 3.11.5 to 3.13.0 this morning and out of about 200 tentacle machines we had about 15 or so fail with this particular error.

    Having it try to upgrade again worked successfully afterwards.

    Here is a sample log entry of one that failed:
    09:17:16 Verbose | Host Name: <redacted>

    09:17:16 Verbose | Running As: <redacted> (Local Administrator: True)

    09:17:17 Verbose | Package: C:\Octopus\Work\20170508141715-775\Tentacle.nupkg

    09:17:17 Verbose | Destination: C:\Octopus\Upgrade\20170508141702-DTY9T

    09:17:17 Verbose | Octopus Deploy: Tentacle version 3.11.5 (3.11.5+Branch.release-3.11.5.Sha.2ec95d93ccfb115671522932f84809b9bb4f3dca) instance Default

    09:17:17 Verbose | Environment Information:

    09:17:17 Verbose | OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1

    09:17:17 Verbose | OsBitVersion: x64

    09:17:17 Verbose | Is64BitProcess: True

    09:17:17 Verbose | CurrentUser: NT AUTHORITY\SYSTEM

    09:17:17 Verbose | MachineName: <redacted>

    09:17:17 Verbose | ProcessorCount: 8

    09:17:17 Verbose | CurrentDirectory: C:\Octopus\Work\20170508141715-775

    09:17:17 Verbose | TempDirectory: C:\Windows\TEMP\

    09:17:17 Verbose | HostProcessName: Tentacle

    09:17:17 Verbose | 5 files extracted

    09:17:17 Info | Waiting for the upgrade to complete

    09:17:17 Verbose | No exit code log exists at 'C:\Octopus\Upgrade\20170508141702-DTY9T'

    09:17:17 Fatal | The remote script failed with exit code 65532

    09:17:17 Verbose | The remote script failed with exit code 65532

                        | Octopus.Shared.Tasks.ActivityFailedException: The remote script failed with exit code 65532

                        | at Octopus.Worker.Scripting.ScriptResult.EnsureSuccessful() in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Scripting\ScriptResult.cs:line 91

                        | at Octopus.Worker.Tentacles.TentacleUpgradeMediator.WaitForExitCode(String installId) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 117

                        | at Octopus.Worker.Tentacles.TentacleUpgradeMediator.PerformUpgrade(TargetManifest targetManifest) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 56

                        | at Octopus.Server.Orchestration.Health.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Orchestration\Health\HealthCheckService.cs:line 77

                        | Octopus.Server version 3.13.0 (3.13.0+Branch.master.Sha.872899cdf487a79ef334d6b3c310d671b65a57df)

    09:17:17 Verbose | Recording health check results

  12. Support Staff 12 Posted by Shane Gill on 09 May, 2017 02:43 AM

    Shane Gill's Avatar

    Thanks.

    3.13.2 contains the extra logging and I am hoping will complete successfully. Is everyone who is running into this issue upgrading a large number of machines?

    Cheers
    Shane

  13. 13 Posted by Mayank Kumar on 09 May, 2017 02:51 AM

    Mayank Kumar's Avatar

    Define "large number of machines" :)

    I upgraded > 100 machines. Does this qualify?

    Cheers,

    Mayank

    ________________________________

  14. Support Staff 14 Posted by Shane Gill on 29 May, 2017 01:41 AM

    Shane Gill's Avatar

    Hi everyone,

    It looks like this error is happening when running on PowerShell version 2. I have reproduced and created an issue to fix: https://github.com/OctopusDeploy/Issues/issues/3529

    Does that seem consistent with what you are seeing?

    Thanks for your patience.

    Cheers,
    Shane

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac