Handling a tentacle reboot during a deployment

Micheal Schexnayder's Avatar

Micheal Schexnayder

11 Aug, 2017 09:45 PM

This isn't so much as a question but more of a 'How to' since I really couldn't find a solution here that was palatable for us. We hope this helps others in this situation.

My team and I have been working to solve the issue where a deployment process step requires a tentacle server to reboot, mid-deployment. Since we don't want the deployment to fail when we issue the restart-computer command, we came up with this solution.

Add a script step immediately after the step that requires the reboot. That step should have it's 'Run Condition' set to "Always Run" AND (critical) be set to run on the Octopus Deploy server.

This has been tested on Octopus Deploy 3.13.x. It would likely require some tweaking to work on earlier versions.

The body of the script step should contain this (this is what we use)

$timeout = 300
$duration = 0
$frequency = 15
$loop = $true
$tentacle = (Resolve-DnsName -Name $OctopusParameters['Octopus.Machine.Name']).Name
$listeningPort = 10933

$ErrorActionPreference = $WarningPreference = 'SilentlyContinue'

# this keeps the script from running too fast and checking the server before it has started the reboot cycle
Start-Sleep 30

while ($loop){
    if ($duration -lt $timeout){
        $serverState = Test-NetConnection -ComputerName $tentacle -Port $listeningPort -InformationLevel Quiet
        if ($serverState -eq $true){
            Write-Output "Tentacle: $tentacle is online."
            $loop = $false
        } else {
            Write-Output "Tentacle: $tentacle is offline. Will try again in $frequency seconds."
            start-sleep $frequency
            $duration += $frequency
        }
    } else {
        Write-Output "[ERROR] Tentacle failed to come back online after $duration seconds"
        $loop = $false
    }
}

This simply attempts to open a connection on the tentacles listening port. If the connection succeed's then the tentacle service is running and ready for some work.

Optionally you can add a health check step to make sure all your tentacles are completely active before proceeding but this should work for most situations.

We've since turned this into a step template for easy re-use when needed.

Good luck! :)

  1. Support Staff 1 Posted by Shane Gill on 13 Aug, 2017 11:48 PM

    Shane Gill's Avatar

    Hi Michael,

    Thanks for the script. Would you open to contributing your step template to our library so that others can more easily benefit from it?

    You can find more information on contributing here: https://github.com/OctopusDeploy/Library

    Thanks!

    Shane

  2. 2 Posted by Micheal Schexna... on 14 Aug, 2017 01:45 PM

    Micheal Schexnayder's Avatar

    Seems reasonable. I'm verifying through our legal department that it is ok before proceeding, however. It may take a few days. I'll post here once I've received the go-ahead.

  3. 3 Posted by Rushikesh Sane on 17 Nov, 2017 06:43 AM

    Rushikesh Sane's Avatar

    Hi Michael,
    Thanks for sharing the script. Where are you populating $OctopusParameters array in the script?

  4. 4 Posted by Micheal Schexna... on 17 Nov, 2017 01:56 PM

    Micheal Schexnayder's Avatar

    We aren't. This is a server side task so Octopus already knows which server(s) it's working with this line:
    $tentacle = (Resolve-DnsName -Name $OctopusParameters['Octopus.Machine.Name']).Name

    We toyed with the idea of populating $OctopusParameters with some values about determining if a reboot was needed or not but eventually decided against it.

    In the long run, we ditched this idea, altogether. While it works effectively pausing the deployment mid-process, there were other variables that we simply couldn't cleanly handle...like other projects deploying when the tentacle goes offline.

    Effectively we now have have a standing policy that any task that requires a tentacle server to reboot as a part of any setup process is handled external to Octopus. Powershell's DSC fits well into this space for initial configuration.

  5. 5 Posted by Rushikesh Sane on 20 Nov, 2017 05:20 AM

    Rushikesh Sane's Avatar

    Oh, Okay. Thanks Michael. This script was of great help to handle tentacle reboots.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac