How to wait for deployment target to become Healthy?

konstantine's Avatar

konstantine

14 Jul, 2017 12:57 AM

First part of my Octopus deployment project provision virtual infrastructure in Azure, including VMs. Then I use Octopus Azure VM Extension to install Tentacle on those VMs in order to use them in the following deployment steps. Unfortunately when Azure Resource Group step finishes new VMs are not yet ready to be used - they still need to download and install Calamari and it takes some time. I need to pause my deployment process until all provisioned VMs become available and healthy. If I continue too early then deployment fails because new machines are not healthy yet.

What would be the right way to wait for all deployment targets in the Environment to become healthy before continuing deployment?

  1. Support Staff 1 Posted by Jason Brown on 17 Jul, 2017 01:02 AM

    Jason Brown's Avatar

    Hi Konstantine,

    Thanks for getting in touch.

    There are a number of ways to achieve infrastructure + code deployments in Octopus, but in general I'd say it's best not to leave a Deployment task waiting on infrastructure provisioning. For one thing, the various clouds can fail provisioning for various reasons, with the potential to leave open tasks hanging around in Octopus. For another, with cloud deployments you may be bringing up one or many instances, and a running deployment project has no way to know what those instances are when you hit 'deploy'.

    It is definitely achievable, though.

    I've achieved this two ways in the past, so I'll try and break those down here.

    1. Completely decoupled Code + Infrastructure

    This approach would use two separate Octopus Projects, one for the infrastructure steps, and another for deploying code to your infrastructure.

    When you want to deploy new infrastructure, you run the infrastructure project, then when the provisioning completes, an Octopus Trigger, or some custom code on the instance(s) would trigger a deployment of your code to the instance(s)

    2. Blended Code & Infrastructure

    With this approach you use a single Octopus Project, which targets to multiple roles, for example:

    Steps concerned with spinning up the instance(s) run on the Octopus server itself or on a role called, say infrastructure. Only the steps concerned with infrastructure run, because there are no instances present yet, and you get a green tick for a successful deployment. This release then becomes the most recent successful deployment.

    When the new instance(s) come online they register with the role mycodeproject, and a trigger (or some custom script) re-runs the same Octopus Project - deploying the last-good release. This is the release you just got a green tick for in the first step. The Project then runs, deploying the code to the newly-provisioned instance(s), and you're up and running.

    Both these approaches need a trigger on the target instances, for which I'd recommend the built-in Deployment Triggers feature available from 3.4, though it can be achieved with custom code using the Octopus API

    The second approach assumes your infrastructure provisioning code is idempotent - so can be run multiple times without duplicating its effects. In Azure that would ideally be an ARM template, in AWS you'd use Cloudformation, but there are ways to achieve this with the various APIs. As long as you can prevent infrastructure being accidentally provisioned twice, you're all set.

    Does this help with your enquiry?

    Regards,
    Jason

  2. 2 Posted by konstantine on 17 Jul, 2017 03:22 AM

    konstantine's Avatar

    Jason, thank you for elaborate answer! First approach with separate projects for infrastructure and code looks more transparent and manageable to me and I already started moving in that direction.

  3. Support Staff 3 Posted by Jason Brown on 17 Jul, 2017 05:40 AM

    Jason Brown's Avatar

    Thanks Konstantine,

    Glad to be of help

    Jason

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac