Run Step Once From a Pool of Tentacles

chris.camburn's Avatar

chris.camburn

01 Feb, 2017 01:59 AM

This is a topic I've seen come up a number of times before, and specifically is referenced by Paul as something in the works in 2014 from this uservoice:

https://octopusdeploy.uservoice.com/forums/170787-general/suggestio...

In it, Paul mentions the ability to run a step on a "worker" that would use a pool of available workers. We need a scenario such as this to cover database deployments. Our database deployments need to occur only once per environment, but we have specific requirements for the servers that handle the database deployment. As such, a single server per environment cannot support the large number of projects we have. However, there is currently no convenient way to have multiple tentacles with a "db-deploy" role in an environment and have the step run on one available server.

I picture this would work like how TeamCity / Jenkins has a pool of available agents that can execute the job, and at runtime it selects one at random.

Is this something anyone has a convenient solution for? Is it something that was ever decided upon by the Octopus team? Is it something I should bring back to uservoice?

  1. Support Staff 1 Posted by Paul Stovell on 02 Feb, 2017 01:24 AM

    Paul Stovell's Avatar

    Hi Chris,

    Originally I thought this would be easy to support - we'd just pick a machine at random from a given set of machines in a role. But for it to really work like a build server, it would need to be quite a bit smarter - Octopus would have to be aware of what activities each machine is running, and queue them intelligently or use a pull-based model (it would be annoying if Octopus picked a machine that was already busy doing something else).

    Supporting that is a pretty significant architecture change, so not something we've tried to do. We are thinking about it this year though.

    The workaround is usually to just use a build server - some people use TeamCity to trigger deployments with Octopus, then run some tasks in TeamCity, then do more things in Octopus.

    Hope this helps,

    Paul

  2. 2 Posted by chris.camburn on 02 Feb, 2017 01:31 AM

    chris.camburn's Avatar

    Fair enough, thanks for the help, Paul.

    I'd imagine this would be a very useful feature for anyone doing database deployments, though I'd also assume most users don't have the scale we do.

    In the meantime, I'm going to work on a custom script queue system with variable expression triggers, and if I get something working I'll post it here for others.

  3. 3 Posted by bruno.rubin on 19 Feb, 2017 09:46 PM

    bruno.rubin's Avatar

    Hi Chris,

    please let us know when you have updates on that. We have exactly the same use case for database deployments.
    Currently our approach is to have one tentacle per environment, but as our environment and projects grow it's starting to get hard to maintain. We are working on some scripting to make that easier and automated.

    I'll let you know when we have any progress.

    Cheers,
    Bruno.

  4. 4 Posted by chris.camburn on 19 Feb, 2017 09:54 PM

    chris.camburn's Avatar

    Hey Bruno,

    I don't yet have the code, but I did work out a plan using some existing code.

    The plan is to have X number of tentacles with the role of 'db-deploy' that will exist in all environments. (Call these Server1, Server2, etc.) When a deployment starts for any project, it will first get a mutex on the Octopus server and create a file-based lock (using a simple .txt file) for Server1. When a second deployment starts, it checks to see if there is a txt file locking Server1. When it sees there is, it then creates a lock for Server2. This continues, with each deployment using a file-based locking system to determine which server to deploy to. Once it finds one, it outputs a variable for which server it will use.

    The database deployment step will be set to run on 'db-deploy' and the first part of that deployment will be to check the output variable for which server it should run on. All other servers immediately return, while the actual db deploy runs on the designated server.

    The final step in the project, which runs on success or failure, will be to unlock the file.

    Note that this process would require the OctopusBypassDeploymentMutex variable for all projects, which could cause other issues.

    I already have the file locking code done, which we use to limit simultaneous deployments of certain types of projects. I can share that code perhaps later in the week. The code for determining the db server shouldn't be too hard to add on, but I haven't had time to do that yet.

  5. 5 Posted by Silas Hansen on 06 Apr, 2017 06:38 AM

    Silas Hansen's Avatar

    Any news on this feature?

    We currently assign a role of "DbDeployer" to one of the nodes and restrict the DbDeploy-step to only run on that role. But it has the unpleasant side-effect of disabling db deploys if that node is taken out of the environment for maintenance or other reasons, which might not be obvious to anyone in the team but me.

  6. Support Staff 6 Posted by Rob Pearson on 11 Apr, 2017 05:34 AM

    Rob Pearson's Avatar

    Hi,

    I've re-opened the UserVoice suggestion as a lot has changed since it was originally set as completed. I'd like to invite everyone to add their comments and we'll review and discuss it.

    Thanks

    Rob

  7. 7 Posted by crmckenzie on 19 May, 2017 11:17 PM

    crmckenzie's Avatar

    The use case for us is that we have octopus packages that contain msi's to be deployed to several workstations. We also need to copy the msi's to a network share. We routinely get failed deployments due to multiple tentacles attempting to copy the msi to the network share even though we check for the existence of the file first. Ideally, we'd be able to specify that a step should only run on one of the available tentacles. I can see how this would be useful for database migrators or other "do once and forget" jobs.

  8. 8 Posted by Joe on 25 Jul, 2017 03:20 AM

    Joe's Avatar

    For me we use cordova to produce simple wrappers and we compile the website in teamcity but to use octopus and a deploy.sh script to package the ios app on a mac... we have multiple macs so it would be great to use them one at a time. Like team city (which I know your not!)

  9. Paul Stovell closed this discussion on 09 Nov, 2017 05:13 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac