Octopus, Azure (Service Fabric) and post-deployment Smoke Tests

Aleksandras's Avatar

Aleksandras

29 Nov, 2017 03:36 PM

What is suggested approach to run tests after deploying to validate if everything works?

We are using TeamCity and Octopus in our CI/CD pipeline. TeamCity is good to run unit/integration tests during continuous build, but Octopus seems more suitable to run smoke tests during deployment - and report status if release can be promoted to higher environment.

We have Smoke Test package, with config file inside, which should be transformed using Octopus variables before executing tests. Tests are NUnit based, and there is NUnit installed on Octopus server.

We are deploying web application to Service Fabric, but I guess it would be something similar deploying to Azure in general.
Main point here is that in Azure deployments all steps are not being run on "deployment target" - they run on Octopus Server, which means we don't have any dedicated deployment server with Octopus tentacles.

Our current approach is that for now we have additional server, which is dedicated to run smoke tests. We deploy Smoke Test package to this server and config transformations are being done during this step. Next step - is "Run script" step, which runs NUnit on the server.
Everything works fine in this case, however we will be forced to get rid of that additional server - since there is really little point to have and maintain a separate server just for smoke tests.

So, the challenge here is that it's seems it's not possible to get Smoke Test package and transform config files without deploying package to a server.
Or are there any ways to do it? Or maybe there are some other suggested and working approaches?

We also considered an option to trigger tests in TeamCity form Octopus deployment, but it has it's own issues - like maintaining build configurations for different environments. However the main issue is that we don't want to overload our TeamCity agents executing smoke tests, because they are quite time consuming and there are a lot of other projects are using TeamCity (less than ones which are using Octopus).

  1. Support Staff 1 Posted by Reece Walsh on 30 Nov, 2017 04:45 AM

    Reece Walsh's Avatar

    Hi Aleksandras,

    Thank you for getting in touch!

    I can definitely understand the position you are in, it does appear to be unnecessary to have an additional server maintained solely for smoke test purposes.

    You should be able to install a Tentacle Agent instance on the Server itself, in this way you are able to utilize the same steps you have already provisioned.

    I thought I'd mentioned this in the event that you were not aware that you could have Octopus Server and the Tentacle Agent running on the same machine side-by-side.

    If you have already considered this option please let me know the challenges you are facing that prevents you from utilizing this option, it's possible there may be a workaround or solution.

    Thank you for your patience and understanding in this matter, it is greatly appreciated.

    If you require any further assistance or clarification please let me know :)

    Kind Regards,

    Reece

  2. 2 Posted by Aleksandras on 30 Nov, 2017 07:40 AM

    Aleksandras's Avatar

    Thanks for the response, Reece!

    It might be partly a solution what you mention with Tentacle agent on the Octopus server itself. Maybe it's easiest solution, but I'm not sure if our corporate Octoupus maintainers will be happy with such approach.;)

    For me it looks strange that some server is still needed for such task, which seems quite trivial and just some minor parts are missing.

    How I see it:
    1. Get and extract package - can be already done with "Run script" step, package is downloaded/extract to some temp location, isn't so?
    2. Transform config files - maybe it's also possible somehow with not-well documented PowerShell cmdlets? I mean without inventing custom transformation logic for the same purpose.
    3. Execute NUnit for some test assembly from extracted package.

    What do you think - is it really so easy as I see it, or there are some other challenges beside "missing" transformations feature?
    I acknowledge that this workflow probably would have same load on Octopus server as if it would be done with Tentacle Agent on it, but it won't require any "deployment" steps, which in general are extra (we don't need to deploy the package, we need just to extract files from it).

  3. 3 Posted by Aleksandras on 30 Nov, 2017 07:54 AM

    Aleksandras's Avatar

    Some more news: I found out that we already have a Tentacle on Octopus server and the scenario you suggested is already possible to do, but..

    Octopus server itself is often quite busy and locked with the following message:
    Cannot start this task yet because (has a read lock) tasks are currently running and this task cannot be run in conjunction with any other tasks

    In the middle of the day deployment step to Tentacle on Octopus server took around ~15 minutes, and then next step to execute tests (which comes immediately after deployment step) - took another ~15minutes. So it's additionally 30 minutes for the deployment. I would say, unfortunately it is not very acceptable approach..

  4. Support Staff 4 Posted by Reece Walsh on 04 Dec, 2017 06:40 AM

    Reece Walsh's Avatar

    Hi Aleksandras,

    Thanks for getting back to me regarding this,

    I appreciate your patience thus far, just to let you know where I'm at with this, I've spoken with the Team and will get back to you as soon as we have a solution available.

    I'll keep you in the loop moving forward as well :).

    Kind Regards,

    Reece

  5. Support Staff 5 Posted by Reece Walsh on 05 Dec, 2017 05:43 AM

    Reece Walsh's Avatar

    Hi Aleksandras,

    Thanks for your patience in regards to this,

    I've caught up with the Team again today regarding this scenario, we've had a few other customers reporting issues with deployments taking a long amount of time with the following message;

    (Cannot start this task yet because (has a read/write lock) tasks are currently running and this task cannot be run in conjunction with any other tasks)

    We currently have deployment logs that we are reviewing to address this, I'll keep you posted moving forward as previously mentioned, it's possible that re-working this would make the Tentacle installation on your server more feasible from a smoke test perspective.

    Kind Regards,

    Reece

  6. 6 Posted by Aleksandras on 05 Dec, 2017 06:37 AM

    Aleksandras's Avatar

    Thank you for the response, looking forward for the solution! Hope it will help. ;)

    Regarding message - who is exactly locking tasks on Tentacles? Is this lock caused by server, by role, or because something else? Maybe having separate role just for smoke-tests would solve the lock issue?

    Also I'm still curious why it's not possible without deployment to any target (just get package, unpack it and execute some script with that). Any clarifications about this point? ;)
    Maybe it's just outside the concept how Octopus is working? Although in this case deployments to the Cloud are also a special case/concept, I would say.

  7. Support Staff 7 Posted by Reece Walsh on 06 Dec, 2017 05:47 AM

    Reece Walsh's Avatar

    Hi Aleksandras,

    Thanks for getting back to me regarding this,

    I'm not sure at this stage what the specific cause of the locks are as our team is currently investigating this, I'll let you know more information as it becomes available to me. It's possible that if it is role related having a separate role would assist.

    Looking into this a bit further, it might be possible to utilize Offline Package Drops, with offline package drops instead of the application being deployed, Octopus will bundle up all of the files needed to perform the deployment on the actual target server.

    You could opt to run the files locally to perform the test instead of waiting the 30 mins that you are currently experiencing, to answer your previous query you're correct, to deploy without specifying a deployment target works outside the concept of Octopus when performing a standard deployment.

    I hope this helps, as I mentioned I'll touch base with you when I hear more about the tasks error side of things but let me know how you travel with this offline package drop option to see if this is a feasible workaround.

    Thanks again for all your patience to date, if you need further information or clarification please let me know, if I've also misunderstood in any way please correct me :).

    Kind Regards,

    Reece

  8. 8 Posted by Aleksandras on 06 Dec, 2017 11:05 AM

    Aleksandras's Avatar

    Hi, thank you for your suggestions.

    I looked a bit at Offline Package Drops - might be it could work, but seems like a workaround.
    Also looks like it still needs a target server and also enough permissions to be able to configure it. I don't have these permissions on our Octopus instance, because it's used for quite many teams.
    Of course I could ask to do configure it, but it's still doesn't look straightforward, so would be hard "to sell" it.;)

    If you say there is some more or less obvious issues with locked tasks, which fix could resolve our issues - then probably it's better just to wait for that fix.;)

  9. Support Staff 9 Posted by Reece Walsh on 07 Dec, 2017 05:52 AM

    Reece Walsh's Avatar

    Hi Aleksandras,

    Thank you for getting back to me regarding this, I appreciate you sharing your thoughts on the offline package drop workaround.

    I'm in communication with the team and will update you when I have an update to provide.

    I appreciate your patience to date, if you have any further queries relating to other issues please don't hesitate to reach out also :)

    Kind Regards,

    Reece

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac