Tentacle windows service disabled when deployed using Azure VM extension

kevin_hubbard's Avatar

kevin_hubbard

02 Jun, 2017 03:42 PM

Hello,
Seeing some odd behavior on my tentacles that were deployed using the Azure vm extension. These are new vm's deployed with the re-leased Octopus Deploy Tentacle extension for Azure ARM virtual machines. After working for a few days, I"m noticing these vm's showing as disconnected in the environment dashboard. When I connect to the vm and launch the tentacle manager, it is apparent that the tentacle service is stopped. When I attempt to restart via the tentacle manager I receive an error (did not capture). When viewing the service properties, the startup type is set to "Disabled". If I change the startup type to "Automatic" and start the tentacle service, everything works fine and my deployments succeed.

I'm not sure what is disabling the tentacle service. Any thoughts?

Thanks,
Kevin

  1. Support Staff 1 Posted by Matt Richardson on 05 Jun, 2017 04:03 AM

    Matt Richardson's Avatar

    Hi Kevin

    Thanks for getting in touch! Sorry to hear that you've hit this issue.

    I pushed out a new release just over a week ago to stop the Tentacle re-registering when the machine restarted, but in the process, I broke the extension upgrade process (unfortunately, Microsoft makes this part very hard to test).

    I've since pushed out a new version (on Friday) which handles this much better, and hopefully will replace the broken version.

    For now, you can just re-enable the service, and you should be good to go. If you'd prefer, you can uninstall and re-install the extension, and that should also get you into a better state.

    Once again, sorry for the pain this has caused.

    Let me know how it goes.

    Regards,
    Matt

  2. 2 Posted by kevin_hubbard on 14 Jun, 2017 08:39 PM

    kevin_hubbard's Avatar

    Thanks for the response and quick fix. This appears to be resolved. Have not seen the issue for a week or so and a couple of restarts of the vm's.

    Glad to have the extension back! Particularly useful w/ deploying IaaS resources with Terraform.

  3. Support Staff 3 Posted by Matt Richardson on 14 Jun, 2017 10:15 PM

    Matt Richardson's Avatar

    Hi Kevin

    Thanks for letting me know its resolved and that you're finding it useful.

    Happy deployments!

    Cheers,
    Matt

  4. Paul Stovell closed this discussion on 22 Sep, 2017 09:23 AM.

  5. kevin_hubbard re-opened this discussion on 14 Dec, 2017 03:50 PM

  6. 4 Posted by kevin_hubbard on 14 Dec, 2017 03:50 PM

    kevin_hubbard's Avatar

    Hello Running into some issues using Terraform to deploy the ARM Tentacle Extension Here are my settings (sensitive values removed):

    resource "azurerm_virtual_machine_extension" "app1tentacle" {
      name = "OctopusDeployWindowsTentacle"
      location = "${var.location}"
      resource_group_name = "${azurerm_resource_group.default.name}"
      virtual_machine_name = "${azurerm_virtual_machine.app1.name}"
      publisher = "OctopusDeploy.Tentacle"
      type = "OctopusDeployWindowsTentacle"
      type_handler_version = "2.0"

      settings = <<SETTINGS
        {
            "OctopusServerUrl": "https:/****.com",
            "CommunicationMode": "Listen",
            "PublicHostNameConfiguration": "PublicIP",
            "Port": 60012,
            "Environments": "${var.od_envstage}",
            "Roles": "app-dbup,app-landfill,app-server",
            "ApiKey": "****"
        }
    SETTINGS
    }

    Here is the error Terraform is returning:
    azurerm_virtual_machine_extension.app1tentacle: compute.VirtualMachineExtensionsClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="VMExtensionProvisioningError" Message="VM has reported a failure when processing extension 'OctopusDeployWindowsTentacle'. Error message: \"System.ArgumentNullException: Value cannot be null.\r\nParameter name: value\r\n at Newtonsoft.Json.Linq.Extensions.Value[T,U](IEnumerable`1 value)\r\n at AzureVMExtension.Configuration.ApplicationConfig.ReadFile(String filename) in Z:\\buildAgent\\workDir\\8d5946312b69ad91\\src\\AzureVMExtension\\Configuration\\ApplicationConfig.cs:line 191\r\n at AzureVMExtension.Configuration.ApplicationConfig.ParseFile(String filename) in Z:\\buildAgent\\workDir\\8d5946312b69ad91\\src\\AzureVMExtension\\Configuration\\ApplicationConfig.cs:line 60\r\n at Polly.RetrySyntax.<>c__DisplayClass10_0.<WaitAndRetry>b__1() in C:\\projects\\polly\\src\\Polly.Shared\\RetrySyntax.cs:line 211\r\n at Polly.Retry.RetryEngine.Implementation[TResult](Func`1 action, IEnumerable`1 shouldRetryExceptionPredicates, IEnumerable`1 shouldRetryResultPredicates, Func`1 policyStateFactory) in C:\\projects\\polly\\src\\Polly.Shared\\Retry\\RetryEngine.cs:line 40\r\n at Polly.RetrySyntax.<>c__DisplayClass10_1.<WaitAndRetry>b__0(Action action, Context context) in C:\\projects\\polly\\src\\Polly.Shared\\RetrySyntax.cs:line 210\r\n at AzureVMExtension.Configuration.ApplicationConfig.Parse(String configFolder, Int32 sequenceNumber) in Z:\\buildAgent\\workDir\\8d5946312b69ad91\\src\\AzureVMExtension\\Configuration\\ApplicationConfig.cs:line 47\r\n at AzureVMExtension.Program.Execute(String[] args) in Z:\\buildAgent\\workDir\\8d5946312b69ad91\\src\\AzureVMExtension\\Program.cs:line 75\"."
    * azurerm_storage_container.sqldata: 1 error(s) occurred:

  7. 5 Posted by kevin_hubbard on 14 Dec, 2017 04:00 PM

    kevin_hubbard's Avatar

    Here is the only log file I can find c:\Octopus\Tentacle\Logs

  8. Support Staff 6 Posted by Michael Noonan on 20 Dec, 2017 04:09 AM

    Michael Noonan's Avatar

    Hi Kevin,

    Thanks for keeping in touch! It looks like its failing on this line which means the Thumbprint is missing. This is odd because that thumbprint token should have a value in it.

    Our best guess is you have configured the ApiKey as a public setting, rather than a private/protected setting, and as there are no private settings Azure is not setting the thumbprint to something sensible.

    Perhaps you need to use protected_settings in your Terraform provider? https://www.terraform.io/docs/providers/azurerm/r/virtual_machine_e...

    Please let me know if this rings true and if it helps you get up and running. :)

    Hope that helps!
    Mike

  9. 7 Posted by kevin_hubbard on 20 Dec, 2017 08:30 PM

    kevin_hubbard's Avatar

    Hi Michael,
    Thanks for this information. I’ve implemented the protected_settings block for the API key setting but still stuck in the same place. Is there a specific release of the 2.0 version that I should specify for the extension?

    Thanks,
    Kevin

  10. 8 Posted by kevin_hubbard on 20 Dec, 2017 08:37 PM

    kevin_hubbard's Avatar

    · This is the error message from the Azure portal extension status:
    System.InvalidCastException: Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JArray'. at AzureVMExtension.Configuration.ApplicationConfig.GetPublicArraySetting(JToken publicSettings, String property) in Z:\buildAgent\workDir\8d5946312b69ad91\src\AzureVMExtension\Configuration\ApplicationConfig.cs:line 158 at AzureVMExtension.Configuration.ApplicationConfig.ParseFile(String filename) in Z:\buildAgent\workDir\8d5946312b69ad91\src\AzureVMExtension\Configuration\ApplicationConfig.cs:line 74 at Polly.RetrySyntax.<>c__DisplayClass10_0.<WaitAndRetry>b__1() in C:\projects\polly\src\Polly.Shared\RetrySyntax.cs:line 211 at Polly.Retry.RetryEngine.Implementation[TResult](Func`1 action, IEnumerable`1 shouldRetryExceptionPredicates, IEnumerable`1 shouldRetryResultPredicates, Func`1 policyStateFactory) in C:\projects\polly\src\Polly.Shared\Retry\RetryEngine.cs:line 40 at Polly.RetrySyntax.<>c__DisplayClass10_1.<WaitAndRetry>b__0(Action action, Context context) in C:\projects\polly\src\Polly.Shared\RetrySyntax.cs:line 210 at AzureVMExtension.Configuration.ApplicationConfig.Parse(String configFolder, Int32 sequenceNumber) in Z:\buildAgent\workDir\8d5946312b69ad91\src\AzureVMExtension\Configuration\ApplicationConfig.cs:line 47 at AzureVMExtension.Program.Execute(String[] args) in Z:\buildAgent\workDir\8d5946312b69ad91\src\AzureVMExtension\Program.cs:line 75

  11. Support Staff 9 Posted by Michael Noonan on 22 Dec, 2017 12:54 AM

    Michael Noonan's Avatar

    Hi Kevin,

    It does sound like you're getting further along. From that error message it looks like the JSON configuration for one of the items is a value, but the code is expecting an array - even if that array has a single value in it. Here is the line of code that is failing: https://github.com/OctopusDeploy/OctopusAzure/blob/master/src/Azure...

    From going back in the stack trace I can see it was trying to load the list of Environments the Tentacle should be added to: https://github.com/OctopusDeploy/OctopusAzure/blob/master/src/Azure...

    That function is actually a pretty good place to look so you can understand where each setting should actually be, whether a protected or public setting, and whether it should be an array or a single value.

    Hope that helps!
    Mike

  12. 10 Posted by kevin_hubbard on 22 Dec, 2017 06:01 PM

    kevin_hubbard's Avatar

    Thanks Michael,
    I’m having some difficulty navigating to the links below – Github returns a 404.

    Any thoughts?

    -Kevin

  13. 11 Posted by kevin_hubbard on 22 Dec, 2017 06:26 PM

    kevin_hubbard's Avatar

    This was the issue - even if that array has a single value in it. Both environments and roles need to be an array eg [“some value”]

    Thank you!!!

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac