applicationHost.config corruption

Michael Knoll's Avatar

Michael Knoll

18 Apr, 2017 08:48 PM

We encountered an issue when deploying multiple website to a production server. The deployment succeeded in test and stage, but when promoting to production, we receive the error:

Attempt 4 of 5 failed: Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
Line number: 1
Error: Configuration file is not well-formed XML
Waiting for 2 seconds before retrying...
Retrying...
get-webitemstate : Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
Line number: 1
Error: Configuration file is not well-formed XML
At C:\xxxxx\Octopus.Features.IISWebSite_BeforePostDeploy.ps1:165 char:3
+ $state = Get-WebAppPoolState $applicationPoolName
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-WebItemState], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException,Microsoft.IIs.PowerShell.Provider.GetItemStateCommand

This error has occurred on multiple web sites on the server, in different parts of the deployment of the site. It has failed setting windows authentication, and updating bindings. Once the error has occurred, the applicationHost.config file has contained a powershell script, and what appears to be the binary of a DLL.
I have compared the applicationHost.config file from our stage and prod server, and found no significant difference. They both appear to be valid XML, and the only differences are expected because of the environment/machine difference.

Are there any known causes of this issue? The deployment steps are Octopus Deploy build-in steps, any recommendations for debugging them?

Just a note for anyone that has had this happen to them, the config file can be recovered by copying it from c:\inetpub\history.

Thanks,
Mike

  1. Support Staff 1 Posted by Nick Josevski on 19 Apr, 2017 12:39 AM

    Nick Josevski's Avatar

    Hello Mike,

    Thanks for getting in touch, this is a strange one. We've seen it at least once before but have not been able to reproduce it. Our suspicion is multiple threads causing the issue, in version 3.11.18 of Octopus we put in some safe guards to make IIS deployments more reliable. If you can you can try and update to a version after that to see if it solves the problem: https://octopus.com/downloads

    Could we get some more details from you to help us work out what's going on. What version of IIS and on which version of Windows is it running on, and which version of Octopus are you currently running.

    Regards,
    Nick

  2. 2 Posted by Michael Knoll on 19 Apr, 2017 05:54 PM

    Michael Knoll's Avatar

    We are running Octopus version 3.7.14. Our servers are Microsoft Windows Server 2012 R2 Standard, running IISversion 8.5.9600.16384.

    Thanks,
    Mike

  3. Support Staff 3 Posted by Nick Josevski on 25 Apr, 2017 11:35 PM

    Nick Josevski's Avatar

    Hello Mike,

    Thanks for that information, have you continued to experience this issue?

    We'd like to know if you're using the OctopusBypassDeploymentMutex flag as part of the deployments where you encounter the applicationHost corruption, there's some good documentation here about how it's used: https://octopus.com/docs/how-to/run-multiple-processes-on-a-tentacl...

    Regards,
    Nick

  4. 4 Posted by Michael Knoll on 26 Apr, 2017 01:27 PM

    Michael Knoll's Avatar

    We are using the OctopusBypassDeploymentMutex flag. However, the IIS steps that were failing were failing when no other steps were in parallel with it.

    We are in the process of scheduling the upgrade of Octopus Deploy.

    Thanks,
    Mike

  5. 5 Posted by chirschi on 27 Apr, 2017 06:21 PM

    chirschi's Avatar

    This is definitely happens in our environment sporadically. Looking at the applicationhost.config file after the failed deploy looks like it contains garbage either from disk or RAM, almost like a memory overflow.

    This has been happening to us for over a year now, so I'm not sure if it's something that will be fixed with a new release unless it was specifically addressed.

    This causes some major issues when release our websites and we would really like to find a solution to this.

  6. Support Staff 6 Posted by Nick Josevski on 28 Apr, 2017 01:18 AM

    Nick Josevski's Avatar

    Hello Chris,

    Thanks for getting in touch, sorry to hear you have also been experiencing this issue too. Would you be able to help us narrow down the cause, by providing some information about your set up and installed version.

    What is the current version of Octopus you are running that still shows the corruption issue? Does it always happen on the same machine that's being deployed to? What versions of Windows Server/IIS is/are the affected machines?

    Are you also making use of the OctopusBypassDeploymentMutex flag for your deployments?

    Regards,
    Nick

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac