applicationHost.config destroyed

marchenko.alexandr's Avatar

marchenko.alexandr

17 Feb, 2017 10:07 AM

Hello, seems that Octopus successfully destroyed applicationHost.config :( thanks to gods that it is staging server and not production

There DEFINITELY should be revert action in case of unsuccessful changes to it!

  1. 1 Posted by marchenko.alexa... on 17 Feb, 2017 11:02 AM

    marchenko.alexandr's Avatar

    If someone gets into same trouble here is way to restore file:

    https://www.codeproject.com/Tips/630880/Recover-IIS-When-Applicatio...

  2. Support Staff 2 Posted by Dalmiro Grañas on 17 Feb, 2017 12:35 PM

    Dalmiro Grañas's Avatar

    Hi,

    Thanks for getting in touch. I'm very sorry your file got into such state. We haven't heard other reports of this behavior, so we'll be very interested in getting all the information possible to try to reproduce it and come up with a fix.

    Could you start by sending us the log of the deployment that broke the config file?

    Thanks,
    Dalmiro

  3. 3 Posted by marchenko.alexa... on 17 Feb, 2017 02:10 PM

    marchenko.alexandr's Avatar

    Hi Dalmiro

    Yes we can try to investigate what happen, at least now when everything operates normal I will be glad to give you any additional info

    Just need to clean up some data from logs which may be sensitive (or is there a way to send you logs directly?)

    From audit logs I do see following:

    there was failed deploy in 11:20

    in error logs for it I do see

    Running: Upload package migrator version 1.0.141-master
        ...
        System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
            ...
            System.IO.FileLoadException: Could not load file or assembly 'NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Provider DLL failed to initialize correctly. (Exception from HRESULT: 0x8009001D)
    

    then in ten minutes there is successful deploy of same project / version

    then in ten more minutes failed deploy of new version where everything broke

    in logs it says that it were broken right after trying to set basic authentication enabled to false which may not be related to problem

    11:41:28   Info     |       Comparing existing IIS bindings with configured bindings...
    11:41:28   Info     |       Looks OK
    11:41:28   Info     |       Bindings are as configured. No changes required.
    11:41:28   Info     |       Anonymous authentication enabled: True
    11:41:28   Info     |       Applied configuration changes to section "system.webServer/security/authentication/anonymousAuthentication" for "MACHINE/WEBROOT/APPHOST/Rabota.ua2 Mobile" at configuration commit path "MACHINE/WEBROOT/APPHOST"
    11:41:28   Info     |       Basic authentication enabled: False
    11:41:28   Info     |       ERROR ( message:Configuration error
    11:41:28   Info     |       Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
    11:41:28   Info     |       Line Number: 1563
    11:41:28   Info     |       Description: Configuration file is not well-formed XML
    11:41:28   Info     |       . )
    

    Here is when detective story begins:

    Looking at applicationHost.config seems that it was overriden by awssdk.s3.xml file which is going with https://www.nuget.org/packages/AWSSDK.S3/

    That xml is shipped with dlls, I have checked its nuget install.ps1 and there is nothig special with it, also nuget packages are shipped inside packages already preinstalled so should no do anything to them

    Also looking at your code I definitely may say that it also wont do anything like that

    So by logs it seems that octopus were able to verify all bindings which makes me thinking that in that point of time applicationHost.config were ok

    Then there was anonymous auth https://gist.github.com/PaulStovell/9522828#file-octopus-features-i... which in my opinion should not brake things, but after that next step with basic auth were not able to read xml

    I bed that you do not hear stories like that every day, but I also see this first time, we are using Octopus for a year now and it is first time when things go wrong

    Just wondering will we be able to identify at least what happen just because from now on it becomes to scary to deploy anything to productions just because in one click whole server may be destroyed

    PS: as about iis config backup seems that system makes them automatically which was our day saver

  4. Support Staff 4 Posted by Dalmiro Grañas on 20 Feb, 2017 05:29 PM

    Dalmiro Grañas's Avatar

    Hi,

    Thanks a lot for sending all that detailed info. Is there any chance you can send us your full log to [email blocked]? I'll have someone in our dev team check it out.

    Thanks,
    Dalmiro

  5. 5 Posted by marchenko.alexa... on 21 Feb, 2017 09:05 AM

    marchenko.alexandr's Avatar

    Thank you Dalmiro

    I sent logs, packages, description to email with subject "Detailed logs for applicationHost.config destroyed [Problems #51448]"

    Hope it will help in investigation

    In meanwhile will not touch anything in case we will need to try reproduce this (thank to gods it was stagin server so if there will be need theoretically I can try reprpoduce it and collect some more data)

    BR
    Alex

  6. Support Staff 6 Posted by Nick Josevski on 27 Feb, 2017 04:02 AM

    Nick Josevski's Avatar

    Hello Alex,
    Thanks for giving us all those details, we had a discussion and some thought about this.

    A popular feature: 'parallel deploys' makes trying to roll back changes to the 'applicationHost.config' not possible, as it would break the behavior for those customers, so we won't be trying to introduce a way to recover such files automatically.

    What you can do in your case to give you more confidence is introduce 2 extra scripted steps; first one being a pre-deploy step to back up that file (and any others). Followed by a deploy-failure step (you run only if needed) to revert those files if it does fail. You could also give the Guided Failures feature a try to give you extra safety: https://octopus.com/docs/deploying-applications/guided-failures

    As for the issue of the file itself becoming corrupted; we had a look and with the details you supplied cannot work out went wrong. Please let us know if you experience it again and hopefully additional information might help pinpoint the problem.

    Best Regards,
    Nick

  7. 7 Posted by marchenko.alexa... on 27 Feb, 2017 09:02 AM

    marchenko.alexandr's Avatar

    Hi Nick

    Thanks for getting in touch, I do understand your point about parallel
    deployments and it seems reasonable, also thank you for pointing out for
    possible solution

    BTW It seems that there is no need to manually backup file, system backs it
    up every time changes occur so need just one step to recover it in case of
    fail

    I hope to never ever again catch such issue

    Thank you in advance
    Alex

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac