applicationHost.config destroyed

marchenko.alexandr's Avatar

marchenko.alexandr

17 Feb, 2017 10:07 AM

Hello, seems that Octopus successfully destroyed applicationHost.config :( thanks to gods that it is staging server and not production

There DEFINITELY should be revert action in case of unsuccessful changes to it!

  1. 1 Posted by marchenko.alexa... on 17 Feb, 2017 11:02 AM

    marchenko.alexandr's Avatar

    If someone gets into same trouble here is way to restore file:

    https://www.codeproject.com/Tips/630880/Recover-IIS-When-Applicatio...

  2. Support Staff 2 Posted by Dalmiro Grañas on 17 Feb, 2017 12:35 PM

    Dalmiro Grañas's Avatar

    Hi,

    Thanks for getting in touch. I'm very sorry your file got into such state. We haven't heard other reports of this behavior, so we'll be very interested in getting all the information possible to try to reproduce it and come up with a fix.

    Could you start by sending us the log of the deployment that broke the config file?

    Thanks,
    Dalmiro

  3. 3 Posted by marchenko.alexa... on 17 Feb, 2017 02:10 PM

    marchenko.alexandr's Avatar

    Hi Dalmiro

    Yes we can try to investigate what happen, at least now when everything operates normal I will be glad to give you any additional info

    Just need to clean up some data from logs which may be sensitive (or is there a way to send you logs directly?)

    From audit logs I do see following:

    there was failed deploy in 11:20

    in error logs for it I do see

    Running: Upload package migrator version 1.0.141-master
        ...
        System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
            ...
            System.IO.FileLoadException: Could not load file or assembly 'NuGet.Core, Version=2.11.1.812, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Provider DLL failed to initialize correctly. (Exception from HRESULT: 0x8009001D)
    

    then in ten minutes there is successful deploy of same project / version

    then in ten more minutes failed deploy of new version where everything broke

    in logs it says that it were broken right after trying to set basic authentication enabled to false which may not be related to problem

    11:41:28   Info     |       Comparing existing IIS bindings with configured bindings...
    11:41:28   Info     |       Looks OK
    11:41:28   Info     |       Bindings are as configured. No changes required.
    11:41:28   Info     |       Anonymous authentication enabled: True
    11:41:28   Info     |       Applied configuration changes to section "system.webServer/security/authentication/anonymousAuthentication" for "MACHINE/WEBROOT/APPHOST/Rabota.ua2 Mobile" at configuration commit path "MACHINE/WEBROOT/APPHOST"
    11:41:28   Info     |       Basic authentication enabled: False
    11:41:28   Info     |       ERROR ( message:Configuration error
    11:41:28   Info     |       Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
    11:41:28   Info     |       Line Number: 1563
    11:41:28   Info     |       Description: Configuration file is not well-formed XML
    11:41:28   Info     |       . )
    

    Here is when detective story begins:

    Looking at applicationHost.config seems that it was overriden by awssdk.s3.xml file which is going with https://www.nuget.org/packages/AWSSDK.S3/

    That xml is shipped with dlls, I have checked its nuget install.ps1 and there is nothig special with it, also nuget packages are shipped inside packages already preinstalled so should no do anything to them

    Also looking at your code I definitely may say that it also wont do anything like that

    So by logs it seems that octopus were able to verify all bindings which makes me thinking that in that point of time applicationHost.config were ok

    Then there was anonymous auth https://gist.github.com/PaulStovell/9522828#file-octopus-features-i... which in my opinion should not brake things, but after that next step with basic auth were not able to read xml

    I bed that you do not hear stories like that every day, but I also see this first time, we are using Octopus for a year now and it is first time when things go wrong

    Just wondering will we be able to identify at least what happen just because from now on it becomes to scary to deploy anything to productions just because in one click whole server may be destroyed

    PS: as about iis config backup seems that system makes them automatically which was our day saver

  4. Support Staff 4 Posted by Dalmiro Grañas on 20 Feb, 2017 05:29 PM

    Dalmiro Grañas's Avatar

    Hi,

    Thanks a lot for sending all that detailed info. Is there any chance you can send us your full log to [email blocked]? I'll have someone in our dev team check it out.

    Thanks,
    Dalmiro

  5. 5 Posted by marchenko.alexa... on 21 Feb, 2017 09:05 AM

    marchenko.alexandr's Avatar

    Thank you Dalmiro

    I sent logs, packages, description to email with subject "Detailed logs for applicationHost.config destroyed [Problems #51448]"

    Hope it will help in investigation

    In meanwhile will not touch anything in case we will need to try reproduce this (thank to gods it was stagin server so if there will be need theoretically I can try reprpoduce it and collect some more data)

    BR
    Alex

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac