2018.2.1 killing IIS boxes

Erick Stubbs's Avatar

Erick Stubbs

12 Feb, 2018 01:00 AM

After upgrading to 2018.2.1, 100% of our iis related deployments were having issues.

The issue manifested as a typical asp.net yellow screen of death. The exception was related to assembly loading, specifically it was unable to find the proper version of assemblies. It seemed as if it was ignoring the assemblyBinding redirects in the web.config file. After turning on fusion logging, the issue was that it was unable to find the web.config file. (Which would explain why the assembly binding section was not having any effect).

After doing a lot of investigating, it was clear there was nothing wrong with the deployment files. We copied them over to clean boxes, and we manually copied good builds and installed them on the tainted octopus boxes.

We cleared out IIS and manually configured and installed simple test sites, and nothing was able to load properly.

Eventually we downgraded to a previous version (2018.1), which was...not simple. Things are working now. But I believe there is an issue with the latest version of octopus or calamari.

I can provide more details about the specifics of our environment. But basically we deploy apps to iis boxes (vanilla 2016 iis install), under "Default Web Site" by creating a vdir (community step template), and then an iis application (builtin step). We let octopus substitute variables in appsettings and connection strings, and depending on the app, we might run additional transforms.

Once we deployed anything like that, the entire boxed was unusable from an IIS and .NET perspective no matter what we did to it.

  1. 1 Posted by Erick Stubbs on 12 Feb, 2018 05:26 PM

    Erick Stubbs's Avatar

    After doing some more testing and troubleshooting: We installed a brand new clean octopus and tentacle, and a brand new app to deploy.

    It is very easy to reproduce this problem. Use a .nupkg and a release with a version such as

    "Package.ID.1.0.0" <---- works fine
    "Package.ID.1.0.0+45" <---- fails.

    I believe it is related to this issue, but maybe not.

    http://help.octopusdeploy.com/discussions/problems/69682-deploy-packaged-urlencoded

  2. Support Staff 2 Posted by Rob Erez on 12 Feb, 2018 10:46 PM

    Rob Erez's Avatar

    Erick,
    As noted on the previous ticket this change was introduced to support packages that had invalid characters. For various reasons the blanket url encode felt like the safest way to solve the problem as we did not expect users to rely on specific naming of deployment folders. I will take a look at what we can do for valid semver characters which are over zealously getting encoded. In the meantime is it possible for you to rely on the Octopus.Action.Package.InstallationDirectoryPath variable to locate the deployed locations rather than assuming based on package Id? Cheers,
    Rob

  3. 3 Posted by Erick Stubbs on 12 Feb, 2018 11:04 PM

    Erick Stubbs's Avatar

    Hi Rob.

    I am not relying on any octopus variables for anything like that. I am just creating a package, a release, and using a single step to install.

    I've attached 2 screen shots

  4. Support Staff 4 Posted by Rob Erez on 15 Feb, 2018 09:41 PM

    Rob Erez's Avatar

    s an update, a fix will be going out in the next release (2018.3.4) which will allow the + characters in the file paths. This may be an interim fix until we can implement a more robust solution to allow for future cases when we may have other feeds which have illegal characters and need to work against IIS.
    Keep an eye out for this release in the next few days and let me know if you do (or don't) have any further problems.
    Cheers,
    Rob

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac