SSL Certificate add failed, Error: 1312

jake.stevenson's Avatar

jake.stevenson

29 Aug, 2014 03:15 PM

In a recent deployment to our production environment Octopus did not apply SSL certificates to the sites. The deployment succeeded, but the application would not run because of SSL requirements. When looking in the logs, I saw messages similar to this one for each binding in IIS:

Finding SSL certificate with thumbprint MYTHUMB
Found certificate: CN=origin.********.ca, OU=PositiveSSL, OU=Domain Control Validated
A different binding exists for the IP/port combination, replacing...
SSL Certificate successfully deleted
SSL Certificate add failed, Error: 1312

I was able to log into each production server and manually associate the certificate with the sites.

A search through the forums brought up http://help.octopusdeploy.com/discussions/problems/16491-fail-to-bind-a-self-signed-ssl-certificate, but it's pretty old and we're at a much higher version (2.5.7.384) now.

What else can I do to hunt down this issue and prevent downtime with our next deployment?

  1. 1 Posted by Nicholas Blumha... on 01 Sep, 2014 05:31 AM

    Nicholas Blumhardt's Avatar

    Hi Jake,

    Thanks for getting in touch - sorry about the difficulties.

    There seem to be a few possible causes of this particular error; is it possible there's some small variation between the thumbprint you're using, and the one shown for the cert in MMC/Certificate Manager?

    The installation script prints:

    A different binding exists for the IP/port combination, replacing...
    

    ...when the port is bound to a different cert; to avoid any future problems one simple approach might be to check the thumbprint and ensure the binding details are exactly the same in Octopus. With that the case, I don't think Octopus will attempt to re-bind the certificate as it has done here.

    We did make changes to this area in 2.5 so it is also entirely possible the problem is with the script - you may turn up some more useful information by running:

    netsh http show sslcert
    

    ...on the target machine. Octopus is looking for an entry here with a matching IP address, port and thumbprint ("Certificate Hash") value. Do you see in the output any reason this might not match up?

    Best regards,
    Nick

  2. 2 Posted by jake.stevenson on 02 Sep, 2014 01:17 PM

    jake.stevenson's Avatar

    netsh http show sslcert returns the expected thumbprint. For IP:Port it's showing 0:0:0:0:443 which makes sense. In Octopus we've set it up to generate several bindings for the site to IP "*" with different hostnames.

    So assuming 0:0:0:0:443 is what Octopus is using when it says "*" for the ip address, it should match up, right?

  3. 3 Posted by Nicholas Blumha... on 03 Sep, 2014 03:04 AM

    Nicholas Blumhardt's Avatar

    Yes, that's correct.

    Is there any server you can attempt the deployment to again in order to verify that it is a persistent issue? If so, let me know and I'll raise a bug for this.

    Regards,
    Nick

  4. 4 Posted by jake.stevenson on 04 Sep, 2014 02:00 PM

    jake.stevenson's Avatar

    We performed another deployment this morning and the issue persisted. After deployment I had to go into IIS Management on each server and manually associate the certificate with the binding.

    One more bit of information -- each server has two certificates available. Of course only one matches the requested thumbprint.

    Please let me know if I can provide any further information.

  5. 5 Posted by Nicholas Blumha... on 08 Sep, 2014 04:21 AM

    Nicholas Blumhardt's Avatar

    Thanks for the follow-up Jake. I've raised https://github.com/OctopusDeploy/Issues/issues/1187 - which we'll update with information as we work through it.

    Best regards,
    Nick

  6. 6 Posted by morten.faester on 08 Sep, 2014 07:41 PM

    morten.faester's Avatar

    This is also an issue in my production environment with a cluster of web servers hosting multiple sites. For the moment Octopus cannot really be used for deployments as it renders the sites unavailable due to certificate warnings.

  7. 7 Posted by Nicholas Blumha... on 09 Sep, 2014 01:16 AM

    Nicholas Blumhardt's Avatar

    I'm digging into this further today. I should have asked originally:

    • Which OS are you running?
    • PowerShell version?
    • Are you using SNI, or anything similar?

    I've also prepared a version of our IIS configuration script that writes much more data to the log; is there any machine you can run this on for debugging purposes?

    Best regards,
    Nick

  8. 8 Posted by Nicholas Blumha... on 09 Sep, 2014 02:12 AM

    Nicholas Blumhardt's Avatar

    Some more notes, based on what I've been able to find online about this error.

    First up, is the certificate in the Local System -> Personal store under MMC? There are a few suggestions on this page: http://social.technet.microsoft.com/Forums/windowsserver/en-US/6845... - if the store is correct, are there any other possibilities there that might apply?

    On Windows Server 2008 R2 a hotfix may be required: http://support.microsoft.com/kb/981506 (guessing that one is a long shot).

    Finally - a bit more of a stretch - the second answer listed on this page suggests a missing configuration item ("Unique container name") may be the issue and a repair operation required: http://stackoverflow.com/questions/13076915/ssl-certificate-add-fai...

    Just thought I'd share this information since it might help identify something simple; running the instrumented configuration script and sending the details requested above will still help us track this down. Thanks!

  9. 9 Posted by morten.faester on 09 Sep, 2014 09:33 AM

    morten.faester's Avatar

    Hey Nicholas

    Thank you for getting back to me.

    First of all I have solved the problem for now by simply letting our Netscaler (load balancer) offload the https connections to one of the two sites involved. This way the IIS only have to handle one certificate, but the end users can access both sites through https.

    I have the certificates in localmachine\my and the bindings are correct.

    But perhaps I am also asking too much from Octopus, since the problem is also related to IIS configuration.

    If I try to setup the bindings manually I have to be able to differentiate the sites on IP address. Binding on hostname alone doesn’t appear to work.

    It seems the certificates are bound in the OS and directed by IIS. So if I execute “netsh http show sslcert” I can see that there is only one cert bound on the box, and that this certificate changes depending on the last deployment. This in time is picked up by IIS, so my https bindings are always tied to the certificate from the latest deployment.

    As mentioned I have a work around, so this is not urgent. But it would be nice having a solution though.

    KR
    Morten

    C:\Windows\system32>netsh http show sslcert

    SSL Certificate bindings:
    -------------------------

        IP:port : 0.0.0.0:443
        Certificate Hash : c36891e5d104d81c9c35800cca29cfae0bf7dbc8
        Application ID : {26587bc3-04ae-4f00-833f-e2ac21dc6d48}
        Certificate Store Name : (null)
        Verify Client Certificate Revocation : Enabled
       Verify Revocation Using Cached Client Certificate Only : Disabled
        Usage Check : Enabled
        Revocation Freshness Time : 0
        URL Retrieval Timeout : 0
        Ctl Identifier : (null)
        Ctl Store Name : (null)
        DS Mapper Usage : Disabled
        Negotiate Client Certificate : Disabled

        IP:port : 0.0.0.0:8172
        Certificate Hash : 14172f02bd81768e69a47b6d10380e82b0e46d22
        Application ID : {00000000-0000-0000-0000-000000000000}
        Certificate Store Name : MY
        Verify Client Certificate Revocation : Enabled
        Verify Revocation Using Cached Client Certificate Only : Disabled
        Usage Check : Enabled
        Revocation Freshness Time : 0
        URL Retrieval Timeout : 0
        Ctl Identifier : (null)
        Ctl Store Name : (null)
        DS Mapper Usage : Disabled
        Negotiate Client Certificate : Disabled

  10. 10 Posted by morten.faester on 09 Sep, 2014 10:04 AM

    morten.faester's Avatar

    Hey

    Tentacles are executing on Windows server 2008.

    PS C:\Windows\system32> $psversiontable

    Name Value
    ---- -----
    PSVersion 3.0
    WSManStackVersion 3.0
    SerializationVersion 1.1.0.1
    CLRVersion 4.0.30319.18444
    BuildVersion 6.2.9200.16481
    PSCompatibleVersions {1.0, 2.0, 3.0}
    PSRemotingProtocolVersion 2.2

    Regarding SNI: There are hostnames bound to the bindings as some are pm *.domainA.dk and others are *.domainB.dk, but I may not understand your question entirely.

    I have a test server where I can run almost anything.

    KR
    Morten

  11. 11 Posted by jake.stevenson on 09 Sep, 2014 12:28 PM

    jake.stevenson's Avatar

    Our servers are windows server 2012.

    PS C:\Users\jakes> $psversiontable

    Name Value
    ---- -----
    PSVersion 3.0
    WSManStackVersion 3.0
    SerializationVersion 1.1.0.1
    CLRVersion 4.0.30319.18449
    BuildVersion 6.2.9200.16628
    PSCompatibleVersions {1.0, 2.0, 3.0}
    PSRemotingProtocolVersion 2.2

    The cert is not in the personal store, it's in Web Hosting.

    Unfortunately the issue is only occurring on our production servers, and I cannot deploy to them off-schedule. I'll see if I can isolate one production server for testing, but that may not be feasible.

    We are also looking at removing the SSL certificates from the instances and only doing SSL at the load balancer. We'll probably be rolling this change out in the next two weeks, effectively solving the issue for us.

  12. 12 Posted by Nicholas Blumha... on 09 Sep, 2014 11:34 PM

    Nicholas Blumhardt's Avatar

    Thank you both for following up.

    Morten - I believe the binding configuration in Octopus allows the IP address to be specified explicitly - does this resolve the issue for you?

    Jake, glad you have an alternative to pursue - is your setup similar to Morten's in that an explicit IP address might need to be specified? Without a non-prod machine to test on I appreciate it's a bit tricky to investigate further.

    Regards,
    Nick

  13. 13 Posted by morten.faester on 10 Sep, 2014 11:42 AM

    morten.faester's Avatar

    Hey Nicholas

    My apologies. I guess IP-binding would solve the problem.

    Sincerely
    Morten

  14. 14 Posted by jake.stevenson on 10 Sep, 2014 04:35 PM

    jake.stevenson's Avatar

    I suppose we could bind to specific IPs, but our deployments go out to a growing number of instances. Does octopus expose the current tentacle's IP address(es) through a variable that we could use in the binding?

  15. 15 Posted by aknipmeyer on 11 Dec, 2014 03:15 PM

    aknipmeyer's Avatar

    is there any follow up to this discussion as it is of interest - the SSL bandwidth is not significant enough to use an off-loading/SSL Termination elsewhere and fixing this in Octopus would be excellent.

  16. Support Staff 16 Posted by Paul Stovell on 11 Dec, 2014 09:12 PM

    Paul Stovell's Avatar

    Hi,

    HTTPS requests typically don't specify a host name, just an IP address, which is why certificates need to be bound to a given IP. However, if your server supports SNI, it is possible to have multiple certificates - we've added support for this in Octopus 2.6. Hope that helps!

    Paul

  17. 17 Posted by aknipmeyer on 12 Dec, 2014 03:37 PM

    aknipmeyer's Avatar

    Thanks Paul - that's great to know and we can take the necessary action to integrate automatic certificate distribution.

  18. Paul Stovell closed this discussion on 04 Jan, 2017 07:47 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac