Not possible to deploy using AWS Cloudformation

Juha Sarvi's Avatar

Juha Sarvi

17 Feb, 2018 02:00 PM

I'm running Octopus Version: 2018.2.3-x64 and I have AWSToolsAndSDKForNet_sdk-3.3.234.0_ps-3.3.234.0_tk-1.14.1.3 as well as latest AWSCLI64 installed on the Octopus Server. When I try to deploy an AWS CloudFormation template running on the Octopus Server, I get error 502, bad gateway.

The same CloudFormation template gets deployed using the same AWS Account with the "Run an AWS CLI script"-step and using the New-CFNStack Powershell command.

I'm suspecting proxy problems as I am running using a corporate NTLM proxy. This is however configured in the Octopus Manager setup as well as the proxy settings are defined for the service account running the Octopus service.

Stacktrace:
13:47:14 Error | Amazon.Runtime.AmazonServiceException: A WebException with status ProtocolError was thrown. ---> System.Net.WebException: The remote server returned an error: (502) Bad Gateway.
13:47:14 Error | at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
13:47:14 Error | at System.Net.HttpWebRequest.GetRequestStream()
13:47:14 Error | at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | --- End of inner exception stack trace ---
13:47:14 Error | at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
13:47:14 Error | at Amazon.CloudFormation.AmazonCloudFormationClient.DescribeStacks(DescribeStacksRequest request)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.StackExists(Boolean defaultValue)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.WaitForStackToComplete(RunningDeployment deployment, Boolean expectSuccess, Boolean missingIsFailure)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.DeployCloudFormation(RunningDeployment deployment)
13:47:14 Error | at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
13:47:14 Error | at Calamari.Deployment.ConventionProcessor.RunConventions()
13:47:14 Error | Running rollback conventions...
13:47:14 Error | A WebException with status ProtocolError was thrown.
13:47:14 Error | Amazon.Runtime.AmazonServiceException
13:47:14 Error | at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller)
13:47:14 Error | at Amazon.CloudFormation.AmazonCloudFormationClient.DescribeStacks(DescribeStacksRequest request)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.StackExists(Boolean defaultValue)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.WaitForStackToComplete(RunningDeployment deployment, Boolean expectSuccess, Boolean missingIsFailure)
13:47:14 Error | at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.DeployCloudFormation(RunningDeployment deployment)
13:47:14 Error | at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
13:47:14 Error | at Calamari.Deployment.ConventionProcessor.RunConventions()
13:47:14 Error | at Calamari.Aws.Commands.DeployCloudFormationCommand.Execute(String[] commandLineArguments)
13:47:14 Error | at Calamari.Program.Execute(String[] args)
13:47:14 Error | --Inner Exception--
13:47:14 Error | The remote server returned an error: (502) Bad Gateway.
13:47:14 Error | System.Net.WebException
13:47:14 Error | at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
13:47:14 Error | at System.Net.HttpWebRequest.GetRequestStream()
13:47:14 Error | at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
13:47:14 Error | at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)

  1. Support Staff 1 Posted by Matthew Caspers... on 19 Feb, 2018 04:04 AM

    Matthew Casperson's Avatar

    Hi Juha, thanks for reaching out.

    We have confirmed this as an issue with the CloudFormation steps, and aGitHub issue at https://github.com/OctopusDeploy/Issues/issues/4298 has been created to track it. Thanks for lettings us know of the problem.

    Regards
    Matt C

  2. 2 Posted by Juha Sarvi on 26 Feb, 2018 03:18 PM

    Juha Sarvi's Avatar

    Re-tested this on latest Octopus Server version: 2018.2.6+Branch.master.Sha.06669b6b137884448caf75d43e96d25dc40c881e

    Still got an error:

    Amazon.Runtime.AmazonServiceException: A WebException with status ProtocolError was thrown. ---> System.Net.WebException: The remote server returned an error: (502) Bad Gateway.
    February 26th 2018 15:51:01
    Error
       at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
    February 26th 2018 15:51:01
    Error
       at System.Net.HttpWebRequest.GetRequestStream()
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       --- End of inner exception stack trace ---
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.AmazonServiceClient.InvokeTRequest,TResponse
    February 26th 2018 15:51:01
    Error
       at Amazon.CloudFormation.AmazonCloudFormationClient.DescribeStacks(DescribeStacksRequest request)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.StackExists(Boolean defaultValue)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.WaitForStackToComplete(RunningDeployment deployment, Boolean expectSuccess, Boolean missingIsFailure)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.DeployCloudFormation(RunningDeployment deployment)
    February 26th 2018 15:51:01
    Error
       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
    February 26th 2018 15:51:01
    Error
       at Calamari.Deployment.ConventionProcessor.RunConventions()
    February 26th 2018 15:51:01
    Error
    Running rollback conventions...
    February 26th 2018 15:51:01
    Error
    A WebException with status ProtocolError was thrown.
    February 26th 2018 15:51:01
    Error
    Amazon.Runtime.AmazonServiceException
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.AmazonServiceClient.InvokeTRequest,TResponse
    February 26th 2018 15:51:01
    Error
       at Amazon.CloudFormation.AmazonCloudFormationClient.DescribeStacks(DescribeStacksRequest request)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.StackExists(Boolean defaultValue)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.WaitForStackToComplete(RunningDeployment deployment, Boolean expectSuccess, Boolean missingIsFailure)
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Deployment.Conventions.DeployAwsCloudFormationConvention.DeployCloudFormation(RunningDeployment deployment)
    February 26th 2018 15:51:01
    Error
       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
    February 26th 2018 15:51:01
    Error
       at Calamari.Deployment.ConventionProcessor.RunConventions()
    February 26th 2018 15:51:01
    Error
       at Calamari.Aws.Commands.DeployCloudFormationCommand.Execute(String[] commandLineArguments)
    February 26th 2018 15:51:01
    Error
       at Calamari.Program.Execute(String[] args)
    February 26th 2018 15:51:01
    Error
    --Inner Exception--
    February 26th 2018 15:51:01
    Error
    The remote server returned an error: (502) Bad Gateway.
    February 26th 2018 15:51:01
    Error
    System.Net.WebException
    February 26th 2018 15:51:01
    Error
       at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
    February 26th 2018 15:51:01
    Error
       at System.Net.HttpWebRequest.GetRequestStream()
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Error
       at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
    February 26th 2018 15:51:01
    Fatal
    The remote script failed with exit code 100
    February 26th 2018 15:51:01
    Fatal
    The action Cloud Test on the Octopus Server failed

    I do see that our corporate proxy settings are used now:

    HEAD: CONNECT cloudformation.eu-central-1a.amazonaws.com:443 HTTP/1.1
    Reading headers (9)...
    HEAD: HTTP/1.1 502 badgateway
    Via => 1.1 172.22.128.10 (McAfee Web Gateway 7.7.1.3.
    0.23237)
    Content-Type => text/html
    Cache-Control => no-cache
    Content-Length => 2660
    Proxy-Connection => Keep-Alive
    Sending headers (8)...
    Body included. Length: 2660
    data_send: read 2048 of 2048 / 2048 of 2660 (errno = ok)
    data_send: wrote 2048 of 2048
    data_send: read 612 of 612 / 2660 of 2660 (errno = ok)
    data_send: wrote 612 of 612
    Body sent.
    forward_request: palive=1, authok=1, ntlm=0, closed=0

  3. 3 Posted by Juha Sarvi on 26 Feb, 2018 06:32 PM

    Juha Sarvi's Avatar
  4. Support Staff 4 Posted by Matthew Caspers... on 27 Feb, 2018 12:33 AM

    Matthew Casperson's Avatar

    Hi Juha, I'm sorry to hear that the latest changes didn't solve your issue.

    What proxy server are you trying to connect to? We may be able to configure a local instance to try and replicate the problem.

    Regards
    Matt C

  5. 5 Posted by Juha Sarvi on 27 Feb, 2018 10:42 AM

    Juha Sarvi's Avatar

    I had a typo in specifying the AWS region. The proxy settings now works as expected.

    Best Regards,
    Juha

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac