Service Fabric deploy - Waiting for upgrade...

john.zen's Avatar

john.zen

11 Jan, 2018 09:44 AM

Octopus version 3.17.14

When I deploy my Service Fabric package first time, it all works fine.

On my subsequent deployment, the deployment stuck at "Waiting for upgrade"...

Log as follow:

Task ID: ServerTasks-13601
Task status: Canceled
Task queued: Thursday, January 11, 2018 10:07 PM
Task started: Thursday, January 11, 2018 10:07 PM
Task duration: 11 minutes
Server version: 3.17.14+Branch.master.Sha.582e2b1203ecda0df06e121f68db11ee20ee583d
Server node: DEV-BLD4-MERLOT

                    | == Failed: Deploy Merlot deploy Service Fabric release 7.3.0.59813 to Dev for Dev_Deploy ==
22:07:55 Verbose | Guided failure is not enabled for this task
22:18:25 Info | Requesting cancellation...
                    |
                    | == Success: Acquire packages ==
22:07:55 Info | Acquiring packages
22:07:55 Info | Making a list of packages to acquire
22:07:55 Verbose | No packages are required on any deployment targets
22:07:55 Verbose | Acquire Merlot.Aero.ServiceFabric.Application.7.3.0.59813 on Octopus Server
22:07:55 Verbose | Checking package cache for package Merlot.Aero.ServiceFabric.Application.7.3.0.59813
22:07:55 Info | Downloading Merlot.Aero.ServiceFabric.Application.7.3.0.59813 to package cache...
22:07:55 Verbose | Downloading Merlot.Aero.ServiceFabric.Application.7.3.0.59813 from NuGet feed 'Team City Builds' at http://localhost/httpAuth/app/nuget/v1/FeedService.svc/ using cache policy UseCache...
22:07:55 Verbose | Downloading package (attempt 1 of 5)
22:07:57 Verbose | Downloaded Merlot.Aero.ServiceFabric.Application.7.3.0.59813 (26.135 MB) to 'E:\Octopus\OctopusServer\PackageCache\feeds-team-city-builds\Merlot.Aero.ServiceFabric.Application.7.3.0.59813_241316217EEC9D458A167550D6C2F4EB.nupkg' with SHA1 9b0b988e979a7e8191706015aa6e5b82c9ae51e2.
22:07:57 Info | Downloaded Merlot.Aero.ServiceFabric.Application.7.3.0.59813 (26.135 MB) with SHA1 9b0b988e979a7e8191706015aa6e5b82c9ae51e2.
22:07:57 Info | All packages have been acquired
22:07:57 Verbose | Acquire Packages completed
                    |
                    | Success: Octopus Server
                    |
                    | Success: Download package Merlot.Aero.ServiceFabric.Application.7.3.0.59813 from NuGet feed 'Team City Builds' at http://localhost/httpAuth/app/nuget/v1/FeedService.svc/
22:07:57 Info | Successfully downloaded Merlot.Aero.ServiceFabric.Application 7.3.0.59813 (26.135 MB)
22:07:57 Verbose | Stored package in E:\Octopus\OctopusServer\PackageCache\feeds-team-city-builds\Merlot.Aero.ServiceFabric.Application.7.3.0.59813_241316217EEC9D458A167550D6C2F4EB.nupkg
                    |
                    | == Failed: Step 1: Deploy Service Fabric ==
22:18:26 Fatal | The step failed: This task has been canceled.
22:18:26 Info | The operation was canceled.
                    |
                    | Failed: Octopus Server
22:07:57 Verbose | Octopus Server version: 3.17.14+Branch.master.Sha.582e2b1203ecda0df06e121f68db11ee20ee583d
22:07:57 Verbose | Environment Information:
                    | OperatingSystem: Microsoft Windows NT 6.3.9600.0
                    | OsBitVersion: x64
                    | Is64BitProcess: True
                    | CurrentUser: NT AUTHORITY\SYSTEM
                    | MachineName: DEV-BLD4-MERLOT
                    | ProcessorCount: 8
                    | CurrentDirectory: C:\Windows\system32
                    | TempDirectory: C:\Windows\TEMP\
                    | HostProcessName: Octopus.Server
                    | PID: 7296
22:08:00 Verbose | Octopus Deploy: Calamari.Azure version 3.7.116
22:08:00 Verbose | Environment Information:
22:08:00 Verbose | OperatingSystem: Microsoft Windows NT 6.3.9600.0
22:08:00 Verbose | OsBitVersion: x64
22:08:00 Verbose | Is64BitProcess: True
22:08:00 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
22:08:00 Verbose | MachineName: DEV-BLD4-MERLOT
22:08:00 Verbose | ProcessorCount: 8
22:08:00 Verbose | CurrentDirectory: E:\Octopus\Work\20180111090757-13601-13
22:08:00 Verbose | TempDirectory: C:\Windows\TEMP\
22:08:00 Verbose | HostProcessName: Calamari.Azure
22:08:00 Info | Deploying package: E:\Octopus\OctopusServer\PackageCache\feeds-team-city-builds\Merlot.Aero.ServiceFabric.Application.7.3.0.59813_241316217EEC9D458A167550D6C2F4EB.nupkg
22:08:02 Verbose | Extracting package to: E:\Octopus\Work\20180111090757-13601-13\staging
22:08:16 Verbose | Extracted 312 files
22:08:17 Info | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessorPkg\Code\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessor.exe.config'
22:08:17 Info | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessorPkg\Code\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessor.exe.config'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessorPkg\Code\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessor.exe.config'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessorPkg\Code\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessor.exe.config'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\ApplicationManifest.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\ApplicationParameters\PublishedParameters.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessorPkg\ServiceManifest.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapRequestProcessorPkg\Config\Settings.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessorPkg\ServiceManifest.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\Merlot.Aero.ServiceFabric.Services.SwapResponseProcessorPkg\Config\Settings.xml'
22:08:17 Verbose | Performing variable substitution on 'E:\Octopus\Work\20180111090757-13601-13\staging\PublishProfiles\PublishedProfile.xml'
22:08:20 Verbose | Name Value
22:08:20 Verbose | ---- -----
22:08:20 Verbose | PSVersion 4.0
22:08:20 Verbose | WSManStackVersion 3.0
22:08:20 Verbose | SerializationVersion 1.1.0.1
22:08:20 Verbose | CLRVersion 4.0.30319.42000
22:08:20 Verbose | BuildVersion 6.3.9600.18773
22:08:20 Verbose | PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
22:08:20 Verbose | PSRemotingProtocolVersion 2.2
22:08:21 Verbose | PowerShell Environment Information:
22:08:21 Verbose | OperatingSystem: Microsoft Windows NT 6.3.9600.0
22:08:21 Verbose | OsBitVersion: x64
22:08:21 Verbose | Is64BitProcess: True
22:08:21 Verbose | CurrentUser: NT AUTHORITY\SYSTEM
22:08:21 Verbose | MachineName: DEV-BLD4-MERLOT
22:08:21 Verbose | ProcessorCount: 8
22:08:21 Verbose | CurrentDirectory: E:\Octopus\Work\20180111090757-13601-13\staging
22:08:21 Verbose | CurrentLocation: E:\Octopus\Work\20180111090757-13601-13\staging
22:08:22 Verbose | TempDirectory: C:\Windows\TEMP\
22:08:22 Verbose | HostProcessName: powershell
22:08:22 Verbose | TotalPhysicalMemory: 14679604 KB
22:08:22 Verbose | AvailablePhysicalMemory: 7248004 KB
22:08:22 Verbose | Connecting to Service Fabric unsecurely.
22:08:22 Verbose | Authenticating with Service Fabric.
22:08:22 Verbose | Using ConnectionParameters:
22:08:22 Verbose | ConnectionEndpoint=devtest-sfabric.westus.cloudapp.azure.com:19000
22:08:29 Verbose | Invoking target script "E:\Octopus\Work\20180111090757-13601-13\staging\DeployToServiceFabric.ps1" with parameters.
22:08:29 Verbose | Importing ServiceFabricSDK modules from C:\Program Files\Microsoft SDKs\Service Fabric\Tools\PSModule\ServiceFabricSDK
22:08:52 Info | Copying application package to image store...
22:09:10 Info | Upload to Image Store succeeded
22:09:10 Info | Registering application type...
22:09:35 Info | Register application type succeeded
22:09:35 Info | Start upgrading application...
22:09:40 Info | ApplicationName : fabric:/Merlot.Aero.AzureServiceFabricApplicati
22:09:40 Info | on
22:09:40 Info | ApplicationParameters : { "SwapResponseProcessor_InstanceCount" = "-1";
22:09:40 Info | "SwapRequestProcessor_InstanceCount" = "-1" }
22:09:40 Info | TargetApplicationTypeVersion : 7.3.0.59813
22:09:40 Info | UpgradeKind : Rolling
22:09:40 Info | ForceRestart : False
22:09:40 Info | UpgradeMode : UnmonitoredAuto
22:09:40 Info | UpgradeReplicaSetCheckTimeout : 49710.06:28:15
22:09:40 Info | Waiting for upgrade...
22:09:43 Info | Waiting for upgrade...
22:09:47 Info | Waiting for upgrade...
22:09:50 Info | Waiting for upgrade...
22:09:53 Info | Waiting for upgrade...
22:09:56 Info | Waiting for upgrade...
22:09:59 Info | Waiting for upgrade...
22:10:03 Info | Waiting for upgrade...
22:10:06 Info | Waiting for upgrade...
22:10:09 Info | Waiting for upgrade...
...

  1. Support Staff 1 Posted by Mark Siedle on 12 Jan, 2018 02:28 AM

    Mark Siedle's Avatar

    Hi John,

    Thanks for getting in touch.

    Could you confirm if that Service Fabric cluster ever completed the upgrade in Azure? As in, if you cancelled the Octopus task, did the deployment to the cluster ever complete its deployment/upgrade operation on Azure's side (and if so, how long did the operation take to complete)?

    Cheers
    Mark

  2. 2 Posted by john.zen on 12 Jan, 2018 03:25 AM

    john.zen's Avatar

    Hi Mark,

    It went on for an hour, then I stopped the deployment on Octopus.

    The status on Service Fabric is still "Upgrading" (as of now), the progress bar shows % completed is 0.

    regards

    John Zen

    ________________________________

  3. 3 Posted by john.zen on 13 Jan, 2018 12:19 AM

    john.zen's Avatar

    Hi Mark,

    I changed the setting of Override upgrade behaviour from Force upgrade to None. It is now able to upgrade.

    regards

    John Zen

    ________________________________
    From: John Zen
    Sent: Friday, 12 January 2018 4:25:28 p.m.
    To: Mark Siedle
    Subject: Re: Service Fabric deploy - Waiting for upgrade... [Problems #67790]

    Hi Mark,

    It went on for an hour, then I stopped the deployment on Octopus.

    The status on Service Fabric is still "Upgrading" (as of now), the progress bar shows % completed is 0.

    regards

    John Zen

    ________________________________

  4. Support Staff 4 Posted by Mark Siedle on 15 Jan, 2018 12:48 AM

    Mark Siedle's Avatar

    Interesting, thanks for letting us know John.

    This OverrideUpgradeBehaviour adds some additional parameters, which then get forwarded through to the Publish-NewServiceFabricApplication Azure PowerShell cmdlet (as can be seen here). When we created this feature, we imported the PowerShell used by Visual Studio (and all the supported parameters). We're not entirely sure of the use-cases for these upgrade parameters, so we'd recommend contacting Azure directly if you'd like to know more about these Service Fabric parameters.

    Glad you got it sorted.
    Mark

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac