System.Threading.Tasks.TaskCanceledException: A task was canceled.

Joe Burnside's Avatar

Joe Burnside

24 Jan, 2018 04:26 PM

Today we have encountered a problem across all deployments on our octopus server that previously worked. We have process steps that stand up a web application within Azure and then push code on to it using a nuget package from a hosted feed. The step that pushes the code fails with the error -
System.Threading.Tasks.TaskCanceledException: A task was canceled.

The task seems to timeout after 100 seconds and octopus then cancels the request. What we are trying to work out is if this timeout is an Octopus issue, or a Microsoft issue. The step that provisions the web application works fine. It is the code deploy that we are having problems with.

If this is a microsoft issue, is there any way that we can prove it so their support team don't just point the finger at the Octopus server? What method does the octopus step use to push code onto an Azure web application so we can try running this step locally?

Attached is the full error message.

  1. Support Staff 1 Posted by Matthew Caspers... on 25 Jan, 2018 12:49 AM

    Matthew Casperson's Avatar

    Hi Joe, thanks for reaching out.

    The code that executes Azure deployments is held in the open source calamari package, which can be found at https://github.com/OctopusDeploy/Calamari/tree/master/source/Calama.... However we do not have an way to map that process to something that can be easily replicated outside of Octopus in a 1 to 1 fashion.

    We have seen similar issues when the app and slot name combined exceeds 40 characters (https://social.msdn.microsoft.com/Forums/en-US/938e59f6-6a83-4640-a...).

    If your deployment might be exceeding this 40 character limit then I would recommend reducing the names and seeing if that solves the issue.

    If that doesn't solve the issue then we'll need to get a copy of the deployment log with the variables expanded to help diagnose the problem. The docs at https://octopus.com/docs/support/debug-problems-with-octopus-variab... detail how to include variables in the log files, and https://octopus.com/docs/support/get-the-raw-output-from-a-task details how to get the raw deployment logs.

    Regards
    Matt C

  2. 2 Posted by Keethanjan Amut... on 20 Mar, 2018 01:32 PM

    Keethanjan Amuthalingam's Avatar

    Hi Matthew,

    We are experiencing the same issue since yesterday. And as far as we know, we didn't modify anything regarding to this. Could you help us? And I also ran a powershell script which executed the scripts which we run on Azure (stopping and starting the ResourceGroup and webjob). This however worked fine (the job was started / stopped accordingly ).

    Get-AzureRmWebApp -Name Name -ResourceGroupName ResourceGroupName | Stop-AzureRmWebApp
    Get-AzureRmWebApp -Name Name -ResourceGroupName ResourceGroupName | Start-AzureRmWebApp

    And
    Invoke-AzureRmResourceAction -ResourceGroup ResourceGroupName -ResourceType microsoft.web/sites/continuouswebjobs -ResourceName resourceName/webjobName -Action Start -ApiVersion 2016-08-01 -Force

    Could you guys help us with this? Thanks.

    Regards,

    Keethanjan Amuthalingam

  3. Support Staff 3 Posted by Matthew Caspers... on 21 Mar, 2018 02:50 AM

    Matthew Casperson's Avatar

    Hi Keethanjan, thanks for reaching out.

    Version 2018.3.6 of Octopus includes some additional logging for this exception that will help us identify the underlying cause of the issue. I expect 2018.3.6 to be released either today or tomorrow. Once it is release can I get you to install the update and capture the error logs again? We can then use the additional output to try and solve the issue.

    Regards
    Matt C

  4. 4 Posted by Keethanjan Amut... on 21 Mar, 2018 05:24 AM

    Keethanjan Amuthalingam's Avatar

    Hi Matthew,

    Of course. Thanks for the quick response! Do I need to do anything to enable the logging after installing 2018.3.6?

    Regards,

    Keethanjan Amuthalingam

  5. Support Staff 5 Posted by Matthew Caspers... on 21 Mar, 2018 05:26 AM

    Matthew Casperson's Avatar

    Hi Keethanjan,

    No, you don't need to enable anything. The additional logging will be enabled by default.

    Regards
    Matt C

  6. 6 Posted by Keethanjan Amut... on 21 Mar, 2018 12:59 PM

    Keethanjan Amuthalingam's Avatar

    Hi Matthew,

    After installing the latest version, 2018.3.6, I tried to rerun the deployment multiple times. There are several things which I noticed (see attachments for the log files):

    1. Invoke-AzureRmResourceAction -ResourceGroup ResourceGroupName -ResourceType microsoft.web/sites/continuouswebjobs -ResourceName is quiet slow. It took about 10 minutes to stop a webjob. (see for logging 1.txt, removed the personal stuff in here).

    2. Invoke-AzureRmResourceAction failed due to Operation failed because a request timed out. However this is a bit strange, because the first time I ran it, it took about 10 minutes and it didn't time out. The second time I ran it, it timed out after 2 minutes.

    3. The feature Deploy website failed due to the operation cancelled exception. However I could not find any extra logging.

    See attachments.

    Regards,

    Keethanjan Amuthalingam

  7. Support Staff 7 Posted by Matthew Caspers... on 21 Mar, 2018 10:05 PM

    Matthew Casperson's Avatar

    Hi Keethanjan,

    In this case the lack of the additional logging is also useful. The error System.Threading.Tasks.TaskCanceledException: A task was canceled with no additional logging would suggest that there is no other exception being thrown to trigger this, and it is most likely a case of the requests to Azure timing out.

    The fact that regular Azure Powershell commands are timing out, in addition to the Octopus code, would also suggest that it is a network issue rather than a misconfigured step or other exception.

    What kind of network connection does the Octopus server have to the Internet and to Azure? Is the connection stable, or is it unreliable? Are there any firewalls, proxies, VPNs or other network devices in between the Octopus server and the Internet/Azure?

    You could measure the reliability of the network connection to Azure by sampling a number of calls to the API.

    To do this, save the following code to a file called time.ps1 on the Octopus server.

    function Measure-Command2 ([ScriptBlock]$Expression, [int]$Samples = 1, [Switch]$Silent, [Switch]$Long) {
    <#
    .SYNOPSIS
      Runs the given script block and returns the execution duration.
      Discovered on StackOverflow. http://stackoverflow.com/questions/3513650/timing-a-commands-execution-in-powershell
     
    .EXAMPLE
      Measure-Command2 { ping -n 1 google.com }
    #>
      $timings = @()
      $failures = 0
      do {
        $sw = New-Object Diagnostics.Stopwatch
        if ($Silent) {
          $sw.Start()
          $null = & $Expression
          $sw.Stop()
          Write-Host "." -NoNewLine
        }
        else {
          $sw.Start()
          try {
            & $Expression
          }
          catch
          {
            ++$failures
          }
          $sw.Stop()
        }
        $timings += $sw.Elapsed
        
        $Samples--
      }
      while ($Samples -gt 0)
     
      Write-Host
     
      $stats = $timings | Measure-Object -Average -Minimum -Maximum -Property Ticks
     
      # Print the full timespan if the $Long switch was given.
      if ($Long) {  
        Write-Host "Avg: $((New-Object System.TimeSpan $stats.Average).ToString())"
        Write-Host "Min: $((New-Object System.TimeSpan $stats.Minimum).ToString())"
        Write-Host "Max: $((New-Object System.TimeSpan $stats.Maximum).ToString())"
      }
      else {
        # Otherwise just print the milliseconds which is easier to read.
        Write-Host "Avg: $((New-Object System.TimeSpan $stats.Average).TotalMilliseconds)ms"
        Write-Host "Min: $((New-Object System.TimeSpan $stats.Minimum).TotalMilliseconds)ms"
        Write-Host "Max: $((New-Object System.TimeSpan $stats.Maximum).TotalMilliseconds)ms"
      }
      Write-Host "Failures: $failures"
    }
    
    Set-Alias time Measure-Command2
    

    In a Powershell window on the Octopus server (i.e. not in Octopus itself, just on the Octopus server as this will prevent any Octopus code from interfering with the results), import the function with:

    . ./time.ps1
    

    Finally run an Azure Powershell command a number of times to generate some statistics (replace AWebApp with the name of one of your web apps):

    time {Get-AzureRmWebApp -Name AWebApp} -Samples 50
    

    You will then get a report like this:

    Avg: 3169.5766ms
    Min: 2630.0123ms
    Max: 4242.0634ms
    Failures: 0
    

    If there were any failures, or if the Max value shows a command took a significant amount of time to complete, then you can be confident that there is a network issue.

  8. 8 Posted by Keethanjan Amut... on 22 Mar, 2018 05:45 AM

    Keethanjan Amuthalingam's Avatar

    Hi Matthew,

    During last night I triggered multiple deployments and all of them succeeded. I also verified the status in Azure, seeing the following text ( https://azure.microsoft.com/en-gb/status/history/ ):

    Summary of impact: Between 22:10 UTC on 20 Mar 2018 and 13:30 UTC on 21 Mar 2018, a subset of customers using App Service and App Service Linux in multiple regions may have experienced issues when performing Service Management functions. Errors may have included timeouts, portal blades failing to load, or failures when leveraging Service Management API's. Runtime availability for App Service was not impacted during this time.

    After running the powershell script twice, the results were:

    Avg: 3574.68
    Min: 2319.05
    Max: 7484.29
    Failures: 0
    
    Avg: 3266.2921ms
    Min: 2385.9344ms
    Max: 8340.1566ms
    Failures: 0
    

    I will keep an eye on this. If this issue persists I will verify it with IT and / or Microsoft Support.

    Thank you for your help!

    Regards,

    Keethanjan Amuthalingam

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Already uploaded files

  • log.PNG 155 KB

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac