Apply retention policy on Tentacles failing after upgrade to 3.12.6

Jan Lønsetteig's Avatar

Jan Lønsetteig

21 Apr, 2017 12:01 PM

After we upgraded to Octopus version 3.12.6 today deployments started failing all over the place. They all fail in the "Apply retention policy on Tentacles" step with the same error. Reading the release notes it looks like this is related to the changes around long directory paths. The relevant paths don't can't be considered long, but they fail anyway. This is a blocking issue for us so I hope you can provide a quick fix

13:21:38   Verbose  |         Octopus Deploy: Calamari version 3.7.53+Branch.master.Sha.416708ce8cf8a2361f341285ce3655453a1be6f6
13:21:38   Verbose  |         Environment Information:
13:21:38   Verbose  |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
13:21:38   Verbose  |         OsBitVersion: x64
13:21:38   Verbose  |         Is64BitProcess: True
13:21:38   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
13:21:38   Verbose  |         MachineName: CO-OSL-TENTA145
13:21:38   Verbose  |         ProcessorCount: 4
13:21:38   Verbose  |         CurrentDirectory: C:\Octopus\Work\20170421112137-75
13:21:38   Verbose  |         TempDirectory: C:\Windows\TEMP\
13:21:38   Verbose  |         HostProcessName: Calamari
13:21:38   Info     |         Keeping this deployment and the previous 1 successful deployments
13:21:38   Verbose  |         Keeping C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.384 and C:\Octopus\Files\feeds-1\Confirmit.TaskSystem.22.0.384_E50F7F74E941694E996427BF20E94312.nupkg as it is the most recent successful release
13:21:38   Verbose  |         Keeping C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374_1 and C:\Octopus\Files\feeds-1\Confirmit.TaskSystem.22.0.374_7C9B93DD0281FE41A10B296A8B45D516.nupkg as it is the 2nd most recent successful release
13:21:38   Info     |         Removing directory 'C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374'
13:21:39   Error    |         (87) The parameter is incorrect: [\\?\C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374\Tasks\customcode]
13:21:39   Error    |         System.IO.IOException
13:21:39   Error    |         at Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
13:21:39   Error    |         at Alphaleonis.Win32.Filesystem.Directory.DeleteDirectoryCore(FileSystemEntryInfo fileSystemEntryInfo, KernelTransaction transaction, String path, Boolean recursive, Boolean ignoreReadOnly, Boolean continueOnNotExist, PathFormat pathFormat)
13:21:39   Error    |         at Calamari.Integration.FileSystem.LongPathsDirectory.Delete(String path, Boolean recursive) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\FileOperations.cs:line 77
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 414
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate`1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 403
13:21:39   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, FailureOptions options) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 369
13:21:39   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.DeleteExtractionDestination(JournalEntry deployment, List`1 preservedEntries) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 85
13:21:39   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.ApplyRetentionPolicy(String retentionPolicySet, Nullable`1 days, Nullable`1 releases) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 62
13:21:39   Error    |         at Calamari.Commands.CleanCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Commands\CleanCommand.cs:line 44
13:21:39   Error    |         at Calamari.Program.Execute(String[] args) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Program.cs:line 45
  1. Support Staff 1 Posted by Vanessa Love on 21 Apr, 2017 12:31 PM

    Vanessa Love's Avatar

    Hi Jan,

    Thanks for reporting this issue. As a temporary workaround could you disable retention policies on your Lifecycle? It will not require a new release to be created.
    Is it happening for all environments or only a few?

    Vanessa

  2. 2 Posted by Jan Lønsetteig on 21 Apr, 2017 12:36 PM

    Jan Lønsetteig's Avatar

    This is happening on several environments and is blocking our build pipeline.
    I can try to disable retention policies for short while, but disks will fill up quickly if I do this

    I've an Octopus installation on my dev machine and there I don't see this problem.

  3. Support Staff 3 Posted by Damian Brady on 21 Apr, 2017 12:53 PM

    Damian Brady's Avatar

    Hi Jan,

    You're right, that looks like an issue to do with the changes around long paths, but not because that's specifically a long path. It looks like a library we're using to deal with long file paths is failing on a recursive delete.

    To help diagnose, can you have a look at the folder it's trying to delete and let us know what's in it? The one in the log you pasted is: C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374.

    There appears to be a folder tree under there that's causing the exception when Calamari tries to delete it: Tasks\customcode.

    Could you try deleting the whole 22.0.374 folder manually (ideally using the same account that Calamari is using) and see if that succeeds?

    Thanks,
    Damian

  4. 4 Posted by Jan Lønsetteig on 21 Apr, 2017 01:10 PM

    Jan Lønsetteig's Avatar

    This has been deployed on hundreds of machines and hundreds of releases prior to 3.12.6 so I don't believe it is an issue with the directory itself

    I think I've found what is triggering the problem: All the directories that fail on delete are actually junctions. The junctions should not be traversed, but just deleted. This has worked fine in the past.

    We typically use junctions to add some shared libraries/folders to the applications. Has worked fine for years.

    I've another one for you here:

                        |       Running: Applying retention policy using 1 Items to set Environments-120/Projects-7/Step-SurveyEngine/Machines-524/<default>
    13:58:44   Verbose  |         Octopus Deploy: Calamari version 3.7.53+Branch.master.Sha.416708ce8cf8a2361f341285ce3655453a1be6f6
    13:58:44   Verbose  |         Environment Information:
    13:58:44   Verbose  |         OperatingSystem: Microsoft Windows NT 6.2.9200.0
    13:58:44   Verbose  |         OsBitVersion: x64
    13:58:44   Verbose  |         Is64BitProcess: True
    13:58:44   Verbose  |         CurrentUser: NT AUTHORITY\SYSTEM
    13:58:44   Verbose  |         MachineName: CO-OSL-TENTA185
    13:58:44   Verbose  |         ProcessorCount: 2
    13:58:44   Verbose  |         CurrentDirectory: C:\Octopus\Work\20170421115844-76
    13:58:44   Verbose  |         TempDirectory: C:\Windows\TEMP\
    13:58:44   Verbose  |         HostProcessName: Calamari
    13:58:44   Info     |         Keeping this deployment and the previous 1 successful deployments
    13:58:44   Verbose  |         Keeping C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.384 and C:\Octopus\Files\feeds-1\Firmglobal.Confirmit.SurveyEngine.Web.22.0.384_AE37837D50526C43903387A2667AC644.nupkg as it is the most recent successful release
    13:58:44   Verbose  |         Keeping C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_2 and C:\Octopus\Files\feeds-1\Firmglobal.Confirmit.SurveyEngine.Web.22.0.374_02765924AB75C640B0F5AB9C6FE86D5D.nupkg as it is the 2nd most recent successful release
    13:58:44   Info     |         Removing directory 'C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_1'
    13:58:45   Error    |         (87) The parameter is incorrect: [\?\C:\Applications\Master.Selenium.Testlab\Firmglobal.Confirmit.SurveyEngine.Web\22.0.374_1\wix\bin]
    13:58:45   Error    |         System.IO.IOException
    13:58:45   Error    |         at Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
    13:58:45   Error    |         at Alphaleonis.Win32.Filesystem.Directory.DeleteDirectoryCore(FileSystemEntryInfo fileSystemEntryInfo, KernelTransaction transaction, String path, Boolean recursive, Boolean ignoreReadOnly, Boolean continueOnNotExist, PathFormat pathFormat)
    13:58:45   Error    |         at Calamari.Integration.FileSystem.LongPathsDirectory.Delete(String path, Boolean recursive) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\FileOperations.cs:line 77
    13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 414
    13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, Predicate1 exclude, FailureOptions options, CancellationToken cancel, Boolean includeTarget) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 403
    13:58:45   Error    |         at Calamari.Integration.FileSystem.CalamariPhysicalFileSystem.PurgeDirectory(String targetDirectory, FailureOptions options) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Integration\FileSystem\CalamariPhysicalFileSystem.cs:line 369
    13:58:45   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.DeleteExtractionDestination(JournalEntry deployment, List1 preservedEntries) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 85
    13:58:45   Error    |         at Calamari.Deployment.Retention.RetentionPolicy.ApplyRetentionPolicy(String retentionPolicySet, Nullable1 days, Nullable`1 releases) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Deployment\Retention\RetentionPolicy.cs:line 62
    13:58:45   Error    |         at Calamari.Commands.CleanCommand.Execute(String[] commandLineArguments) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Commands\CleanCommand.cs:line 44
    13:58:45   Error    |         at Calamari.Program.Execute(String[] args) in Z:\buildAgent\workDir\14ffc968155e4956\source\Calamari\Program.cs:line 45
                        |
    | Failed: CO-OSL-TENTA183 13:58:52 Fatal | The remote script failed with exit code 100
  5. Support Staff 5 Posted by Damian Brady on 21 Apr, 2017 01:18 PM

    Damian Brady's Avatar

    Hi Jan,

    Ah, yes, that will definitely be it.

    I've had a look and this will require some work to resolve. Bear with me, I'm trying to work out how to downgrade your Calamari to the previous version - that's likely to be the safest option until we can create a fix.

    Damian

  6. 6 Posted by Jan Lønsetteig on 21 Apr, 2017 01:27 PM

    Jan Lønsetteig's Avatar

    It turns out to be easy to reproduce. Just create a junction in one of the deplyoment directories that will be cleaned up. Retention policy will always fail.

    In the fix, make sure that the retention policy don't delete the content of the junction target directory. This is the behavior prior to version 3.12.6. Deleting the content of the junction target directory will have VERY bad effects

    A downgrade of Calamari sounds like the best short term option. I've blocked all attempts for deployment to production now until this is resolved or Calamari is downgraded on all hundreds of machines

  7. Support Staff 7 Posted by Damian Brady on 21 Apr, 2017 01:32 PM

    Damian Brady's Avatar

    Hi Jan,

    It looks like you may just be able to deploy the previous version of Octopus Server over the top, and it will downgrade the Calamari version for you when the next deployment occurs. The version is checked as part of the deployment, so it should have immediate effect.

    Are you able to try this and let me know if it resolves the issue for the time being?

    I'm really sorry for the problems here - it's an edge case we didn't consider. And yes, definitely noted that we don't want to delete the contents of the junction.

    Damian

  8. 8 Posted by Jan Lønsetteig on 21 Apr, 2017 01:36 PM

    Jan Lønsetteig's Avatar

    Hi Damian,

    We have installed Octopus using chocolatey. What procedure should I follow to downgrade from version 3.12.6 to 3.12.5?

    All tentacles have also been upgraded to 3.12.6. Will/can they still stay on 3.12.6?

    /Jan

  9. 9 Posted by Jan Lønsetteig on 21 Apr, 2017 02:16 PM

    Jan Lønsetteig's Avatar

    I'll try with this one:
    choco install OctopusDeploy --version 3.12.5 -yf

    and keeping the tentacles on version 3.12.6

  10. Support Staff 10 Posted by Damian Brady on 21 Apr, 2017 02:17 PM

    Damian Brady's Avatar

    Hi Jan,

    I'm not an expert on Chocolately, but I think you can downgrade an installation using these instructions. I'd recommend not doing an uninstall and reinstall (although that should still work), but doing an install and specifying the previous version instead.

    The tentacles should be fine running a version higher. Because it's a patch release, the versions should be compatible.

    I've created an issue in GitHub that you can follow along with as well: https://github.com/OctopusDeploy/Issues/issues/3432

    Thanks,
    Damian

  11. 11 Posted by Jan Lønsetteig on 21 Apr, 2017 02:34 PM

    Jan Lønsetteig's Avatar

    Hi Damian,

    Downgrade to 3.12.5 on the server fixed the issue. Calamari was downgraded also and are now working again.

    Thanks for your help

    We like to stay current on the Octopus version so hopefully the next version will have a fix for this issue

    Thanks,
    Jan

  12. Jan Lønsetteig closed this discussion on 27 Apr, 2017 12:09 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac