Nuget Push fails - Internal server error - The directory is not empty

henrik.carlsson's Avatar

henrik.carlsson

18 Apr, 2017 10:38 AM

When pushing an octopack to the nuget feed of octopus it quite often results in "Internal Server Error (500)"

There is an (seemingly identical) issue here from another user which got closed:
http://help.octopusdeploy.com/discussions/problems/49803-internal-nuget-server-push-fails-with-500-internal-server-error

When looking into the logs I get this the message: "The directory is not empty."

Best regards

// Henrik

Full log from the occasion:

2017-04-18 12:24:43.4060 12 WARN Outstanding Retention tasks were not completed before the next task was due to be scheduled. If this error persists, check the Tasks tab for any running Retention tasks, and cancel them manually.
2017-04-18 12:24:46.3889 29 INFO Reader took 2264ms (132ms until the first record): SELECT ALIAS_VariableSet_0.* FROM (SELECT * FROM dbo.[VariableSet]) ALIAS_VariableSet_0 INNER JOIN (SELECT * FROM dbo.[Deployment] WHERE ([TaskId] IN (SELECT [Id] FROM [ServerTask] WHERE [State] IN ('Success', 'Failed')))) ALIAS_Deployment_1 ON ALIAS_VariableSet_0.OwnerId = ALIAS_Deployment_1.Id ORDER BY ALIAS_VariableSet_0.Id
2017-04-18 12:25:17.3521 32 INFO Reader took 350ms (59ms until the first record): SELECT * FROM dbo.[DeploymentProcess] WHERE ([IsFrozen] = 0) ORDER BY Id
2017-04-18 12:25:23.0539 32 INFO Insert took 1654ms: INSERT INTO dbo.[EventRelatedDocument] (EventId, RelatedDocumentId) values (@0__EventId, @0__RelatedDocumentId)
2017-04-18 12:25:23.0539 32 INFO Insert took 1659ms: INSERT INTO dbo.[Event] (RelatedDocumentIds, ProjectId, EnvironmentId, TenantId, Category, UserId, Username, Occurred, Message, Id, Json) values (@RelatedDocumentIds, @ProjectId, @EnvironmentId, @TenantId, @Category, @UserId, @Username, @Occurred, @Message, @Id, @Json)
2017-04-18 12:25:25.4817 32 ERROR Unhandled error on request: http://byggserver:8080/nuget/packages/ by TeamcityPublisherAccount : The directory is not empty.

System.IO.IOException: The directory is not empty.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound)
   at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
   at Octopus.Shared.Util.OctopusPhysicalFileSystem.DeleteDirectory(String path) in Y:\Work\refs\tags\3.4.12\source\Octopus.Shared\Util\OctopusPhysicalFileSystem.cs:line 85
   at Octopus.Server.Web.Api.NuGet.PackagePushActionBase`1.Execute() in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Api\NuGet\PackagePushActionBase.cs:line 105
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context) in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Infrastructure\Api\Responder.cs:line 162
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at Octopus.Server.Web.Api.OctopusRestApiModule.<>c__DisplayClass0_0.<.ctor>b__0(Object o) in Y:\Work\refs\tags\3.4.12\source\Octopus.Server\Web\Api\OctopusRestApiModule.cs:line 46
   at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)

  1. Support Staff 1 Posted by Dalmiro Grañas on 18 Apr, 2017 05:45 PM

    Dalmiro Grañas's Avatar

    Hi there,

    Thanks for reaching out! Its very possible that the temp folder in the Octopus Server is a full, causing this problem.

    If your Octopus Server service is running under a custom account, try deleting the contents of C:\Users\[user]\AppData\Local\Octopus.Server\Temp. If the service is running under Local System, then you'll have to clean up C:\Windows\Temp.

    Let me know if that works.

    Thanks!
    Dalmiro

  2. 2 Posted by henrik.carlsson on 19 Apr, 2017 07:45 AM

    henrik.carlsson's Avatar

    Hello!

    The C:\Windows\Temp contained 6mb files which are now deleted. There is 24GB available on C:. The folder can’t really be full? If files are locked causing this behavior it should be considered a bug, don’t you agree?

    Let’s see if this fixed the error, we cannot know for sure until we have had some releases.

    // Henrik

    HENRIK CARLSSON
    Utvecklare

    Växel: 031-83 36 70
    Mobil: 0725 – 63 00 65
    [email blocked]<mailto:[email blocked]>

    Johan På Gårdas Gata 5A
    412 50 Göteborg
    [cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

  3. 3 Posted by henrik.carlsson on 19 Apr, 2017 11:18 AM

    henrik.carlsson's Avatar

    Hello,

    This did not fix the issue, which is not really a surprise.

    Please let me know what to do next.

    // Henrik

    HENRIK CARLSSON
    Utvecklare

    Växel: 031-83 36 70
    Mobil: 0725 – 63 00 65
    [email blocked]<mailto:[email blocked]>

    Johan På Gårdas Gata 5A
    412 50 Göteborg
    [cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

    From: Henrik Carlsson
    Sent: den 19 april 2017 08:27
    To: 'Dalmiro Grañas' <[email blocked]>
    Subject: RE: Nuget Push fails - Internal server error - The directory is not empty [Problems #53692]

    Hello!

    The C:\Windows\Temp contained 6mb files which are now deleted. There is 24GB available on C:. The folder can’t really be full? If files are locked causing this behavior it should be considered a bug, don’t you agree?

    Let’s see if this fixed the error, we cannot know for sure until we have had some releases.

    // Henrik

    HENRIK CARLSSON
    Utvecklare

    Växel: 031-83 36 70
    Mobil: 0725 – 63 00 65
    [email blocked]<mailto:[email blocked]>

    Johan På Gårdas Gata 5A
    412 50 Göteborg
    [cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

  4. Support Staff 4 Posted by Dalmiro Grañas on 19 Apr, 2017 01:53 PM

    Dalmiro Grañas's Avatar

    Hi Henrik,

    Sad to hear that didn't help.

    Could you please confirm:

    • Under which account is your Octopus Server service running?
    • Which version of the Octopus server are you using? From the error log I can see 3.4.12, but I just want you to confirm that for me if possible.

    Regards,
    Dalmiro

  5. 5 Posted by henrik.carlsson on 19 Apr, 2017 01:59 PM

    henrik.carlsson's Avatar

    Hello

      * Local System
      * Yes, version 3.4.12

    Regards,
    Henrik

    HENRIK CARLSSON
    Utvecklare

    Växel: 031-83 36 70
    Mobil: 0725 – 63 00 65
    [email blocked]<mailto:[email blocked]>

    Johan På Gårdas Gata 5A
    412 50 Göteborg
    [cid:0662D5A3-F41E-464F-9D02-9AA2E265D603]

  6. Support Staff 6 Posted by Dalmiro Grañas on 20 Apr, 2017 12:52 PM

    Dalmiro Grañas's Avatar

    Hi Henrik,

    Sorry for the delay here. I wanted to consult this one with the dev team before replying again.

    So the 2 thoughts that came from the discussion where:

    1) In 3.7.6 we introduced the below fix to do a clean retry when we encounter a problem deleting temp files after a package push. Upgrading to at least this version ("latest" version is always recommended) would be the best thing you could do at the moment.

    https://github.com/OctopusDeploy/Issues/issues/3004

    2) Is there any chance you are running an AV scanner on your filesystem (particularly on any parent directory of the temp folder)? 9/10 times we've seen that AV scanners can get files locked up, causing all sorts of things like the one you are experiencing. If you upgrade as mentioned in (1), rather than failing, we'll display a warning when the temp file cannot be deleted after a few tries.

    Best regards,
    Dalmiro

  7. Paul Stovell closed this discussion on 25 Jul, 2017 11:07 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac