Upgrade issues from 3.13.7 to 3.17.13

Lucas's Avatar

Lucas

09 Nov, 2017 03:46 PM

Hello,
I've performed OD upgrade in my 2-node HA test environment.
I did all the backups, entered maintenance mode and drained node connections.
I then performed upgrade on 01 node - upgrade went fine, but right after that I saw duplicate entry under Nodes tab in the console for the 02 server.
I then upgraded node 02 without any issues.

Now I have 3 nodes present under the Nodes tab. Server 01, Server 02 and again Server 02 (last entry for server 02 indicates that server is offline). I cannot delete that offline node. When I try to do so, I get the following error message:

'Server02' is not a valid document Id. It should have the format '<DocumentType>-<Identity>'. Example: 'Projects-101'.
System.Exception
   at Octopus.Core.Model.Events.EventBuilder.AppendReference(String displayText, String documentId)
   at Octopus.Server.Web.Infrastructure.Api.DeleteOnBackgroundResponseDescriptor`2.Responder.Execute()
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.ExecuteRegistered()
   at Octopus.Server.Web.Infrastructure.Api.Responder`1.Respond(TDescriptor options, NancyContext context)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at CallSite.Target(Closure , CallSite , Object , Object , NancyContext )
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object x)
   at CallSite.Target(Closure , CallSite , Func`2 , Object )
   at Nancy.Routing.Route.<>c__DisplayClass4.<Wrap>b__3(Object parameters, CancellationToken context)

Did I miss a step or is there something wrong with this upgrade process?

  1. 1 Posted by Lucas on 09 Nov, 2017 04:25 PM

    Lucas's Avatar

    Looking at the database under [dbo].[OctopusServerNode] it appears that by upgrading server 01, both nodes information has been converted to new format, leaving original node 2 name in there.

    Working nodes have entries seen as:
    OctopusServerNodes-Server01
    OctopusServerNodes-Server02

    and the 3rd entry is:
    Server02

    I suspect the same issue will occur in my Production environment so I'd appreciate your feedback on proper remediation steps.

  2. Support Staff 2 Posted by Michael Richard... on 10 Nov, 2017 02:46 AM

    Michael Richardson's Avatar

    Hi Lucas,

    We apologize for this.

    As you can see, we changed the format of the ID's of the OctopusServerNode table, and it caused this problem for HA installations.

    You can safely delete the Server02 entry from the OctopusServerNode table.

    You are correct, it is possible you will encounter the same issue on your production cluster.

    The two options are:

    • Stop the Octopus Server service on all nodes in the cluster, and bring them back up as you upgrade them. We're very aware this defeats the purpose of a HA cluster, so...
    • You can do the same as in your test environment, and delete the phantom record (if it appears) from the database after the upgrade. For safety, I would recommend putting the cluster in maintenance mode, to ensure the legacy node ID is not written into any other tables.

    Again, we sincerely apologize for the inconvenience. Please don't hesitate to ask if there is anything else we can do to help.

    Regards,
    Michael

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac