Upgrade from 3.14.1 to 3.17.5

gavin.woolley's Avatar

gavin.woolley

10 Oct, 2017 04:07 PM

We just performed an upgrade and when we came to log back in. Octopus server was at the first run wizard and it appeared to have lost all our projects, environments, users, roles, variables.

We logged into the server and re-enabled local user access and created a new local user

Octopus.Server.exe service --stop
Octopus.Server.exe configure --usernamePasswordIsEnabled=true
Octopus.Server.exe admin --username=XXX --password=YYY
Octopus.Server.exe service --start

This allowed us to get in find that we do have projects, environments, variables etc. But for some reason the AAD mapping of roles does not appear to be working.
Temporarily i added some roles to the everyone group to get them going again.

When i first setup the AAD integration, i tested using your steps here : https://octopus.com/docs/administration/authentication-providers/az...
This did used to return the id_token value which i could inspect. it no longer appears to return that token.

The other thing i noticed in the [User] table, the ExternalIdentifiers used to just be an email address value, now it has the following.
|Azure AD:[email blocked]|Azure AD:Gavin Woolley|Azure AD:rv6-SOSOR7fvp_aHboidhfgpsudWBxyC4h2VVCA|

Could you help assist us in returning the AAD login functionality and group to role mappings please.

  1. 1 Posted by gavin.woolley on 11 Oct, 2017 10:36 AM

    gavin.woolley's Avatar

    More digging, i have now found that in the [User] table, the [JSON] Column contains the following "SecurityGroups":{}}
    After migrating to 3.17.5. It cleared ExternalID column, and lost all group memberships.
    I deleted my user account from Octopus.
    Logged in again, which dynamically created me a new account with the correct SecurityGroups
    "SecurityGroups":{"Azure AD":{"GroupIds":["octopusAdmins","octopusDevelopers"],"LastUpdated":"2017-10-11T09:36:52.8891223+00:00"}}}

    I was then able to login and confirm i had the correct permissions.
    5 minutes later, the Server Task Synchronize external security groups ran. Which promptly wiped the values from "SecurityGroups":{}} in the [User] Table JSON column.

    I can only put it down to one of these changes

    https://github.com/OctopusDeploy/Issues/issues/3796

    https://github.com/OctopusDeploy/Issues/issues/3667

  2. Support Staff 2 Posted by Dean Malone on 11 Oct, 2017 08:17 PM

    Dean Malone's Avatar

    Hi Gavin,

    Thanks for getting in touch and I'm sorry this has been your experience in upgrading Octopus.

    Thanks for providing the additional information, it does appear that this is an issue in 3.17. We have received another support request from a customer with the same issue of AAD mapping of roles not working after 3.17.5 upgrade.

    We are currently investigating this issue and will get back to you as soon as we know more.

    Regards,
    Dean.

  3. Support Staff 3 Posted by Dean Malone on 12 Oct, 2017 01:03 AM

    Dean Malone's Avatar

    Hi Gavin,

    We have confirmed that this is indeed an issue. I have created an issue so we can keep track of this.

    We are currently working on a fix and should have that available very soon.

    Regards,
    Dean.

  4. Support Staff 4 Posted by Dean Malone on 13 Oct, 2017 06:09 AM

    Dean Malone's Avatar

    Hi Gavin,

    Just letting you know that we have resolved this issue and the fix should be available early next week.

    Regards,
    Dean.

  5. 5 Posted by gavin.woolley on 13 Oct, 2017 10:09 AM

    gavin.woolley's Avatar

    Hi Dean

    Many thanks. I look forward to it.

    There is another issue we have since installing that version.

    We use the certificates store in the library of octopus for deploying to azure app service.
    We used to either not scope or scope the certificates to the environment only.

    Since the upgrade, if the cert does not have a tenant in the scope, it wont use it. So we have had to add every tenant to every cert, rather than leaving it blank as before.

    I don’t know if this is now supposed to be this way or if it is another bug?

    Gavin

    Gavin Woolley
    Senior DevOps Engineer

  6. Support Staff 6 Posted by Michael Richard... on 16 Oct, 2017 05:20 AM

    Michael Richardson's Avatar

    Hi Gavin,

    Regarding your certificate-scoping issue:
    This isn't exactly a bug, it was an intentional change, but I do feel we haven't catered nicely for your scenario.

    This behavior change was introduced in Octopus 3.15, as part of the resolution for an issue with scoping machines to tenants.
    We standardised the tenant-scoping for machines, accounts, and certificates.

    Because your certificates weren't previously scoped to any tenants or tenant-tags, they will have defaulted to Exclude from tenanted deployments, as shown in the attached image.

    To have them included in tenanted deployments, you will need to select one of the other options, and select the tenants or tags you wish them to be scoped to (as you have apparently realized).

    There isn't currently a way to say Include in tenanted deployments and make available to all tenants. Which I believe is essentially what you want.
    The closest I can suggest is to create an All tenants tag, and add it to every tenant, and every appropriate certificate.

    We do apologize that this experience hasn't been great for you.
    Just to give you an insight into our thinking when designing this: Due to the sensitive nature of certificates (and accounts), our primary concern was safety-first. We didn't want them to be accidentally included in a deployment. This is why we made it very explicit. Obviously this came at the expense of your scenario.

    If you have any other queries on this, please don't hesitate to ask.

    Regards,
    Michael

  7. Support Staff 7 Posted by Dean Malone on 16 Oct, 2017 05:56 AM

    Dean Malone's Avatar

    Hi Gavin,

    Octopus Deploy version 3.17.6 is now available which contains a fix for the AAD role issue. With this update your users will be correctly assigned to the groups mapped from your AAD roles upon login.

    Please let us know how you go with this.

    Regards,
    Dean.

  8. Support Staff 8 Posted by Dean Malone on 17 Oct, 2017 06:10 AM

    Dean Malone's Avatar

    Hi Gavin,

    Unfortunately there is a new issue in which the Synchronize external security groups task incorrectly clears the groups linked to AzureAD (or any other OIDC provider).

    When your users login they will be correctly assigned to groups based on your AzureAD roles, however whenever the sync task runs (every hour) these groups will be removed.

    We are working on a fix and will let you know when that is available.

    Regards,
    Dean.

  9. 9 Posted by gavin.woolley on 17 Oct, 2017 07:21 AM

    gavin.woolley's Avatar

    Hi Dean

    Thanks for letting me know. I was experiencing that issue with the previous version also.
    I’ll hang on till you get the next fix out, please let me know when that will be.

    Regards

    Gavin

    Gavin Woolley
    Senior DevOps Engineer

  10. Support Staff 10 Posted by Dean Malone on 19 Oct, 2017 09:01 PM

    Dean Malone's Avatar

    Hi Gavin,

    Octopus Deploy version 3.17.8 is now available which fixes the sync external group task and permission caching update issues relating to OpenID Connect authentication providers.

    Regards,
    Dean.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac