Migrating variables with sensitive values

Lyn Ko's Avatar

Lyn Ko

12 Dec, 2017 05:11 AM

Hi, I migrated our octopus server to another server and db using the export/import utililty (I could not simple restore the database because the version of the db server was lower at the destination).

I was just wondering if
1. Are they additional steps required for variables with sensitive values
2. How to link existing packages in the package repository on the server with what's in the db because I am getting an error when trying to deploy - package does not exist

thanks

  1. Support Staff 1 Posted by Alex Rolley on 13 Dec, 2017 04:30 AM

    Alex Rolley's Avatar

    Hi Lyn,

    Thanks for getting in touch!

    With your first question, as long as you have a copy of your Master Key you can access the sensitive parts of the database with your new instance. As for your second question it should just be a case of copying all the data to the new server, can you confirm that you have migrated all of the required folders as per our migration documentation?

    Please let me know if you are still having issues, we are here to help!

    Regards,

    Alex

  2. 2 Posted by Lyn Ko on 13 Dec, 2017 04:54 AM

    Lyn Ko's Avatar

    Hi Alex

    Interestingly enough I did no have to migrate the master key.
    Here’s what I did

    1. Installed octopus on new server

    2. Export data from old server

    3. Import data to new server

    4. Copied folders

    We deployed the releases and they worked and decrypted the sensitive data without problem
    But if try and look at the application and try to deploy it says ‘The resource ‘xxxxxxx version xxxxx’ was not found.

    It seems that the pointer to the package is not correct…..I ensured that I copied the packages to the same folder structure on the new server and the package does exist.

    Any ideas?

  3. Support Staff 3 Posted by Michael Richard... on 14 Dec, 2017 02:01 AM

    Michael Richardson's Avatar

    Lyn,

    I'm sorry, I don't understand what you are attempting to do when receiving the error. Could you possibly explain further what you are doing and where you are seeing the error?

    If I understand correctly, you were able to successfully deploy your release, which means the packages were found.

    The master key is not required when importing\exporting, so that part makes sense.

    Regards,
    Michael

  4. 4 Posted by Lyn Ko on 14 Dec, 2017 02:03 AM

    Lyn Ko's Avatar

    What I mean is, after I have migrated, if I go back into a Project and click on a previously deployed package. It’s not available to deploy.
    What I need to do is create a new package(pushed from teamcity) and then deploy the new package

    Thanks
    Lyn

  5. Support Staff 5 Posted by Michael Richard... on 14 Dec, 2017 02:10 AM

    Michael Richardson's Avatar

    Could you possibly attach a screenshot of the error?

    Just to clarify, you can create and deploy new releases, but not the existing (migrated) releases? Is this correct?

    If you go to Library -> Packages can you see the migrated packages?

  6. 6 Posted by Lyn Ko on 14 Dec, 2017 02:21 AM

    Lyn Ko's Avatar

    Hi there,

    Screenshot attached

    Thanks!
    Lyn

  7. Support Staff 7 Posted by Michael Richard... on 14 Dec, 2017 02:25 AM

    Michael Richardson's Avatar

    Thanks for the screenshot Lyn, that helps.

    Just to clarify, you can create and deploy new releases, but not the existing (migrated) releases? Is this correct?

    If you go to Library -> Packages can you see the migrated packages?

    If you look in the directory where you copied the migrated packages, do you now see the newly pushed packages along side them?

  8. 8 Posted by Lyn Ko on 17 Dec, 2017 10:36 PM

    Lyn Ko's Avatar

    Hi Michael,

    Sorry, I missed this email !

    Yes, we can create an deploy new releases.
    If I go to Library-> Packages, the older packages are not present.

    In the directory on the server, I can see the older packages (which I copied over manually as part of the migration process).

    I guess the important thing is that we can create new packages and releases and can deploy..

    The difference at the moment is that I am seeing a difference in the number of packages on the server in the PACKAGES folder compared to what’s on the UI.
    Somehow when I migrated the data into the database the older packages were not migrated…

    If we can find why, it would be good but not necessary.

    What is more important now is how to maintain the stored packages on the Repository, the Server and the Tentacle.
    My understanding is as follows

    - Packages not used in a release are deleted after a set period (we have set it up to 30 days)

    - Packages used in a release will not be deleted unless the release is deleted – via retention policy???

    For the two above points – does the physical package stored on the Octopus server get deleted as well as from the repository?
    I have set the retention policy for the releases to keep 1 release only on the Tentacle but the cleanup process (after the deployment step) does not seem to remove releases as defined by our retention policy

    Can you assist?

    Thanks
    Lyn

  9. Support Staff 9 Posted by Michael Richard... on 18 Dec, 2017 02:52 AM

    Michael Richardson's Avatar

    Lyn,

    If the packages are on the disk but not showing in Octopus, it is possible you just need to re-index the built-in package store. This can be done by clicking the button as shown in the attached image reindex-package.png.

    Regarding retention-policies, the server and tentacle retention-policies are applied differently. In this case, it's the server retention policies that you are interested in.

    On the server, a release will never be removed while it still appears on the Octopus dashboard. Once it no longer appears on the dashboard (because it has been replaced by newer releases), and it is eligible for retention according to the server retention policy, then it will be removed.

    Once a package is no longer referenced by any releases on the server, and is eligible for retention according to the package-retention policy, then the package will be removed from the Octopus Server (both from the database and the file on disk).

    The task which removes the releases and packages runs every 4 hours. You can see these tasks by viewing the Tasks page, and setting the advanced filter to show by type Apply Retention Policies (as shown in the attached retention-policy-tasks.png). The task logs of these tasks give a nice insight into what they are doing.

    I hope that helps,
    Michael

  10. 10 Posted by Lyn Ko on 19 Dec, 2017 05:12 AM

    Lyn Ko's Avatar

    Hi Michael,

    I kicked off the re-indexing task but it took a long time so I cancelled it.

    As for the retention policy of packages on the server, does that mean that a package must be deployed to all environments before it is removed (as it is still referenced in the dashboard for some environments)

    For the tentacles, I have set up our retention policy to keep 1 release only however I still see more than 1 release kept on the tentacle (the deployment log does show the Apply retention policies step which is successful.

    Is there any circumstance where the package is not deleted from the tentacle? Eg if I have configured the octopus tentacle directory to not be on the default directory?

  11. Support Staff 11 Posted by Michael Richard... on 19 Dec, 2017 11:46 AM

    Michael Richardson's Avatar

    Hi Lyn,

    I kicked off the re-indexing task but it took a long time so I cancelled it.

    The package indexing can take a significant time. It is however perfectly safe to let it run in the background. Perhaps re-start it before the holidays if you wish.

    As for the retention policy of packages on the server, does that mean that a package must be deployed to all environments before it is removed (as it is still referenced in the dashboard for some environments)

    No, it doesn't need to be deployed to all environments. It must be replaced (by another deployment) in all environments which it was deployed to. Our assumption is that if it is still the current release in an environment, it is likely that you may want to promote it to another environment, redeploy to the same environment, or access the packages it contained.

    For the tentacles, I have set up our retention policy to keep 1 release only however I still see more than 1 release kept on the tentacle (the deployment log does show the Apply retention policies step which is successful.

    How do you see the release kept on the Tentacle? Which files do you see?

    This page provides a wealth of information (possibly more than you ever wanted to know) on how retention policies behave on the Tentacles.

    If I may ask, is there a particular reason you are trying to achieve such limited retention policies?

  12. 12 Posted by Lyn Ko on 19 Dec, 2017 09:19 PM

    Lyn Ko's Avatar

    Thanks for info.
    I think that makes sense regarding the package retention on the server.
    We find that we do not need more than 1 releases to be kept on the tentacle and we have limited disk space on our servers.

    Now onto a seperate question regarding lifecylces.
    Is there any information regarding best practices for defining them?
    We have 7 environments and would only have one where the sdlc will need to apply. That is, only one environment that needs to be deployed to in order for the release to be promoted to production.

    All the other environments can be deployed to without a dependency on another environment

    Thanks
    Lyn

    Get Outlook for iOS<https://aka.ms/o0ukef>
    ________________________________

  13. Support Staff 13 Posted by Michael Richard... on 20 Dec, 2017 06:05 AM

    Michael Richardson's Avatar

    Sure, you can achieve that with a single Lifecycle.

    I have attached an example (sdlc-phases.png).

    Key points:

    • The first phase contains all your optional environments, and the phase is marked as optional.
    • The second phase contains only the environment required to progress to production (I have called it Staging), and the phase is marked as required.
    • The final phase contains your production environment.

    I believe that should specify the the desired behaviour.

    I hope that helps!

  14. 14 Posted by Lyn Ko on 21 Dec, 2017 12:09 AM

    Lyn Ko's Avatar

    Thanks Michael

    I tried setting up one life cycle where EnvF must be deployed to before anything can be promoted to EnvB for a project named RegulatoryReporting.
    My understanding was that if I tried to deploy to B, Octopus would prevent me from doing so unless it was already promoted to F.

    However, this didn’t happen. I created a new release and deployed to B and it worked.

    Am I missing something?

    Another thing I just tried to do was move the database to another server using the octopus.server.exe configure –storageConnectionString but it came back with an error saying it wasn’t a valid argument.

    When I ran octopus.server.exe help configure , the –storageConnectionString was not listed as an argument.

    What’s the deal here?

    Also the UI says that package reindexing starts at startup so theoretically, the packages should be reindexed already but I am still seeing anomalies in what is stored in the repository and what’s on the server.

    Thanks for your replies

  15. Support Staff 15 Posted by Michael Richard... on 21 Dec, 2017 06:03 AM

    Michael Richardson's Avatar

    Hi Lyn,

    Octopus will prevent deploying to an Environment if your Lifecycle doesn't allow it.

    Could you attach a screenshot of your Lifecycle? Are you sure your project is configured to use the Lifecycle (this can be changed on the Process tab of the Project)?

    We apologize, the Moving the Database document was incorrect. We changed the commands around a little in 4.0, and evidently missed updating that page. The correct command is:

    octopus.server.exe database --connectionString="value"
    

    I have now updated that document (though it may be cached for a little while).

    You are correct, the package-indexing will run at server startup if it is enabled. Is it possible the packages were copied in after it had commenced? If you search the view the Package Indexing tasks (I mentioned how in a previous message), can you see a successful run?

    Regards,
    Michael

  16. 16 Posted by Lyn Ko on 21 Dec, 2017 10:02 PM

    Lyn Ko's Avatar

    I did notice that in the release, the lifecycle wasn’t correct.
    I did update it in the process.

    I am now on leave but I will check once I get back in the new year.

    Also, Thanks for the update on the configure option!

    Have a merry Xmas!

    Get Outlook for iOS<https://aka.ms/o0ukef>
    ________________________________

  17. Support Staff 17 Posted by Michael Richard... on 22 Dec, 2017 05:59 AM

    Michael Richardson's Avatar

    You're most welcome and merry Christmas to you too Lyn.

  18. 18 Posted by Lyn Ko on 08 Jan, 2018 05:01 AM

    Lyn Ko's Avatar

    Hi Michael

    Happy New Year.

    I am back from leave and just checking our servers and I think I have some confusion in the package retention set up for our projects because no packages seem to be deleted from the server. (retention on the tentacles is working find)

    So for your reference, our environment is not really following an SDLC path.
    We have 4 non-production environments and 1 Prod and 1 DR environment.

    Our apps do not need to be deployed to all environments in order because every environment is used for a different project.
    Attached is the default lifecycle that we have set up which just contains all the environments which a project can be deployed to.

    Each stage is set to be deployed to manually. Retention is set to 2 releases (for server?) and 2 for tentacles .

    Attached is also the list of packages that have been retained on the server for the project RegulatoryReporting.

    The oldest package deployed in any of the listed environments on the dashboard is Dec 19th, however the older packages do not seem to be deleted.

    Can you advise?

    Thanks
    Lyn

    This email message and any accompanying attachments may contain information that is confidential and is subject to legal privilege. If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or attachments. If you have received this message in error, please notify the sender immediately and delete this message. Any views expressed in this message are those of the individual sender, except where the sender expressly, and with authority, states them to be the views of AMP. Before opening any attachments, please check them for viruses and defects.

  19. Support Staff 19 Posted by Michael Richard... on 08 Jan, 2018 06:40 AM

    Michael Richardson's Avatar

    Have you also configured the built-in feed retention?

    If so, how many days is it set to?

  20. 20 Posted by Lyn Ko on 09 Jan, 2018 01:34 AM

    Lyn Ko's Avatar

    Hi Michael,

    Yes, I had it set to 30!

    Have just updated it to 15 now…thanks a lot

    Lyn

  21. Support Staff 21 Posted by Michael Richard... on 09 Jan, 2018 05:11 AM

    Michael Richardson's Avatar

    You're welcome. I can certainly understand how this can be confusing. There are a number of places various retention-policies are configured.

    Hopefully that does the trick.

  22. 22 Posted by Lyn Ko on 09 Jan, 2018 09:45 PM

    Lyn Ko's Avatar

    Thanks Michael,

    That did the trick…all good now.

    Listen, I have other question regarding cloning of steps from project to project.
    Automating set up of library sets etc.

    All these are finicky to do via the gui…

    Can I direct these questions to you from this forum or shall I post on the octopus forum??

    Thanks!!
    Lyn

  23. Support Staff 23 Posted by Michael Richard... on 10 Jan, 2018 05:15 AM

    Michael Richardson's Avatar

    Oh great, I'm glad to hear that helped.

    For your additional questions I would suggest raising them as new threads on this forum, or alternatively asking in our community Slack if you prefer a more informal conversation.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac