Upgrade from 4.0.1 to 4.0.2

Upgrading instructions

Follow these instructions if you are upgrading from 4.0.1 to 4.0.2. The current installation 4.0.1 could have been installed using Helm (Scenario A) or the GitOps installer (Scenario B). Please follow the steps as per your present scenario.

WARNING: Before starting the upgrade, take a backup of all the databases, particularly the Postgres DB. The backup procedures may vary depending on your use of external DBs and Spinnaker configuration.

Scenario A

Use these instructions if:

  • You have a 4.0.1 ISD installed using the helm installer

  • Already have a "gitops-repo" for Spinnaker Configuration

  • Have values.yaml that was used for helm installation

Execute the following commands, replace "gitops-repo" with your repo

  • git clone https://github.com/.../gitops-repo

  • git clone https://github.com/OpsMx/standard-isd-gitops.git -b 4.0

  • cp -r standard-isd-gitops/upgrade gitops-repo

  • cd gitops-repo

  • Copy the existing "values.yaml" used for the previous installation into this folder. We will call it values-401.yaml.

  • diff values-401.yaml values-402.yaml and merge all of your changes into "values.yaml".

    Note: Most of the time, it suffices to replace images with version number 4.0.1 to 4.0.2.

  • Copy the updated values file as "values.yaml" (File name is mandatory).

  • Create gittoken secret. This token will be used to authenticate to the gitops-repo. 

    kubectl -n oes create secret generic gittoken --from-literal gittoken=PUT_YOUR_GITTOKEN_HERE

  • Create the secrets as mentioned above. Note that, these secrets only need to be created if they differ from the default. 

    kubectl -n oes create secret generic ldapconfigpassword --from-literal ldapconfigpassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic ldappassword --from-literal ldappassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic miniopassword --from-literal miniopassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic redispassword --from-literal redispassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic saporpassword --from-literal saporpassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic rabbitmqpassword --from-literal rabbitmqpassword=PUT_YOUR_SECRET_HERE
kubectl -n oes create secret generic keystorepassword --from-literal keystorepassword=PUT_YOUR_SECRET_HERE

Scenario B

Use these instructions if:

  • You have a 4.0.1 ISD installed using GitOps installer

  • Already have a gitops-repo for ISD (AP and Spinnaker) Configuration

Execute the following commands, replace "gitops-repo" with your repo

  • git clone https://github.com/.../gitops-repo

  • git clone https://github.com/OpsMx/standard-isd-gitops.git -b 4.0

  • cp -r standard-isd-gitops/upgrade gitops-repo

  • cd gitops-repo

  • Check that a "values.yaml" file exists in this directory (root of the gitops-repo).

Common Steps

Upgrade sequence: (4.0.1 to 4.0.2)

  1. Ensure that "default" account is configured to deploy to the ISD namespace (E.g. oes)

  2. If you have modified "sampleapp" or "opsmx-gitops" applications, take a backup of them using "syncToGit" pipeline opsmx-gitops application.

  3. Update the halyard version in config file i.e deploymentConfigurations.version 1.28.1

  4. If any custom settings are done for spinnaker, update those changes accordingly in gitops-repo/default/profiles and gitops-repo/default/service-settings

  5. The echo-local.yml file in gitops-repo/default/profiles need to be updated as mentioned in the following repo https://github.com/OpsMx/standard-gitops-repo/blob/v4.0/default/profiles/echo-local.yml

  6. If custom settings are not done for spinnaker, execute the below command cp -r standard-isd-gitops.git/default gitops-repo/

  7. cd upgrade

  8. Update upgrade-inputcm.yaml:

    • URL, username, and gitemail must be updated. If you have install/inputcm.yaml from previous installation, copy-paste those lines here.

    • If ISD Namespace is different from "opsmx-isd": Update namespace (default is opsmx-isd) to the namespace where ISD is installed.

  9. If ISD Namespace is different from "opsmx-isd": Edit serviceaccount.yaml and edit "namespace:" to update it to the ISD namespace (E.g: oes).

  10. Push changes to git: git add -A; git commit -m"Upgrade related changes";git push

  11. kubectl -n opsmx-isd apply -f upgrade-inputcm.yaml

  12. kubectl -n opsmx-isd apply -f serviceaccount.yaml # Edit namespace if changed from the default "opsmx-isd".

  13. Scale down audit service by executing the following command: kubectl -n opsmx-isd scale deploy oes-audit-service --replicas=0

  14. Upgrade DB: This can be executed as a Kubernetes job

    kubectl -n opsmx-isd apply -f ISD-DB-Migrate-job.yaml

  15. Scale up audit service by executing the following command: kubectl -n opsmx-isd scale deploy oes-audit-service --replicas=1

  16. kubectl -n opsmx-isd replace --force -f ISD-Generate-yamls-job.yaml [ Wait for isd-generate-yamls-* pod to complete]

  17. Compare and merge branch: This job should have created a branch on the gitops-repo with the helmchart version number specified in upgrade-inputcm.yaml. Raise a PR and check what changes are being made. Once satisfied, merge the PR.

  18. kubectl -n opsmx-isd replace --force -f ISD-Apply-yamls-job.yaml Wait for isd-yaml-update-* pod to complete, and all pods to stabilize 17 isd-spinnaker-halyard-0 pod should restart automatically.

    If not, execute the following command: kubectl -n opsmx-isd delete po isd-spinnaker-halyard-0

  19. Restart all pods:

    • kubectl -n opsmx-isd scale deploy -l app=oes --replicas=0 Wait for few a minutes.

    • kubectl -n opsmx-isd scale deploy -l app=oes --replicas=1 Wait until every pod comes to the ready state.

  20. Go to ISD UI and check that the version number has changed in the bottom-left corner.

  21. Wait for about five minutes for autoconfiguration to take place.

  22. If required: Connect Spinnaker again and Configure pipeline promotion again. To do this, in the ISD UI follow the below steps :

    • Click Setup

    • Click Spinnaker tab at the top. Check if "External Accounts" and "Pipeline-promotion" columns show "yes".

    • If any of them is "no": Click "edit" on the three dots on the far right. Check the values already updated in, make changes if required, and click "update".

    • Restart the halyard pod by clicking "Sync Accounts to Spinnaker" in the Cloud Accounts tab or simply delete the halyard pod.

If an upgrade fails

As we have a gitops installer, recovering from a failed install/upgrade is easy. In summary, simply delete all objects or re-apply. Follow the steps below to recover.

As a first step. Please try the "Troubleshooting" section if you are facing any issues during the Installation.

Reinstall ISD

Make changes to upgrade-inputcm and/or values.yaml as required. Ensure that the changes are pushed to git.

  1. kubectl -n oes delete sts isd-spinnaker-halyard

  2. kubectl -n oes delete deploy --all

  3. kubectl -n oes delete svc --all

  4. kubectl -n oes replace --force -f ISD-Apply-yamls-job.yaml

  5. Wait for all the pods to come up

PreviousChanges from 3.12 to 4.0.3 branchNextUpgrade from 3.12.x to 4.0.2

Last updated