# Upgrade from 3.12.x to 4.0.3 ## Upgrading instructions Follow these instructions if you are upgrading from **3.12.x to 4.0.3**. The current installation 3.12.x could have been installed using **Helm** ([**Scenario A**](#scenario-a)) or the **GitOps** installer ([**Scenario B**](#scenario-b)). Please follow the steps as per your present scenario. {% hint style="warning" %} **WARNING**: Before starting the upgrade, take a backup of all the databases, particularly the Postgres DB. The backup procedures may vary depending on your use of external DBs and Spinnaker configuration. {% endhint %} ### **Scenario A** Use these instructions if: * You have a 3.12.x ISD installed using the **helm installer** * Already have a "**gitops-repo**" for Spinnaker Configuration * Have **values.yaml** that was used for helm installation Execute the following commands, replace "**gitops-repo**" with your repo * `git clone` * `git clone https://github.com/OpsMx/standard-isd-gitops.git -b 4.0.3` * `cp -r standard-isd-gitops/upgrade gitops-repo` * `cd gitops-repo` * Copy the existing "**values.yaml**" used for the previous installation into this folder. We will call it **values-312.yaml.** * diff **values-312.yaml** **values-403.yaml** and merge all of your changes into "**values.yaml**".

Note: Most of the time, it suffices to replace images with version number 3.12.x to 4.0.3.

* Copy the updated values file as "values.yaml" (File name is mandatory). * Create gittoken secret. This token will be used to authenticate to the gitops-repo.
kubectl -n opsmx-isd create secret generic gittoken --from-literal gittoken=PUT_YOUR_GITTOKEN_HERE
* Create the secrets as mentioned above. Note that, these secrets only need to be created if they differ from the default.
kubectl -n opsmx-isd create secret generic ldapconfigpassword --from-literal ldapconfigpassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic ldappassword --from-literal ldappassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic miniopassword --from-literal miniopassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic redispassword --from-literal redispassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic saporpassword --from-literal saporpassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic rabbitmqpassword --from-literal rabbitmqpassword=PUT_YOUR_SECRET_HERE
  kubectl -n opsmx-isd create secret generic keystorepassword --from-literal keystorepassword=PUT_YOUR_SECRET_HERE
### **Scenario B** Use these instructions if: * You have a 3.12.x ISD installed using **GitOps installer** * Already have a **gitops-repo** for ISD (AP and Spinnaker) Configuration Execute the following commands, replace "**gitops-repo**" with your repo * `git clone` * `git clone https://github.com/OpsMx/standard-isd-gitops.git -b 4.0.3` * `cp -r standard-isd-gitops/upgrade gitops-repo` * `cd gitops-repo` * Check that a "**values.yaml**" file exists in this directory (root of the gitops-repo). ### Common Steps Upgrade sequence: (3.12.x to 4.0.3) 1. Ensure that "**default**" account is configured to deploy to the ISD namespace (E.g. opsmx-isd) 2. If you have modified "**sampleapp**" or "**opsmx-gitops**" applications, take a backup of them using "**syncToGit**" pipeline opsmx-gitops application. 3. Copy the bom from standard-isd-gitops.git to the gitops-repo.\ `cp -r standard-isd-gitops/bom gitops-repo/` 4. If there are any custom settings done for spinnaker please update those changes accordingly in ***gitops-repo/default/profiles***. 5. Please refer to the below link to check the differences from 3.12 to 4.0.3 of git-repo and updated it accordingly.\ 6. `cd upgrade` 7. Update **upgrade-inputcm.yaml**: * URL, username, and gitemail must be updated. If you have **install/inputcm.yaml** from previous installation, copy-paste those lines here. * **If ISD Namespace is different from** "**opsmx-isd**": Update namespace (default is opsmx-isd) to the namespace where ISD is installed. * If you are external Mysql for Spinnaker, update the spinnakerStorage value to "**sql**". 8. **If ISD Namespace is different from** "**opsmx-isd**": Edit **serviceaccount.yaml** and edit "**namespace**:" to update it to the ISD namespace (E.g: opsmx-isd). 9. **Push changes to git**: `git add -A; git commit -m"Upgrade related changes";git push` 10. `kubectl -n opsmx-isd apply -f upgrade-inputcm.yaml` 11. `kubectl -n opsmx-isd apply -f serviceaccount.yaml` # Edit namespace if changed from the default "**opsmx-isd**". 12. `kubectl -n opsmx-isd create secret generic mysqlcredentials --from-literal host=PUT_YOUR_MYSQL_HOST_NAME --from-literal username=PUT_YOUR_MYSQL_USER_NAME --from-literal password=PUT_YOUR_MYSQL_PASSWORD` (**NOTE: Do not change the default values**) 13. **DB Upgrade - Schema update**: This can be be executed as a kubenetes job. * `kubectl -n opsmx-isd apply -f ISD-Pre-Helm-job.yaml` # Edit namespace if changed from the default "opsmx-isd". * Once the above command is executed a new pod will be created. Please check the pod logs to verify if the Schema is updated or not. * If message indicating **success** - "Successfully updated databases" * If message indicating **failure** - "Exception occurred while updating databases" `kubectl -n opsmx-isd logs isd-pre-helm-migrate-xxx` #Replacing the name of the pod name correctly. * Use below command to scale down the services: `kubectl -n opsmx-isd scale deploy oes-audit-client oes-audit-service oes-autopilot oes-dashboard oes-datascience oes-platform oes-sapor oes-visibility --replicas=0` 14. `kubectl -n opsmx-isd replace --force -f ISD-Generate-yamls-job.yaml` \[ Wait for isd-generate-yamls-\* pod to complete ]. * Once the pod is completed please check the pod logs to verify manifest files are updated in GIt or not.\ `kubectl -n opsmx-isd logs isd-generate-yamls-xxx -c git-clone` #Replacing the name of the pod name correctly, check if your gitops-repo is cloned correctly 15. **Compare and merge branch**: This job should have created a branch on the **gitops-repo** with the helmchart version number specified in **upgrade-inputcm.yaml**.\ Raise a PR and check what changes are being made. Once satisfied, merge the PR. 16. `kubectl -n opsmx-isd replace --force -f ISD-Apply-yamls-job.yaml` Wait for isd-yaml-update-\* pod to complete. * Once pod completed, please check the pod logs to verify whether manifest files are updated in Git or not.\ \ `kubectl -n opsmx-isd logs isd-apply-yamls-xxx -c git-clone` #Replacing the name of the pod name correctly, check if your gitops-repo is cloned correctly `kubectl -n opsmx-isd logs isd-apply-yamls-xxx -c script` #Replacing the name of the pod name correctly, check the log of the script that pushes the yamls and applies them. 17. isd-spinnaker-halyard-0 pod should restart automatically. If not, execute this: `kubectl -n opsmx-isd delete po isd-spinnaker-halyard-0` 18. **Restart all pods**: * `kubectl -n opsmx-isd scale deploy -l app=oes --replicas=0`Wait for few a minutes. * `kubectl -n opsmx-isd scale deploy -l app=oes --replicas=1`Wait until every pod comes to the ready state. 19. Go to ISD UI and check that the version number has changed in the the top-right corner. 20. Wait for about five minutes for autoconfiguration to take place. 21. If required: Connect Spinnaker again and Configure pipeline promotion again. To do this, in the ISD UI follow the below steps : * Click **Setup** * Click **Spinnaker tab** at the top. Check if "**External Accounts**" and "Pipeline-promotion" columns show "**yes**". * If any of them is "**no**": Click "**edit**" on the three dots on the far right. Check the values already updated in, make changes if required, and click "**update**". * Restart the halyard pod by clicking "**Sync CD Accounts** " on the Cloud Targets page or simply delete the halyard pod. 22. **DB Upgrade - Data update**: This can be be executed as a kubenetes job. * `kubectl -n opsmx-isd apply -f ISD-Post-Helm-job.yaml` # Edit namespace if changed from the default "opsmx-isd". * Once the above command is executed a new pod will be created. Please check the pod logs to verify if the Schema is updated or not. * If message indicating **success** - "Successfully updated databases" * If message indicating **failure** - "Exception occurred while updating databases" `kubectl -n opsmx-isd logs isd-post-helm-migrate-xxx` #Replacing the name of the pod name correctly. ### **If an upgrade fails** As we have a gitops installer, recovering from a failed install/upgrade is easy. In summary, simply delete all objects or re-apply. Follow the steps below to recover. As a first step. Please try the "[**Troubleshooting**](https://docs.opsmx.com/additional-resources/troubleshooting/troubleshooting-isd-gitops-installation-issues)" section if you are facing any issues during the Installation. #### **Reinstall ISD** Make changes to **upgrade-inputcm** and/or **values.yaml** as required. Ensure that the changes are pushed to git. 1. `kubectl -n oes delete sts isd-spinnaker-halyard` 2. `kubectl -n oes delete deploy --all` 3. `kubectl -n oes delete svc --all` 4. `kubectl -n oes replace --force -f ISD-Apply-yamls-job.yaml` 5. Wait for all the pods to come up ### **Rollback from v4.0.3 to v3.12.x** You can rollback from **v4.0.3 to v3.12.x** by following the instructions provided [**here**](https://docs.opsmx.com/additional-resources/rollback-instructions).