A kubernetes cluster with 32GB RAM and 8 CPU free capacity with admin permissions.
It is assumed that an ingress controller is available on the cluster (ingress-nginx is the default)
It is also assumed that cert-manager (with LetsEncrypt) is available in the cluster in case automatic TLS certificates generation is required.
It is assumed that the cluster provides dynamic PV provisioning. If not please see instructions here to create the PVs in advance.
Internet connection, both to the laptop/machine from where the commands are executed and for the kubernetes cluster.
Access to DNS to route web-URLs to services in a kubernetes cluster.
Helm uses “YAML” format. Some familiarity with YAML is assumed. For example, the documentation refers to “forwarder.enabled” to mean sub-element” enabled” under the “forwarder” block.
Some basic knowledge of kubernetes is required for debugging.
The installation requires preparing up to 3 git-repos for providing storage for gitOps operations.
GitOps is an operational paradigm that says that the target environment is in sync with a git repo. While we do not have full GitOps, the model being followed can be extended to GitOps in future. Fundamentally, all the configuration information for Spinnaker is stored in a directory “.hal” inside the halyard-pod. The “normal” spinnaker installation requires a persistent store (PV) to be mounted at this mount point (/home/spinnaker). In gitOps halyard, we have an init container in the statefulset that clones a git-repo (or Vault or S3) and processes the files before placing them in .hal directory. There is no persistent storage, thereby reducing the associated security risks. Any change required can be done in the git-repo. Once the halyard pod is restarted, the changes become effective.
Follow the steps given below to setup a GitRepo:
Choose a source git-repository based on your installation type, by default it is standard-gitops-repo. The other options are:
X509-gitops-repo: If you want to enable x509 communication enabled.
Self-signed-gitops-repo: This is if you need self-signed CA certificates to be generated.
X509-self-signed-gitops-repo: x509 with self-signed.
Git clone source-repo https://github.com/OpsMx/standard-gitops-repo.git
Create a repo in github for this installation e.g. gitops-k8s121, select “Add readme”
In a different directory in your local: git clone https://github.com/<org-name/username>/<repo-name>.git
Copy the following files from source-repo-clone into this new repo-clone: config, halyard.yml and default directory to the root of the repo.
git add -A; git commit -m”my gitops repo”; git push
Create git token that can read this repo.
Dynamic Accounts Repo
We can use the same gitOps Repo for storing dynamic accounts as well. Otherwise, create an empty repo in git. Ensure that the default branch is called “master” (not “main”). If required, simply rename the “main” branch to “master”.
You can use the same gitOps Repo for storing pipelines as well. Otherwise, create an empty repo in git.
Installing OES involves preparing a
values.yaml file and using helm install command. This is followed by configuring OES to connect to Spinnaker.
values.yaml into a local file (e.g. values-k8s121.yaml) from SAMPLES directory in “standard-gitops-repo”.
imageCredentials : This creates a docker secret for pulling OES images. Please update username and password for quay.io repo in plain text. Alternatively, these can be given on the command line when doing helm-install.
global.certManager or self-signed: If you have cert-manager in your cluster, please set it to true. Else, setting true for self-signed will create self-signed TLS certificates.
Edit the following:
For spinDeck, spinGate, oesUI and oesGate : set the protocol (http/https) and the url you expect to use. Please update the DNS records (e.g. godaddy) with the urls/domain-name so as to reach the services. The default helm package install creates ingress-files for nginx Ingress along with TLS certificates using cert-manager+Letsencrypt combination. One would need to update the DNS to reach the ingress controller in the cluster (or whichever way you want to reach the services). All the http routing requirements are documented here.
OPTIONAL: Update the forwarder.externaName of the controller that the agent will contact from a remote cluster. In case you do NOT want to deploy to a remote cluster, please set forwarder.enabled to false.
gitOps Configuration: Locate the block “gitopsHalyard:” towards the end of the file and do the following:
repo.repository: fill in organization (e.g. opsmx), dynamicAccRepository (created above),username and token used to access the git repository.
repo.pipelinePromotion.enabled: true and provide organization, repo, user and token. Note that the pipeline gitOps runs as a kubernetes job in the default namespace using default account. The default account should have permissions to deploy to the default namespace. This is particularly important when creating accounts from OES as the 1st kubernetes account that is created masks the default account.
spinCli.gate.endpoint: Please update this to make the spinGate URL in the https://<spinGate> format.
Once done, install OES in “oes” namespace using helm command. Follow the steps below:
Please ensure that kubectl points to the correct cluster. If needed, please export the path to the kubeconfig as follows:
Export KUBE_CONFIG=<path to the kubeconfig file>
helm repo add opsmx https://helmcharts.opsmx.com/
helm repo update
kubectl create namespace oes
helm install oes opsmx/oes -f <your values>.yaml -n oes --timeout 30m
The installation takes about 15-20 minutes depending on the cluster performance.
If using vault:
You need to initialize vault with seed data. Execute the following code:
export VAULT_URI=<vault uri e.g https://vault.example.com>export VAULT_TOKEN=<token>helm install oes <repo> -f values-k8s121.yaml --timeout 30m -n oes --set secretStore=vault --set VAULT_TOKEN=$VAULT_TOKEN --set VAULT_URI=$VAULT_URI
a. Configure Spinnaker connection in OES. Details in a section below. Summary is here.
b. kubectl get ing # - This should show your URLs. Navigate to oes-ui url, use admin and opsmxadmin123 as username and password. If using non-default LDAP or authentication parameters, please substitute the username and password as appropriate.
c. Go to “Setup” in OES-UI, click on Integrations tab.
d. Configure “GITHUB” account(s) to point to our gitops and dynamic account credentials.
NOTE: The “External Account” information should match the information provided in gitOps configuration
e. For default LDAP configuration, configure “Spinnaker” with spin-gate url and a token as given below:
YWRtaW46b3BzbXhhZG1pbjEyMw== This is “user:password” in base64 encoded form, generated as:
echo -ne "admin:opsmxadmin123" | base64 -w0
f. OPTIONAL: Controller configuration: Controller is installed by default and is used for deploying to a remote cluster that is not accessible via inbound network connection. However, routing the gRPC traffic to “agent-grpc” service is the user’s responsibility. This can be done by one of the two methods:
1) Locate the LoadBalancer IP address of the agent-grpc service using:
kubectl get svc | grep agent-grpc
Configure DNS server to point to this address for the host name mentioned as:
forwarder.externaName in values.yaml
2) Alternatively, one can set-up the ingress controller to point to agent-grpc service. Instructions for this can be found here. This configuration requires additional information related to TCP routing via ingress (as against HTTP routing).
g. Create a kubernetes account named “default”. If should contain a minimum of 2 namespaces: default and oes (or the install). Go to
setup->cloud accounts->new account and upload the kubeconfig file with permissions for these namespaces.
h. In certain cases, when an integration is added, e.g. jenkins, Spinnaker needs to be triggered to reload the configuration. When this is required, the “Sync Accounts with Spinnaker” button in the
Setup->Integration screen will get enabled and should be clicked.
This works by triggering a pipeline in Spinnaker. The “restart-halyard” pipeline in “sampleapp” in spinnaker can be used for this purpose. For this to work, note that:
“default” kubernetes account should have been created that allows deploying to the “oes” namespace (or whichever namespace Spinnaker is installed in) and “default” namespace.
“restart-halyard” pipeline should be configured for “default” account and “oes” namespace. Alternatively, we can use any account that is able to delete the spinnaker halyard pod. The application “Opsmx-GitOps” already has this pipeline and should have been created automatically.
Alternatively, you can manually delete the Spinnaker halyard pod to force it to reload the configuration from git.
At this point OES should show “connected” status. You can check using the following:
=> Setup->Applications->Sync Spinnaker Applications=> Security->Audit Trail=> Dashboard->Delivery Dashboard
You can deploy to a cluster and access resources that are not accessible to the installed instance by creating an agent. The steps for this are as follows:
In OES UI, Navigate to setup->Agents.
New Agent->Fill-in the details and save.
Download the yaml, using the link that gets activated few seconds after “save”.
On TARGET Environment (remote Cluster)
Create an agent-service-config file using the samples available in the SAMPLES directory in standard-gitops-repo. Ensure that configMap name is correct and is suffixed with agent name.
Apply the downloaded agent-yaml in the remote cluster’s default namespace:
kubectl apply -f <downloaded file>
Note: In case we want to use a different namespace, please locate and change “namespace: default” in the downloaded file for the service-account config.
3. Wait for the agent pods to come-up and be in “ready” state.
On OES UI
Once the agent pods come-up, refresh the OES agent screen to check that the agent shows a “green” dot status
Go to Integrations and Cloud Accounts pages to see the accounts configured in the agent-services configMap listed along with local accounts.
One needs to “accept” and “save” each of the accounts in Integrations to make them active.
On the “Integrations” screen, if the “Sync Accounts to Spinnaker” link on the top-right is active, click it. If possible, ensure that oes-spinnaker-halyard-0 is restarted. Else manually delete the pod to trigger a restart.
NOTE: “restart-halyard” pipeline in sampleapp in Spinnaker is configured correctly, halyard pod will restart automatically.
5. Wait for a few minutes, refresh the Spinnaker UI to see the agent-accounts available.
App-sync keeps going round-and-round: Restart the sapor pod by terminating the running instance.
Spinnaker shows “Not Connected: Check credentials once again and restart the sapor pod by terminating the running instance. If credentials are incorrect, please correct, save and restart the sapor pod, if required.
oes-spinnaker-halyard-0 pod goes into crash loop: Check the git-repo values in values.yaml. If not clear, look at the logs of the create-halyard-local container has follows:
kubectl logs oes-spinnaker-halyard-0 -c create-halyard-local
Agent pods remain in “creating container” state: Ensure that service-configuration is applied. Sample configurations are available here
Agent unable to connect to the controller: “Context deadline exceeded” means that agent is unable to connect to the controller. Check the DNS-name/ingress routing and ensure that the agent is able to reach the agent-grpc service. Note that the agent talks to the controller using gRPC protocol. Hence TCP routing is required (not HTTP routing). If using an ingress-controller, ensure that TCP passthrough/routing is enabled.
Unable to deploy to k8s agent account from Spinnaker: This is usually not a problem related to the agent. Please check if the clouddriver pods are running correctly.
Unable to list jobs in Jenkins stage in Spinnaker: This is usually due to an issue with URL, username or password being incorrect. Please check and update. Once done, restart the agent-pods (kubectl scale deploy <deployment name> --replicas=0; followed by the same command with replicas=1)
Sample application not present: This can happen if the spin-cli process during installation fails due to some reason, typically if the gate-URL is incorrect in values.yaml. Please follow the instructions here: https://github.com/OpsMx/sample-pipelines“Sync Accounts with Spinnaker” button does not work: This makes a web-hook call to a Spinnaker pipeline (restartHalyard, in Opsmx-GitOps application). While there are many reasons for this to not work, most common is the account and namespace for this pipeline needs to be correct. Workaround is to manually delete the halyard pod.
In case we decide to use the Vault backend for storing secrets, it needs to be initialized with some seed data that allows us to get started and login to the UI.
Login to Vault
In case of k8s installation, exec into the vault container, login and execute the below commands.
vault secrets enable -path=secret kv-v2
Create Key-Value Pair
Common Data (same username and password for all DBs, they are different please put them under the appropriate service)
vault kv put secret/application spring.datasource.username=postgresvault kv patch secret/application spring.datasource.password=networks123
For LDAP :
vault kv patch secret/application spring.ldap.managerPassword=opsmxadmin123vault kv patch secret/application spring.ldap.url=ldap://oes-openldap:389vault kv patch secret/application ldap.url=ldap://oes-openldap:389vault kv patch secret/application ldap.managerPassword=opsmxadmin123
For SAML :
For Outh2 :
vault kv put secret/oes-platform redis.connection=redis://:[email protected]:6379vault kv patch secret/oes-platform spring.datasource.url=jdbc:postgresql://oes-db:5432/platformdb
vault kv put secret/oes-sapor spring.datasource.url=jdbc:postgresql://oes-db:5432/oesdb
vault kv put secret/oes-visibility spring.datasource.url=jdbc:postgresql://oes-db:5432/visibilitydb
vault kv put secret/oes-autopilot secret.datasource.username=postgresvault kv patch secret/oes-autopilot secret.datasource.password=networks123vault kv patch secret/oes-autopilot secret.datasource.url=jdbc:postgresql://oes-db:5432/opsmxvault kv patch secret/oes-autopilot secret.platform.url=http://oes-platform:8095vault kv patch secret/oes-autopilot secret.ds.protocol=http://vault kv patch secret/oes-autopilot secret.ds.url=localhost:5005
Once Spinnaker and OES are installed successfully, we need to add Spinnaker in OES in case the automatic configuration fails or is disabled, passwords have been updated or x509 configuration is desired.
Navigate to Setup → Spinnaker Setup → Add Spinnaker
Spinnaker Name : The name of Spinnaker Instance.
Spinnaker URL : The Gate URL of the Spinnaker instance with which OES will communicate to.
Authentication Type: Authentication Mechanism with which OES communicates to Spinnaker.
LDAP : For Basic Authentication (username with password)
X509 : For 2 Factor Authentication systems.
Using the LDAP Mechanism:
In Order to fetch the application using the LDAP, user has to provide the username and password of the user (who has the access to all the pipelines) in the encrypted base64 format using this command:
echo -ne “username:password” | base 64 -w0
Using the x509 Mechanism:
For the x509 method of authentication, we need to configure x509 method of Authentication in Spinnaker.
To Enable x509 method of Authentication for Spinnaker, please refer to the steps in
Spinnaker-gate listens to x509 communication on a different port (8085). This is in addition to the normal port (8084) used for web-login
Spinnaker-gate will need to be configured with SSL. ISTIO needs to be configured to allow SSL communication between OES and Spinnaker-gate’s x509 port and between web-browser and Spinnaker-gate. TLS termination should happen at Spinnaker-gate (and not at ISTIO/ingress).
Once the Spinnaker is Enabled with x509 type of authentication, in order for OES to communicate with Spinnaker using x509, we need a client p12 key. Using the client.p12 key and password, we can configure OES to communicate with Spinnaker on the x509 port.
If we did not have this name added as the SAN, then we need to have an alternate External URL with the Proper SAN Name added to the Gate Certificate, and configure this URL in the OES.
Spinnaker gate SSL and JKS creation instructions
Client-P12 creation instructions Assuming that we have client tls certificate and key we can generate a p12 using the below openssl command :
openssl pkcs12 -export -clcerts -in x509lb.crt -inkey x509lb.key -out x509lb.p12 -name gate509lb -password pass:changeit