OES Installation

System Requirements

  • A kubernetes cluster with 32GB RAM and 8 CPU free capacity with admin permissions.

    • It is assumed that an ingress controller is available on the cluster (ingress-nginx is the default)

    • It is also assumed that cert-manager (with LetsEncrypt) is available in the cluster in case automatic TLS certificates generation is required.

    • It is assumed that the cluster provides dynamic PV provisioning. If not please see instructions here to create the PVs in advance.

    • In case ingress/cert-manager is not available, please follow the instructions here to install nginx and cert-manager packages in your cluster. If using ISTIO, please follow the instructions here.

  • Internet connection, both to the laptop/machine from where the commands are executed and for the kubernetes cluster.

  • Access to DNS to route web-URLs to services in a kubernetes cluster.

  • Helm uses “YAML” format. Some familiarity with YAML is assumed. For example, the documentation refers to “forwarder.enabled” to mean sub-element” enabled” under the “forwarder” block.

  • Some basic knowledge of kubernetes is required for debugging.

Prepare Git Repo

The installation requires preparing up to 3 git-repos for providing storage for gitOps operations.

GitOps Repo

GitOps is an operational paradigm that says that the target environment is in sync with a git repo. While we do not have full GitOps, the model being followed can be extended to GitOps in future. Fundamentally, all the configuration information for Spinnaker is stored in a directory “.hal” inside the halyard-pod. The “normal” spinnaker installation requires a persistent store (PV) to be mounted at this mount point (/home/spinnaker). In gitOps halyard, we have an init container in the statefulset that clones a git-repo (or Vault or S3) and processes the files before placing them in .hal directory. There is no persistent storage, thereby reducing the associated security risks. Any change required can be done in the git-repo. Once the halyard pod is restarted, the changes become effective.

Follow the steps given below to setup a GitRepo:

  • Choose a source git-repository based on your installation type, by default it is standard-gitops-repo. The other options are:

    • X509-gitops-repo: If you want to enable x509 communication enabled.

    • Self-signed-gitops-repo: This is if you need self-signed CA certificates to be generated.

    • X509-self-signed-gitops-repo: x509 with self-signed.

NOTE: SAMPLES directory contains sample files used later in the documentation and should not be copied further.

  • Create a repo in github for this installation e.g. gitops-k8s121, select “Add readme”

    In a different directory in your local: git clone https://github.com/<org-name/username>/<repo-name>.git

  • Copy the following files from source-repo-clone into this new repo-clone: config, halyard.yml and default directory to the root of the repo.

    git add -A; git commit -m”my gitops repo”; git push

  • Create git token that can read this repo.

NOTE: All contents need to be in the “root” of the repo.

Dynamic Accounts Repo

We can use the same gitOps Repo for storing dynamic accounts as well. Otherwise, create an empty repo in git. Ensure that the default branch is called “master” (not “main”). If required, simply rename the “main” branch to “master”.

Pipelines Repo

You can use the same gitOps Repo for storing pipelines as well. Otherwise, create an empty repo in git.

Note: We can use the same repo for all three items.

Install OES

Installing OES involves preparing a values.yaml file and using helm install command. This is followed by configuring OES to connect to Spinnaker.

Prepare values.yaml

  • Copy values.yaml into a local file (e.g. values-k8s121.yaml) from SAMPLES directory in “standard-gitops-repo”.

  • imageCredentials : This creates a docker secret for pulling OES images. Please update username and password for quay.io repo in plain text. Alternatively, these can be given on the command line when doing helm-install.

  • global.certManager or self-signed: If you have cert-manager in your cluster, please set it to true. Else, setting true for self-signed will create self-signed TLS certificates.

  • Edit the following:

    • For spinDeck, spinGate, oesUI and oesGate : set the protocol (http/https) and the url you expect to use. Please update the DNS records (e.g. godaddy) with the urls/domain-name so as to reach the services. The default helm package install creates ingress-files for nginx Ingress along with TLS certificates using cert-manager+Letsencrypt combination. One would need to update the DNS to reach the ingress controller in the cluster (or whichever way you want to reach the services). All the http routing requirements are documented here.

    • OPTIONAL: Update the forwarder.externaName of the controller that the agent will contact from a remote cluster. In case you do NOT want to deploy to a remote cluster, please set forwarder.enabled to false.

  • gitOps Configuration: Locate the block “gitopsHalyard:” towards the end of the file and do the following:

    • enabled: true

    • repo.repository: fill in organization (e.g. opsmx), dynamicAccRepository (created above),username and token used to access the git repository.

    • repo.pipelinePromotion.enabled: true and provide organization, repo, user and token. Note that the pipeline gitOps runs as a kubernetes job in the default namespace using default account. The default account should have permissions to deploy to the default namespace. This is particularly important when creating accounts from OES as the 1st kubernetes account that is created masks the default account.

    • spinCli.gate.endpoint: Please update this to make the spinGate URL in the https://<spinGate> format.

NOTE: This information should match the information updated in OES-UI for “External Account” repo

NOTE: Instructions for obtaining a git token are here.

Helm install

Once done, install OES in “oes” namespace using helm command. Follow the steps below:

  • Please ensure that kubectl points to the correct cluster. If needed, please export the path to the kubeconfig as follows:

Export KUBE_CONFIG=<path to the kubeconfig file>

  • helm repo add opsmx https://helmcharts.opsmx.com/

  • helm repo update

  • kubectl create namespace oes

  • helm install oes opsmx/oes -f <your values>.yaml -n oes --timeout 30m

The installation takes about 15-20 minutes depending on the cluster performance.

If using vault:

You need to initialize vault with seed data. Execute the following code:

export VAULT_URI=<vault uri e.g https://vault.example.com>
export VAULT_TOKEN=<token>
helm install oes <repo> -f values-k8s121.yaml --timeout 30m -n oes --set secretStore=vault --set VAULT_TOKEN=$VAULT_TOKEN --set VAULT_URI=$VAULT_URI

Note: Here the first “oes” is the release name, -n oes is the namespace to install to. We recommend using “oes” release name and “oes” namespace.

Post installation configuration

Note: Steps (a) through (e) can be automatically executed by setting --set oesAutoConfiguration=true option (set by default) during the helm installation command.

a. Configure Spinnaker connection in OES. Details in a section below. Summary is here.

b. kubectl get ing # - This should show your URLs. Navigate to oes-ui url, use admin and opsmxadmin123 as username and password. If using non-default LDAP or authentication parameters, please substitute the username and password as appropriate.

c. Go to “Setup” in OES-UI, click on Integrations tab.

d. Configure “GITHUB” account(s) to point to our gitops and dynamic account credentials.

NOTE: The “External Account” information should match the information provided in gitOps configuration values.yaml.

e. For default LDAP configuration, configure “Spinnaker” with spin-gate url and a token as given below:

YWRtaW46b3BzbXhhZG1pbjEyMw== This is “user:password” in base64 encoded form, generated as:

echo -ne "admin:opsmxadmin123" | base64 -w0

f. OPTIONAL: Controller configuration: Controller is installed by default and is used for deploying to a remote cluster that is not accessible via inbound network connection. However, routing the gRPC traffic to “agent-grpc” service is the user’s responsibility. This can be done by one of the two methods:

1) Locate the LoadBalancer IP address of the agent-grpc service using:

kubectl get svc | grep agent-grpc

Configure DNS server to point to this address for the host name mentioned as:

forwarder.externaName in values.yaml

2) Alternatively, one can set-up the ingress controller to point to agent-grpc service. Instructions for this can be found here. This configuration requires additional information related to TCP routing via ingress (as against HTTP routing).

g. Create a kubernetes account named “default”. If should contain a minimum of 2 namespaces: default and oes (or the install). Go to setup->cloud accounts->new account and upload the kubeconfig file with permissions for these namespaces.

h. In certain cases, when an integration is added, e.g. jenkins, Spinnaker needs to be triggered to reload the configuration. When this is required, the “Sync Accounts with Spinnaker” button in the Setup->Integration screen will get enabled and should be clicked.

This works by triggering a pipeline in Spinnaker. The “restart-halyard” pipeline in “sampleapp” in spinnaker can be used for this purpose. For this to work, note that:

  1. “default” kubernetes account should have been created that allows deploying to the “oes” namespace (or whichever namespace Spinnaker is installed in) and “default” namespace.

  2. “restart-halyard” pipeline should be configured for “default” account and “oes” namespace. Alternatively, we can use any account that is able to delete the spinnaker halyard pod. The application “Opsmx-GitOps” already has this pipeline and should have been created automatically.

  3. Alternatively, you can manually delete the Spinnaker halyard pod to force it to reload the configuration from git.

At this point OES should show “connected” status. You can check using the following:

=> Setup->Applications->Sync Spinnaker Applications
=> Security->Audit Trail
=> Dashboard->Delivery Dashboard

Deploying to a remote cluster

You can deploy to a cluster and access resources that are not accessible to the installed instance by creating an agent. The steps for this are as follows:

  1. In OES UI, Navigate to setup->Agents.

  2. New Agent->Fill-in the details and save.

  3. Download the yaml, using the link that gets activated few seconds after “save”.

On TARGET Environment (remote Cluster)

  1. Create an agent-service-config file using the samples available in the SAMPLES directory in standard-gitops-repo. Ensure that configMap name is correct and is suffixed with agent name.

  2. Apply the downloaded agent-yaml in the remote cluster’s default namespace:

kubectl apply -f <downloaded file>

Note: In case we want to use a different namespace, please locate and change “namespace: default” in the downloaded file for the service-account config.

3. Wait for the agent pods to come-up and be in “ready” state.

On OES UI

  1. Once the agent pods come-up, refresh the OES agent screen to check that the agent shows a “green” dot status

  2. Go to Integrations and Cloud Accounts pages to see the accounts configured in the agent-services configMap listed along with local accounts.

  3. One needs to “accept” and “save” each of the accounts in Integrations to make them active.

  4. On the “Integrations” screen, if the “Sync Accounts to Spinnaker” link on the top-right is active, click it. If possible, ensure that oes-spinnaker-halyard-0 is restarted. Else manually delete the pod to trigger a restart.

NOTE: “restart-halyard” pipeline in sampleapp in Spinnaker is configured correctly, halyard pod will restart automatically.

5. Wait for a few minutes, refresh the Spinnaker UI to see the agent-accounts available.

Troubleshooting Tips

  • App-sync keeps going round-and-round: Restart the sapor pod by terminating the running instance.

  • Spinnaker shows “Not Connected: Check credentials once again and restart the sapor pod by terminating the running instance. If credentials are incorrect, please correct, save and restart the sapor pod, if required.

  • oes-spinnaker-halyard-0 pod goes into crash loop: Check the git-repo values in values.yaml. If not clear, look at the logs of the create-halyard-local container has follows:

kubectl logs oes-spinnaker-halyard-0 -c create-halyard-local

  • Agent pods remain in “creating container” state: Ensure that service-configuration is applied. Sample configurations are available here

  • Agent unable to connect to the controller: “Context deadline exceeded” means that agent is unable to connect to the controller. Check the DNS-name/ingress routing and ensure that the agent is able to reach the agent-grpc service. Note that the agent talks to the controller using gRPC protocol. Hence TCP routing is required (not HTTP routing). If using an ingress-controller, ensure that TCP passthrough/routing is enabled.

  • Unable to deploy to k8s agent account from Spinnaker: This is usually not a problem related to the agent. Please check if the clouddriver pods are running correctly.

  • Unable to list jobs in Jenkins stage in Spinnaker: This is usually due to an issue with URL, username or password being incorrect. Please check and update. Once done, restart the agent-pods (kubectl scale deploy <deployment name> --replicas=0; followed by the same command with replicas=1)

  • Sample application not present: This can happen if the spin-cli process during installation fails due to some reason, typically if the gate-URL is incorrect in values.yaml. Please follow the instructions here: https://github.com/OpsMx/sample-pipelines“Sync Accounts with Spinnaker” button does not work: This makes a web-hook call to a Spinnaker pipeline (restartHalyard, in Opsmx-GitOps application). While there are many reasons for this to not work, most common is the account and namespace for this pipeline needs to be correct. Workaround is to manually delete the halyard pod.

Vault Initialization requirements

In case we decide to use the Vault backend for storing secrets, it needs to be initialized with some seed data that allows us to get started and login to the UI.

Login to Vault

In case of k8s installation, exec into the vault container, login and execute the below commands.

Initialize vault

vault secrets enable -path=secret kv-v2

Create Key-Value Pair

Common Data (same username and password for all DBs, they are different please put them under the appropriate service)

vault kv put secret/application spring.datasource.username=postgres
vault kv patch secret/application spring.datasource.password=networks123

For LDAP :

vault kv patch secret/application spring.ldap.managerPassword=opsmxadmin123
vault kv patch secret/application spring.ldap.url=ldap://oes-openldap:389
vault kv patch secret/application ldap.url=ldap://oes-openldap:389
vault kv patch secret/application ldap.managerPassword=opsmxadmin123

For SAML :

  • saml.keyStorePassword

  • saml.redirectHostname

  • saml.issuerId

For Outh2 :

  • oauth2.accessTokenUri

  • oauth2.userAuthorizationUri

  • oauth2.userInfoUri

  • oauth2.userInfoMapping.username

  • oauth2.provider

Platform secrets

vault kv put secret/oes-platform redis.connection=redis://:[email protected]:6379
vault kv patch secret/oes-platform spring.datasource.url=jdbc:postgresql://oes-db:5432/platformdb

Sapor

vault kv put secret/oes-sapor spring.datasource.url=jdbc:postgresql://oes-db:5432/oesdb

Visibility

vault kv put secret/oes-visibility spring.datasource.url=jdbc:postgresql://oes-db:5432/visibilitydb

Autopilot

vault kv put secret/oes-autopilot secret.datasource.username=postgres
vault kv patch secret/oes-autopilot secret.datasource.password=networks123
vault kv patch secret/oes-autopilot secret.datasource.url=jdbc:postgresql://oes-db:5432/opsmx
vault kv patch secret/oes-autopilot secret.platform.url=http://oes-platform:8095
vault kv patch secret/oes-autopilot secret.ds.protocol=http://
vault kv patch secret/oes-autopilot secret.ds.url=localhost:5005

Manually configuring Spinnaker in OES-UI

Once Spinnaker and OES are installed successfully, we need to add Spinnaker in OES in case the automatic configuration fails or is disabled, passwords have been updated or x509 configuration is desired.

Navigate to Setup → Spinnaker Setup → Add Spinnaker

Spinnaker Name : The name of Spinnaker Instance.

Spinnaker URL : The Gate URL of the Spinnaker instance with which OES will communicate to.

Authentication Type: Authentication Mechanism with which OES communicates to Spinnaker.

LDAP : For Basic Authentication (username with password)

X509 : For 2 Factor Authentication systems.

Using the LDAP Mechanism:

In Order to fetch the application using the LDAP, user has to provide the username and password of the user (who has the access to all the pipelines) in the encrypted base64 format using this command:

echo -ne “username:password” | base 64 -w0

Note: If you are using Multi Factor Authentication or oAuth, then this username password will not work directly as it will redirect to your third party app for passcode like things.

So in this case, we need to use x509 certificate based authentication for fetching the applications from Spinnaker to OES.

Using the x509 Mechanism:

For the x509 method of authentication, we need to configure x509 method of Authentication in Spinnaker.

To Enable x509 method of Authentication for Spinnaker, please refer to the steps in

https://spinnaker.io/setup/security/authentication/x509/

Key points:

  1. Spinnaker-gate listens to x509 communication on a different port (8085). This is in addition to the normal port (8084) used for web-login

  2. Spinnaker-gate will need to be configured with SSL. ISTIO needs to be configured to allow SSL communication between OES and Spinnaker-gate’s x509 port and between web-browser and Spinnaker-gate. TLS termination should happen at Spinnaker-gate (and not at ISTIO/ingress).

Once the Spinnaker is Enabled with x509 type of authentication, in order for OES to communicate with Spinnaker using x509, we need a client p12 key. Using the client.p12 key and password, we can configure OES to communicate with Spinnaker on the x509 port.

Note: As OES will communicate with Spinnaker gate using internal kubernetes service (spin-gate or spin-x509gate). We should ensure that this service name is added as one of the SANs (Subject Alternate Names) to the Gate certificate, so that gate can trust the API calls from OES and send back the requested details by OES.

If we did not have this name added as the SAN, then we need to have an alternate External URL with the Proper SAN Name added to the Gate Certificate, and configure this URL in the OES.

Spinnaker gate SSL and JKS creation instructions

https://spinnaker.io/setup/security/ssl/

Client-P12 creation instructions Assuming that we have client tls certificate and key we can generate a p12 using the below openssl command :

openssl pkcs12 -export -clcerts -in x509lb.crt -inkey x509lb.key -out x509lb.p12 -name gate509lb -password pass:changeit