Installation process

The cluster administrator is responsible for:

• Creating projects (namespaces)

• Creating roles, role bindings and service-accounts

• Installing Custom Resource (CR) definitions

• Configuring the OpenShift image-registry

The project administrator is responsible for:

• Installing Solution Hub and Designer

• Installing Envoys on prepared namespaces

• Providing necessary configuration data

• Pushing images to the image-registry

• Executing Helm after installation

Cluster-wide resources are setup via executing the script ssob_admin_setup.sh, that is part of the installer. It must be executed as a user with cluster-admin permissions in the OpenShift cluster (Role: Red Hat OpenShift cluster administrator). It can be configured via command line arguments, that are described later on.

Executing the ssob_admin_setup.sh script will create and configure the following:

• Custom Resource definitions

  • solutions.sol.rt.cp.knowis.de

  • envoys.env.rt.cp.knowis.de

  • k5clients.k5.project.operator

  • k5dashboards.k5.project.operator

  • k5pipelinemanagers.k5.project.operator

  • k5projects.k5.project.operator

  • k5realms.k5.project.operator

  • k5topics.k5.project.operator

• Instances of Custom Resource ConsoleYAMLSample

• Optionally: create a route for the OpenShift docker registry

• Roles

  • k5-aggregate-edit (provides edit access to the created Custom Resource Definitions above and aggregates it to the cluster roles admin and edit)

  • k5-aggregate-view (provides view access to the created Custom Resource Definitions above and aggregates it to the cluster role view)

  • cpd-admin-additional-role (provides admin access to routes and pods/portforward)

  • cpd-webhook-role (provides admin access to webhooks integration)

• ServiceAccounts

  • k5-operator-sa

• Creates role bindings (associates permissions to service-accounts in the namespace)

  • Service Account k5-operator-sa: cpd-admin-role, cpd-viewer-role, edit

  • Service Account cpd-viewer-sa: view

  • Service Account cpd-admin-sa-rolebinding: cpd-admin-additional-role

  • Service Account cpd-admin-sa-webhook-rolebinding: cpd-webhook-role

The ssob_admin_setup.sh script has the following parameters:

To check whether the internal container registry is already exposed via a route use the following command:

oc -n openshift-image-registry get routes

Example:

$ cd deployments
$ chmod +x ssob_admin_setup.sh
$ ./ssob_admin_setup.sh  \
  --accept_license  \
  --cpd_namespace=zen \
  --external_address_image_registry=image-registry.apps.{your.cluster.domain}

The installation of IBM Financial Services Workbench is done by executing the script ssob_install.sh. It requires to be executed as a user with OpenShift project admin permissions in the chosen OpenShift namespace, i.e. zen . The installation script requires a set of command line arguments.

The installer will configure and install the following components via Helm charts into the already existing namespace of CPD:

• Solution Designer

• Solution Hub

Without any further configuration the components will be reusing the existing cpd service-accounts respectively the created ones.

Each package contains all used docker images and Helm charts. During the installation all provided and used docker images are pushed into the configured registry. For convenience it should be the internal OpenShift registry.

Also after the installation of each Helm chart some Helm tests will be executed. The installation will fail, if one or more Helm tests are failing. Per default the Helm test is enabled and the Helm tests pods will be removed automatically. These steps can be skipped. In case of failing Helm tests, the pod description and if available the pod logs will be printed.

The installer is parametrized and adapted to your particular environment via CLI parameters and two value files.

There are two different types of parameters:

• Those that you will always have to adapt to your environment. They are given as CLI parameters to the ssob_install.sh script.

• Those that are likely to be used in their default form. These are configured via the yaml files solution-hub-values.yaml and solution-designer-values.yaml

You can find these files in the ssob-install/deployments/configs directory of the unpacked installation package.

Connect via scp to the server host and download the files to your local computer, for example:

scp user@remote_host:remote_file ~/remote_directory
scp root@11.11.11.11:/root/install/ssob_3.0/ssob-install/deployments/configs/install_init_data.yaml ~/install

Locally edit your files and then connect via scp to the server host and upload each edited file, for example:

scp local_file user@remote_host:remote_file
scp install_init_data.yaml root@11.11.11.11:/root/install/ssob_3.0/ssob-install/deployments/configs/install_init_data.yaml

Backup the following files:

• install.yaml

• solution-designer-values.yaml

• solution-hub-values.yaml

For example:

cd ~/install/ssob_3.0/ssob-install/deployments
mv configs configs.bak
mkdir configs
cp ./configs.bak/* configs

The ssob_install.sh script has the following parameters:

Run the following to check for the image registry service (look for the service running on port 5000):

oc get services -n openshift-image-registry

To obtain the TLS certificate data (created by the IBM Cloud Pak for Data installation) run the following commands:

oc get secret helm-secret -n zen -o jsonpath='{.data.ca\.cert\.pem}' | base64 -d > /root/install/ssob_3.0/zenhelm/ca.cert.pem
oc get secret helm-secret -n zen -o jsonpath='{.data.helm\.cert\.pem}' | base64 -d > /root/install/ssob_3.0/zenhelm/helm.cert.pem
oc get secret helm-secret -n zen -o jsonpath='{.data.helm\.key\.pem}' | base64 -d > /root/install/ssob_3.0/zenhelm/helm.key.pem

If the OpenShift image repository is used, the value for the parameter external_address_image_registry can be obtained by viewing the route of the service image-registry in the namespace openshift-image-registry:

oc -n "openshift-image-registry" get route image-registry

The value for the parameter internal_address_image_registry is composed like {image-registry service}.openshift-image-registry.svc:{port}. You can find the port by searching for the service with name like image-registry in the list of all services in the openshift-image-registry:

oc -n "openshift-image-registry" get services

YAML files for the installation:

The installer replaces values that are surrounded by # with appropriate values, e.g. registry: #DOCKER_REGISTRY#/#NAMESPACE# will be resolved to registry: image-registry.openshift-image-registry.svc:5000/zen.

Optional: YAML file solution-hub-values.yaml

This file contains settings for Helm deployment of the Solution Hub. It contains settings for the Configuration Management, k5-query component and the k5-project-operator. The solution-hub-values.yaml file is located in the directory ../ssob-install/deployments/configs and only needs to be adjusted if the default settings do not fit.

Format Example of solution-hub-values.yaml:

## Global (potentially shared) settings
global:
  hosts:
    domain: "#DOMAIN#"
  image:
    registry: "#DOCKER_REGISTRY#/#NAMESPACE#"
  configurationManagement:
    name: "k5-configuration.#DOMAIN#"
  k5configurator:
    name: "k5-configurator.#DOMAIN#"
  hub:
    name: "k5-hub.#DOMAIN#"
  k5pipelinemanager:
    name: "k5-pipeline-manager.#DOMAIN#"
## Operator namespace settings
k5projectoperator:
  watch:
    namespaces:
      blacklist: [ ]
      whitelist: [ ]
k5-hub-backend:
  endpoints:
    openshiftConsole:
      url: "https://console-openshift-console.#DOMAIN#"

Optional: YAML file solution-designer-values.yaml:

The solution-hub-values.yaml file is located in the directory ../ssob-install/deployments/configs and only needs to be adjusted if the default settings do not fit.

Format Example solution-designer-values.yaml:

## Global (potentially shared) settings
global:
  domain: "#DOMAIN#"
  hosts:
    domain: "#DOMAIN#"
  image:
    registry: "#DOCKER_REGISTRY#/#NAMESPACE#"
  ## Environment information
  environment:
    ## Optionally provide a name for the Solution Designer installation
    name: "designer"
  ## Hostnames of the sub components, for optionally overriding the default endpoints:
  endpoints:
    designer:
      host: "k5-designer.#DOMAIN#"
    gitIntegrationController:
      host: "k5-git-integration-controller.#DOMAIN#"
    solutionController:
      host: "k5-solution-controller.#DOMAIN#"
    codeGenerationProvider:
      host: "k5-code-generation-provider.#DOMAIN#"
    cliProvider:
      host: "k5-cli-provider.#DOMAIN#"
    queryService:
      host: "k5-s3-storage.#DOMAIN#"
    localMarketplaceController:
      host: "k5-local-marketplace-controller.#DOMAIN#"
    s3Storage:
      host: "k5-s3-storage.#DOMAIN#"

Example call of ssob_install.sh:

cd deployments
chmod +x ssob_install.sh
./ssob_install.sh \
  --accept_license \
  --external_address_image_registry=image-registry.apps.{your.cluster.domain} \
  --internal_address_image_registry=image-registry.openshift-image-registry.svc:5000 \
  --cpd_namespace=zen \
  --host_domain=apps.{your.cluster.domain} \
  --helm-tls-ca-cert=/root/.helm/ca.pem \
  --helm-tls-cert=/root/.helm/cert.pem \
  --helm-tls-key=/root/.helm/key.pem

For a new installation at least, the following configuration must be provided:

• ArgoCD: Configures the properties to access ArgoCD service

• HelmRepo: Configures the properties to access the Helm repository

• IAM: Configures the properties to access the Identity and Access Management system (IAM), respectively Keycloak

• MasterKey: Configures the master key required by the REST API to encrypt some sensitive user data, such as git-tokens or api-keys.

• MongoDB: Configures the connection to the Mongo database, which is used by the Solution Designer

• S3Storage: Configures properties to access a S3Storage, which is used as a persistence layer for the k5-marketplace

• Truststore: Set the truststore, which holds the certificates for external TLS/SSL handshake that should be trusted within IBM Financial Services Workbench.

For a detailed description of all values and in-depth explanation, please see configuring IBM Financial Services Workbench

For above command to work you need a running docker daemon on the host you are running the command on (your bastion host). If you get an error message like the following

$ Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Start docker like this (system dependent):

systemctl start docker
systemctl enable docker

If your container registry uses self signed certificates you might see something like the following:

Extracting modules
checking structure
structure check skip for now
docker login https://image-registry.apps.{your.cluster.domain}
Error response from daemon: Get https://image-registry.apps.{your.cluster.domain}/v1/users/: x509: certificate signed by unknown authority

To fix the issues with self signed certificates, please follow the instructions on https://docs.docker.com/registry/insecure/ and perform the following steps:

• retrieve the CA Cert from the image registry

• add it to the local trust store

• reload the trusted certificates

$ oc -n openshift-ingress get secret router-certs-default -o jsonpath='{.data.tls\.crt}' | base64 -d  > /etc/pki/ca-trust/source/anchors/image-registry.apps.{your.cluster.domain}.crt
update-ca-trust
$ systemctl restart docker
$ docker login -u admin -p {password} https://image-registry.{your.cluster.domain}

In case of failing k5-s3-storage with that log

ERROR Unable to initialize backend: Insufficient permissions to access path
Please ensure the specified path can be accessed

And if the storage class is nfs

$ oc get pvc k5-s3-storage -o yaml | grep "storageClassName"
storageClassName: nfs

the permissions must be repaired.

To do so, please follow these steps

1. Get the user id of the k5 s3 storage

   oc get pod  | grep k5-s3-storage
   k5-s3-storage-775db6d5fc-sjlqn                                  1/1     Running     0          6h6m
   oc get pod k5-s3-storage-775db6d5fc-sjlqn -o yaml | grep runAsUser
       runAsUser: 1000320900

2. Login to your nfs mount

3. Check the current permissions

   ls -la *
   total 336
   drwxr-xr-x. 2 1001 root   4096 17. Jun 19:27 .
   drwxrwxrwx. 4 root root     53 23. Jun 13:46 ..
   -rw-r--r--. 1 1001 root    462 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-meta.yml
   -rw-r--r--. 1 1001 root   1269 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-preview.json
   -rw-r--r--. 1 1001 root 144790 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb.zip
   -rw-r--r--. 1 1001 root    960 17. Jun 19:27 index.yml

4. Change owner

   chown -R 1000320900:1000320900 .

5. Check result

   ls -la *
   insgesamt 336
   drwxr-xr-x. 2 1000320900 1000320900   4096 17. Jun 19:27 .
   drwxrwxrwx. 4 1000320900 1000320900     53 23. Jun 13:48 ..
   -rw-r--r--. 1 1000320900 1000320900    462 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-meta.yml
   -rw-r--r--. 1 1000320900 1000320900   1269 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-preview.json
   -rw-r--r--. 1 1000320900 1000320900 144790 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb.zip
   -rw-r--r--. 1 1000320900 1000320900    960 17. Jun 19:27 index.yml

6. Change permission

   chmod g+rwx . -R

7. Check result

   ls -la *
   insgesamt 336
   drwxrwxr-x. 2 1000320900 1000320900   4096 17. Jun 19:27 .
   drwxrwxrwx. 4 1000320900 1000320900     53 23. Jun 13:48 ..
   -rw-rwxr--. 1 1000320900 1000320900    462 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-meta.yml
   -rw-rwxr--. 1 1000320900 1000320900   1269 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb-preview.json
   -rw-rwxr--. 1 1000320900 1000320900 144790 17. Jun 19:22 c48a5afe-3d1b-4c18-ad61-2dd9c5d078eb.zip
   -rw-rwxr--. 1 1000320900 1000320900    960 17. Jun 19:27 index.yml

If the fss clone command is failing on MacOS like this

fss clone -s MYSOLUTION -p "my-git"
========= Cloning Solution to filesystem =================================================
--------- > Authenticating ---------------------------------------------------------------
--------- > Cloning Solution from Solution Git Repository --------------------------------
Cloning into '/dev/MYSOLUTION'...
fatal: unable to access 'https://my-git/MYSOLUTION.git/': error setting certificate verify locations:
CAfile: /Users/MyUser/.knowis/fss-cli/default/designtime.ca.crt
CApath: /Users/MyUser/.knowis/fss-cli/default

[ERROR] Cloning failed, removing directory: /dev/MYSOLUTION

Then please verify, if the file /Users/MyUser/.knowis/fss-cli/default/designtime.ca.crt has proper base64 encoded values only. To do so, open the file and verify, that all lines between the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- do not exceed 64 characters. For manual and local fixing you can adjust the lines by breaking after 64 characters. And verify, that this is solving the experienced issue.

To fix it generally, the value of global.truststore.trustMap.identity must be adjusted in a similar way. Afterwards the setup of fss must be reset by downloading the designtime.config.json and executing fss setup --file ./cli-config.json.

If a request is rejected and the response contains invalid_token, then it is helpful to decode the JWT itself by using for example jwt.io. So it is easier to see, if the JWT is decode-able and what kind of content it has, and to understand, what might cause the unexpected rejections.

If a request is rejected and the response contains invalid_token in combination of The iss claim is not valid, then the JWT was created by an OIDC provider using a different issuer URL, than the configured one.

It is helpful to decode the JWT itself by using for example jwt.io and check the value of iss. That must be the same as it is configured described by configuring OIDC provider for solutions and configuring deployment targets.