Upgrading from 3.1 (TP) to 4.0

Backup the following secrets:

• Login to your cluster and switch into your installation namespace

oc login
oc project namespace

• Save the default configurations (for the runtime mongodb, kafka instance and oidc provider)

 oc get secret k5-default-document-storage-service-binding -o=yaml > k5-default-document-storage-service-binding.yaml
 oc get secret k5-default-message-service-binding -o=yaml > k5-default-message-service-binding.yaml
 oc get secret k5-default-iam-service-binding -o=yaml > k5-default-iam-service-binding.yaml

• Save the product configuration secrets, see also configure the product

 oc get secret --selector='k5-configurable=true' -o=yaml > backup-config-secrets.yaml

• At least the following secrets must be saved and re-created in case they get deleted in the cluster by mistake:

oc get secret --selector='k5-configurable=true' -o=name
secret/k5-argocd-binding
secret/k5-designer-mongodb
secret/k5-encryption-master-key
secret/k5-hub-truststore
secret/k5-iam-secret
secret/k5-iam-settings

The uninstallation needs to be done via helm cli (version 2). There are 2 options to do this:

• Option 1: Use a local helm tool (on your machine), then you need to get the tiller/helm certificates from the instance running in your installation namespace

  • To obtain the TLS certificate data (created by the IBM Cloud Pak for Data installation) run the following commands:

oc project namespace
oc get secret helm-secret -o jsonpath='{.data.ca\.cert\.pem}' | base64 -d > ./ca.cert.pem
oc get secret helm-secret -o jsonpath='{.data.helm\.cert\.pem}' | base64 -d > ./helm.cert.pem
oc get secret helm-secret -o jsonpath='{.data.helm\.key\.pem}' | base64 -d > ./helm.key.pem

• The saved certificates can then be used for the following command by adding the flags: --tls-ca-cert ./ca.cert.pem --tls-cert ./helm.cert.pem --tls-key ./helm.key.pem

• Option 2: Connect to the running tiller pod in your cluster and execute the following command in this pod

  • This can be done by searching for the tiller pod in the web console in your installation namespace

  • Then open the tiller pod and switch to the Terminal

  • Use the helm tool from the tiller pod (can be tested with /helm list --tls)

The following commands need to be executed for the uninstallation:

• List all installed helm deployments

helm list --tls

• Delete/uninstall all helm deployments of Cloud Pak for Data, Solution-Hub and Solution-Designer

helm delete RELEASE_NAME --purge --tls

For example:

helm delete 0010-infra --purge --tls
helm delete 0015-setup --purge --tls
helm delete 0020-core --purge --tls
helm delete namespace-ssob-solution-hub --purge --tls
helm delete namespace-ssob-solution-designer --purge --tls

All installation configurations you may have set in one of the following files in your previous installation can now be set as well during the new operator based installation in the ISW Custom Resource, see also Configure ISW Custom Resource:

• solution-designer-values.yaml

• solution-hub-values.yaml

Especially if you have used specific urls or database names you need to add these parameters to your ISW Custom Resource, see especially spec.values.global.endpoints, spec.values.service-builder.k5-designer-backend.mongoDb.dbName and spec.values.service-builder.k5-git-integration-controller.mongoDb.dbName in the following example:

apiVersion: k5.ibm.com/v1beta1
kind: ISW
metadata:
  name: k5-tools
  namespace: k5-tools
spec:
  designer:
    enabled: true
  domain: apps.openshift.my.cloud
  license:
    accept: true
  values:
    global:
      endpoints:
        assetManager:
          host: k5-asset-manager.apps.openshift.my.cloud
        configurationManagement:
          host: k5-configuration-management.apps.openshift.my.cloud
        designer:
          host: k5-designer.apps.openshift.my.cloud
        hub:
          host: k5-hub.apps.openshift.my.cloud
        query:
          host: k5-query.apps.openshift.my.cloud
    service-builder:
      k5-designer-backend:
        mongoDb:
          dbName: my-designer-database
      k5-git-integration-controller:
        mongoDb:
          dbName: my-gic-database

Java pro-code uses several functionalities provided by a dependency "cp-framework-sdk", this dependency has been updated to include new fixes, and versions of underlying used dependencies.

All existing Java pro-code are required to migrate to Download and migrate to this new version.

Within Domain namespace a "Date" property used to be generated as "offsetDateTime" data type, however customers are using "Date" properties only to store dates, hence "Date" property is now generated as "LocalDate" data type.

Api namespace schema setter/getter was not following proper camelCase naming.

For Example:

• Proeprty named "myProperty" getters / setters where generated as setmyProperty() and getmyProperty

• Now this is aligned to be setMyProperty() and getMyProperty()

Java low-code uses several functionalities provided by a dependency "cp-framework-sdk", this dependency has been updated to include new fixes, and versions of underlying used dependencies.

All existing Java low-code are required to migrate to this new version.

Previously service, command and agent used to get "DomainEventBuilder" and "EventProducerService" classes injected as constructor parameters, even if they are not modelled to fire business events.

Now only service, command and agent that are modelled to fire events will get these injected constructor parameters.

If you have compilation errors because these parameters were removed, remove them from your implementation class.

// Before change constructor getting DomainEventBuilder and EventProducerService
public class FailedCompensationAgent extends FailedCompensationAgentBase {

  private static Logger log = LoggerFactory.getLogger(FailedCompensationAgent.class);

  public FailedCompensationAgent(
    DomainEntityBuilder entityBuilder,
    DomainEventBuilder eventBuilder,
    EventProducerService eventProducer, 
    Repository repo) {
    super(entityBuilder, eventBuilder, eventProducer, repo);
  }
}

// After change base class constrcutor no longer have DomainEventBuilder and EventProducerService
public class FailedCompensationAgent extends FailedCompensationAgentBase {

  private static Logger log = LoggerFactory.getLogger(FailedCompensationAgent.class);

  public FailedCompensationAgent(
    DomainEntityBuilder entityBuilder,
    Repository repo) {
    super(entityBuilder, repo);
  }
}

Integer, Decimal and Date used to be string type. Now Integer has the native Javascript (BigInt) type, Decimal now has BigNumber type from bignumber.js and Date now has the native Javascript Date type.

IBM Industry Solutions Workbench 4.0 completely changes how the operation file is generated and structured, to be more aligned with Typescript and express controller style, thus allowing it to support more complex api design.

In order to do a successful migration we need to first understand what are the structure and elements of an Operation File, so we can do a proper migration for the logic of each operation.

1. Only one API dependency is allowed per integration namespace, if you are working with a version that supported many dependencies in an integration namespace, you need to create additional integration namesapces and upload these dependencies to.

2. Calling an Api dependency operation is different

   • It's directly under this.apis.integrationNamespaceAcronym

   • Return Type is different, All the api dependency operation now return full axios response, to access the response body, its mapped to data property.

   • Schemas are all grouped under integrationSchema.integrationNamespaceAcronymSchema

See below illustrative comparison between IBM Industry Solutions Workbench 3.1 and IBM Industry Solutions Workbench 4.0

This approach involves two steps: one before upgrading to IBM Industry Solutions Workbench 4.0 and another step is after upgrading to IBM Industry Solutions Workbench 4.0.

• Before upgrading to IBM Industry Solutions Workbench 4.0: User need to go to designer and

  1. Import all project templates (one by one) as projects into the designer.

  2. Create an initail commit from designer

• After upgrading to IBM Industry Solutions Workbench 4.0: Now for each of the projects created in above step, user have to share those projects as an asset and make it available in an asset catalog. The asset catalog needs to be configured first, see Configuring Asset Catalogs and Assets Overview.

This approach involves three steps:

1. Download all existing templates in local machine (to be performed before upgrade to IBM Industry Solutions Workbench 4.0)

2. Transform all templates to correct format (can be done before or after upgrade once the templates are downloaded)

3. Upload templates to designer (to be performed after upgrade to IBM Industry Solutions Workbench 4.0)

Step 1: This task has to be performed before upgrading to new release.

Requirements:

1. A machine to execute the task.

2. Machine should have access to cluster.

3. A user with valid Keycloak user credentials.

Outcome: All the templates available in designer are now available in local machine of customer.

Steps:

1. Copy the bash script below into a get-templates.bash file in user's machine

2. Add missing values for configuration in bash script :

   1. Add the values for the user credentials (keycloak/designer username and password)

   2. keycloak details: configured in authentication binding for a project configuration management

      1. KEYCLOAK_BASE_URL= {{hostname}}

      2. CLIENT_ID= {{client.id}}

      3. CLIENT_SECRET= {{client.secret}}

      4. DESIGNER_REALM= {{realm}}

   3. Destination folder in local machine where templates can be saved.

   4. External route url for local market place controller service.

3. Open terminal in directory where bash script is copied and run command bash get-templates.bash.

4. Bash Script will start downloading the templates one by one and store them in the directory specified for DESTINATION.

5. Bash Script will print the logs in the terminal and also any error message (in case of error).

6. Once all the templates are downloaded, script will stop

#Configuration for keycloak client
# https://auth.apps.openshift.example.cloud
KEYCLOAK_BASE_URL=  
CLIENT_ID=
CLIENT_SECRET=
USER_NAME=
PASSWORD=
DESIGNER_REALM=

# Provide template download location
DESTINATION="./templates"


# provide the url for the local marketplace controller api https://local-marketplace-controller.apps.openshift.example.cloud
LOCAL_MARKETPLACE_CONTROLLER_HOST={{LOCAL_MARKETPLACE_CONTROLLER_URL}}


# Get access token
TOKEN_STATUS_CODE=$(curl -X POST -o token.json -w "%{http_code}" -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" -d "grant_type=password" -d "username=$USER_NAME" -d "password=$PASSWORD" "$KEYCLOAK_BASE_URL/auth/realms/$DESIGNER_REALM/protocol/openid-connect/token")

TOKEN_RESPONSE=`cat ./token.json`
if [ $TOKEN_STATUS_CODE == "200" ]; then
  TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.access_token')
  # # Get list of all templates available in marketplace
  RESPONSE_CODE=$(curl -X GET  -w "%{http_code}"  -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" "$LOCAL_MARKETPLACE_CONTROLLER_HOST/api/v1/templates" -o ./templates.json)

  # read templates json
  TEMPLATES=`cat ./templates.json`
else
    echo "Failed to get token, please check keycloak config"
    echo $TOKEN_RESPONSE
fi

# check if api call is successful, other wise log the error
if [[ $RESPONSE_CODE != "200" ]]; then
      echo "invalid response from local marketplace api"
      echo $TEMPLATES
      rm ./templates.json
      exit 1
fi

cat $TOKEN 

#iterate through the Templates array and read template id of all the templates
templateids=$(echo "$TEMPLATES" | jq -c -r '.[].id')


# download each template in to a zip file using local market place controller api
for templateid in ${templateids}
 do
  echo $templateid
  # zip file path
  FILE=$DESTINATION/$templateid-export.zip
  # check if template zip file already exists in the destination folder
  if [[ ! -f "$FILE" ]]
  then
    echo "$FILE does not exist on your filesystem."
    echo "downloading template zip"
  # download the template
  STATUS=$(curl -X GET -w "%{http_code}" -H "accept: application/octet-stream" -H "Authorization: Bearer $TOKEN"  $LOCAL_MARKETPLACE_CONTROLLER_HOST/api/v1/templates/$templateid --output $FILE)
    # get a new access_token if the token has expired 
    if [ $STATUS == "401" ]; then
      STATUS_CODE=$(curl -X POST -o token.json -w "%{http_code}" -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" -d "grant_type=password" -d "username=$USER_NAME" -d "password=$PASSWORD" "$KEYCLOAK_BASE_URL/auth/realms/$DESIGNER_REALM/protocol/openid-connect/token")
      TOKEN_RESPONSE=`cat ./token.json`
      TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.access_token')      
      # retry for last template
      STATUS=$(curl -X GET -w "%{http_code}" -H "accept: application/octet-stream" -H "Authorization: Bearer $TOKEN"  $LOCAL_MARKETPLACE_CONTROLLER_HOST/api/v1/templates/$templateid --output $FILE)
    # print error and remove error file if response is not 200
    elif [ $STATUS != "200" ]; then
      echo "Failed to download template"
      cat $FILE
      rm $FILE
    fi
  fi
  sleep 2s
done
#  remove token.json from file system
rm ./token.json

Step 2: Transform the templates to get them ready for upload. Requirements:

1. A machine to execute the task.

2. Machine should already have downloaded templates.

Outcome: All the templates are transformed and available in new directory in local machine of customer.

Steps:

1. Copy the bash script below into a transform-templates.bash file in user's machine

2. Add missing values for configuration in bash script:

   1. Directory path where templates are saved from step 1.

   2. Destination directory path for transformed templates.

3. Open terminal in directory where bash script is copied and run command bash transform-templates.bash.

4. Bash Script will start transforming the templates and save transformed templates to transformed destination.

5. Once all the templates are downloaded, script will stop

# directory where template zip files are stored
TEMPLATE_DIRECTORY=

# directory used to store the templates after transformation
TRANSFORMED_TEMPLATES=

# traverse through template directory and transform to update the preview.json for all templates
for entry in "$TEMPLATE_DIRECTORY"/*
do
  echo "$entry"
  TemplateZipName=$(echo "${entry##*/}")
  echo "$TemplateZipName"
  TemplateDirectoryName=$(echo $TemplateZipName| cut  -d'.' -f 1); 

  # unzip the export zip
  unzip -d ./$TemplateDirectoryName/ $entry
  SolutionZipPath=$(find ./$TemplateDirectoryName/* -name "*.zip")
  SolutionZipName=$(echo "${SolutionZipPath##*/}")
  SolutionDirectoryName=$(echo $SolutionZipName| cut  -d'.' -f 1); 

  echo $SolutionDirectoryName
  #  unzip the inner template zip
  unzip -d ./$TemplateDirectoryName/$SolutionDirectoryName $SolutionZipPath
  #  read preview json from root folder
  previewJson=$(find ./$TemplateDirectoryName/ -name "*-preview.json")
  # copy preview json from root folder to template data folder
  cp $previewJson ./$TemplateDirectoryName/$SolutionDirectoryName/preview.json
  rm $SolutionZipPath
  cd $TemplateDirectoryName
  cd $SolutionDirectoryName
  zip -r $SolutionDirectoryName *
  mv $SolutionZipName ../$SolutionZipName
  cd ..
  rm -rf ./$SolutionDirectoryName
  zip -r $TemplateDirectoryName *
  mv $TemplateZipName .$TRANSFORMED_TEMPLATES
  cd ..
  rm -rf ./$TemplateDirectoryName
done

Step 3: This task has to be performed after upgrading to new release. Requirements:

1. A machine to execute the task.

2. Machine should have access to cluster.

3. A user with valid Keycloak user credentials.

4. User should have access (developer or above) to all the service projects in designer.

5. The asset catalog is configured to publish assets, see Configuring Asset Catalogs and Assets Overview

6. The external route for Asset Manager need to enabled by updating configuration in Extended configuration

Outcome: All the templates now available in designer and under asset catalog configured.

Steps:

1. Copy the bash script below into a upload-templates.bash file in user's machine

2. Add missing values for configuration in bash script :

   1. Add the values for the user credentials (keycloak/designer username and password)

   2. keycloak details: configured in authentication binding for a project configuration management

      1. KEYCLOAK_BASE_URL= {{hostname}}

      2. CLIENT_ID= {{client.id}}

      3. CLIENT_SECRET= {{client.secret}}

      4. DESIGNER_REALM= {{realm}}

   3. Catalog alias of asset catalog where all assets need to be published.

   4. Directory path where transformed templates are available.

   5. External route url for K5 Asset Manager Service.

3. Open terminal in directory where bash script is copied and run command bash upload-templates.bash.

4. Bash Script will start uploading the templates one by one.

5. Bash Script will print the logs in the terminal and also any error message (in case of error).

6. Once all the templates are downloaded, script will stop

7. User can see all the uploaded assets by browsing the asset catalog in designer.

Once the template is uploaded as new asset, it will be removed from local directory, so make a copy of the templates in case they are needed for future.

#!/bin/bash

#Configuration for keycloak client
# https://auth.apps.openshift.example.cloud
KEYCLOAK_BASE_URL=  
CLIENT_ID=
CLIENT_SECRET=
USER_NAME=
PASSWORD=
DESIGNER_REALM=

# URL for asset manager service https://k5-asset-manager.apps.openshift.example.cloud
ASSET_MANAGER_HOST={{ASSET_MANAGER_URL}}

# Catalog alias where new assets should be created
ASSET_CATALOG={{CATALOG_ALIAS}}

# Directory where transformed template zip files are stored
TEMPLATE_DIRECTORY=

# Get access token
STATUS_CODE=$(curl -X POST -o token.json -w "%{http_code}" -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" -d "grant_type=password" -d "username=$USER_NAME" -d "password=$PASSWORD" "$KEYCLOAK_BASE_URL/auth/realms/$DESIGNER_REALM/protocol/openid-connect/token")
TOKEN_RESPONSE=`cat ./token.json`
if [ $STATUS_CODE == "200" ]; then
  TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.access_token')
else
    echo "Failed to get token, please check keycloak config"
    echo $TOKEN_RESPONSE
fi


# traverse through template directory and call /convertAsset api for each template
for entry in "$TEMPLATE_DIRECTORY"/*
do
  echo "$entry"

# upload template to convert it to new asset
STATUS=$(curl -X POST -s -w "%{http_code}" -H "Accept: application/json" -H "Content-Type: multipart/form-data" -H "Authorization: Bearer $TOKEN" -F "file=@$entry;type=application/zip" "$ASSET_MANAGER_HOST/api/v1/$ASSET_CATALOG/assets/importTemplate" --output ./output.json)
echo $STATUS
#  check if successfully uploaded
if [ $STATUS == "201" ]; then
    # remove zip file from directory
    rm $entry
# get a new access_token if the token has expired 
elif [ $STATUS == "401" ]; then
      STATUS_CODE=$(curl -X POST -o token.json -w "%{http_code}" -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" -d "grant_type=password" -d "username=$USER_NAME" -d "password=$PASSWORD" "$KEYCLOAK_BASE_URL/auth/realms/$DESIGNER_REALM/protocol/openid-connect/token")
      TOKEN_RESPONSE=`cat ./token.json`
      if [ $STATUS_CODE == "200" ]; then
        TOKEN=$(echo $TOKEN_RESPONSE | jq -r '.access_token')
        # retry for last template
        STATUS=$(curl -X POST -s -w "%{http_code}" -H "Accept: application/json" -H "Content-Type: multipart/form-data" -H "Authorization: Bearer $TOKEN" -F "file=@$entry;type=application/zip" "$ASSET_MANAGER_HOST/api/v1/$ASSET_CATALOG/assets/importTemplate" --output ./output.json)
      else
          echo "Failed to get token, please check keycloak config"
          echo $TOKEN_RESPONSE
      fi
else
    echo "Failed to convert template"
    # print error response body
    cat output.json
    # remove output file
    rm output.json
fi
sleep 2s
done