Deploying Workloads in Bulk

The following commands require a session

Check the Quickstart guide on how to login.

Deploying workloads in bulk

To assist with managing a large number of workload deployments, the deploy-bulk command provides a way to specify the particulars of each deployment in a file (CSV, YAML, JSON) so that the deployments can be performed in one repeatable step.

bash/terminal

$ kelvin workload deploy-bulk --help
Usage: kelvin workload deploy-bulk [OPTIONS] FILENAME

  Deploy workloads in bulk.

Options:
  --file-type [csv|json|yaml]  Type of the workload file.
  -o, --output-filename TEXT

  -y, --ignore-failures        Ignore deployment failures and automatically
                               continue.  [default: False]

  -s, --skip-successes         Skip deployments already marked as successful.
                               [default: False]

  -d, --dry-run                Dry-run to validate inputs only.  [default:
                               False]

  -v, --verbose                Display all executed steps to the screen.

The bulk deployment input file

The deploy-bulk command takes an input file to iterate. This input file must indicate the headers that are going to be associated with each and every deployment step:

• node_name is the Node name to deploy to.
• app_name is the app name to deploy.
• app_version is the version of the app to deploy.
• workload_name is the unique name of the workload to deploy.
• workload_title is the title of the workload.
• app_config is the path of the application configuration to use (e.g. app.yaml)
• field_1,...,field_n are optional additional fields that can be used to inject values into the application configuration via <+ field +> variables.

This input file, can either by in CSV format(e.g. workloads.csv) containing the input fields as columns.

csv headers

node_name,app_name,app_version,workaload_name,workload_title,app_config,field_1,...,field_n

or, alternatively be provided as a YAML file (e.g. workloads.yaml) as a list of entries.

YAML

- node_name: ...
  app_name: ...
  app_version: ...
  workload_name: ...
  workload_title: ...
  app_config: ...
  field_1: ...
  field_n: ...

Bulk deployment example

1. This example will deploy a producer application with both a minimum and a maximum producer values. The key difference is that each workload will have personalised values depending on the Node it is deployed to.

2. Parameterize the app.yaml with template variables (e.g. <+ ... +>).

   app:
     kelvin:
       configuration:
         min: <+ min_value +>
         max: <+ max_value +>
       language:
         python:
           entry_point: producer_app:App
           requirements: requirements.txt
         type: python
     type: kelvin
   info:
     description: producer-app
     name: producer-app
     title: producer-app
     version: 1.0.1
   spec_version: 2.0.0
   

3. Create the workloads.yaml file with the workloads to be deployed from the template:
   

   # workloads.yaml
   - node_name: example-node
     app_name: producer-app
     app_version: 1.0.2
     workload_name: example-workload-1
     workload_title: "Example workload 1"
     app_config: app.yaml
     min_value: 1.0
     max_value: 200.0
   - node_name: example-node
     app_name: producer-app
     app_version: 1.0.2
     workload_name: example-workload-2
     workload_title: "Example workload 2"
     app_config: app.yaml
     min_value: 50.0
     max_value: 375.0
   

4. Process the workloads.yaml to deploy each workload, using -y to continue to the next deployment if any fail to deploy:

   $ kelvin workload deploy-bulk -y workloads.yaml
   
   [kelvin.sdk][2020-05-28 11:06:56][I] Deploying configuration in bulk..
   [kelvin.sdk][2020-05-28 11:07:00][I] Generating runtime configuration.
   generating runtime configuration: 100%|████████████████████| 2/2
   [kelvin.sdk][2020-05-28 11:07:00][I] Deploying workloads
   my-workload-1                   :  50%|██████████         | 1/2
   my-workload-2                   : 100%|████████████████████| 2/2
   [kelvin.sdk][2020-05-28 11:07:00][R] Bulk deployment operation concluded with success.
   

5. Check the log output and the workloads_result.yaml file to see the result/status of each deployment:

   - app_config: app.yaml
     app_name: producer-app
     app_version: 1.0.1
     max_value: 200.0
     min_value: 1.0
     node_name: example-node-1
     result: success
     status: valid
     workload_name: example-workload-1
     workload_title: Example workload 1
   - app_config: app.yaml
     app_name: producer-app
     app_version: 1.0.1
     max_value: 375.0
     min_value: 50.0
     node_name: example-node-1
     result: success
     status: valid
     workload_name: example-workload-2
     workload_title: Example workload 2
   

Note

Should any deployments fail the output file can be reprocessed and optionally skip any deployments that were already successful (--skip-successes)

$ kelvin workload deploy-bulk -y --skip-successes -o workloads_result.yaml

and iterated until all deployments were successful.