Skip to content

Add CSV Publisher Connection - How To

When developing or testing new Kelvin SmartApps™ there may be times when you have a set of data you want to test on your Kelvin SmartApp™.

You can easily achieve this by using this CSV publisher which will use a set of data in a CSV file and simulate a live Connection.

Only available for x86_64 devices. arm64 version coming soon.

Connections was formerly called Bridges. You will find the Kelvin SDK and Kelvin API commands still refer to Connections as Bridges.

We first need to prepare the CSV file with data.

You can download a sample template in Step 3 of Creating a Connection in the Kelvin UI, or you can simply create your own CSV file as there are only three main rules for the CSV file;

  1. The first row is the header where each column name is a Data Stream name.
  2. Optionally the first column of the header can be named timestamp which is used to control when data is released to the Asset / Data Stream pairs.
  3. Each following row is one set of data to be sent to the Asset / Data Stream pairs.

The headers are referencing the Data Stream name which must be lowercase alphanumeric without spaces. If you use a different name, you will still have opportunity to manually connect the header name with the correct Data Stream in kelvin UI during Step 4.

Once you have created this file, which for these examples we have called publisher-example.csv, then proceed to with one of these methods to deploy the Connection.

To start click on the Connections menu option from the left side bar menu.

click on the Create Connection button.

Step 1

Select the Kelvin CSV Publisher Connector option and click Next.

Step 2

Type in a memorable name in the Display Name text input. You can use any letters, numbers and special characters.

The Connection Name text input will be automatically converted and filled in as you type in the Connection Display Name section. The conversion ensures the Connection Name only contains lowercase alphanumeric characters and ., _ or - characters.

Then select which Cluster to deploy the new Connector to.

It is important that the asset is reachable from the selected Cluster.

And finally select the Asset the data will be sent to.

You can only select one Asset per Connection to send the data to.

When ready click Next to move to Step 3.

Step 3

Drag and drop or select your file to upload.

At this stage you can also download a sample template for reference on how to create a valid publishing CSV file.

Once you have selected the right CSV file you can click Next to go to Step 4.

Step 4

The file is now validated.

Once done you can click Next to go to Step 5.

Step 5

In this step you can now connect the column name to the Data Stream name.

Kelvin UI will attempt to automatically select the right Data Stream based on the column name.

You do not have to accept this and can select any Data Stream name in the Instance.

Once you are satisfied you can click Next to go to Step 5.

Step 6

Now you can select how the data will be replayed to the Asset / Data Stream pairs.

When you are ready, click Connect and the Connection will be deployed and automatically started.

The two main playback options are;

This option allows you to loop the data to produce an infinite amount of data or you can send the data once.

Very useful for both testing long term reliability of your Kelvin SmartApp™ with infinite loop

OR testing the accuracy of your Kelvin SmartApp™ with a known set of data through one iteration to monitor its response.

This option allows you to select what time intervals to use to send data to the Asset / Data Stream pairs.

CSV Playback

This option uses the timestamp information in the timestamp column of the CSV file on when to release the data.

The timestamp of the first data row is used as the starting point reference and each consecutive row will be sent based on the time different of the first row and next row to be sent.

Row Publish Rate

This option will send each row of data at a fixed interval and ignore any timestamp column in the CSV file.

Before deploying a CSV Publisher Connection using the Kelvin API you will need to upload the CSV file to File Storage first and get the reference ID.

API cURL Example
1
2
3
4
5
6
7
curl -X "POST" \
"https://<url.kelvin.ai>/api/v4/filestorage/upload" \
-H "Authorization: Bearer <Your Current Token>" \
-H "Accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F 'file=@/home/ubuntu/publisher-example.csv;type=text/plain" \
-F 'metadata={"customkey": "Custom metadata information"}'

The response will look like this;

API cURL Example Response
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
{
"file_id": "66fce65f-d43f-4424-89b8-e1dd01f9193b",
"file_name": "publisher-example.csv",
"file_size": 6,
"checksum": "66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18",
"created": "2024-03-19T10:29:55.673471Z",
"source": "krn:user:demo@kelvin.ai",
"metadata": {
    "customkey": "Custom metadata information"
}
}

You can now deploy the CSV Publisher connection

You will need to note the file_id which will be used as part of the csv_url in the Connection as reference to the data to be published.

The column header in the csv file and the data stream name in metrics_map does not need to be the same.

It is only important that the column name exists in the csv file and the data stream name exists in Kelvin.
API cURL Example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
curl -X "POST" \
  "https://<url.kelvin.ai>/api/v4/bridges/deploy" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "app_name": "kelvin-csv-publisher-connector",
    "app_version": "1.0.9",
    "cluster_name": "demo-cluster",
    "name": "doc-demo-csv-publisher",
    "title": "Doc Demo CSV Publisher",
    "payload": {
        "configuration": {
            "csv_url": "https://<url.kelvin.ai>/api/v4/filestorage/188d8d2a-2daa-4b2b-90fe-7ccd25f2cd45/download",
            "replay": true,
            "timestamp": {
                "type": "playback"
            }
        },
        "language": {
            "python": {
                "entry_point": "kelvin_python_sdk"
            },
            "type": "python"
        },
        "metrics_map": [
            {
                "access": "RO",
                "asset_name": "pcp_50",
                "configuration": {
                    "column": "tubing_pressure"
                },
                "data_type": "number",
                "name": "tubing_pressure"
            }
        ]
    }
}'

You will get a response similar to this;

API cURL Example Response
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
   "name":"doc-demo-csv-publisher",
   "title":"Doc Demo CSV Publisher",
   "cluster_name":"demo-cluster",
   "workload_name":"doc-demo-csv-publisher",
   "payload":{
      "configuration":{
         "csv_url":"https://<url.kelvin.ai>/api/v4/filestorage/100ccdb3-a587-488c-87d3-a498bf3cdc06/download",
         "replay":true,
         "timestamp":{
            "type":"playback"
         }
      },
      "language":{
         "python":{
            "entry_point":"kelvin_python_sdk"
         },
         "type":"python"
      },
      "metrics_map":[
         {
            "access":"RO",
            "asset_name":"pcp_50",
            "configuration":{
               "column":"tubing_pressure"
            },
            "data_type":"number",
            "name":"tubing_pressure"
         }
      ]
   },
   "enabled":true,
   "app_name":"kelvin-csv-publisher-connector",
   "app_version":"1.0.9",
   "status":{
      "state":"pending_deploy",
      "message":"Pending for deploy",
      "last_seen":"2024-06-01T07:03:38.77038855Z",
      "warnings":null
   },
   "created":"2024-06-01T07:03:38.488434Z",
   "updated":"2024-06-01T07:03:38.488434Z"
}

And after a while the state will change from pending_deploy to running.

You will need to first upload a file to the file storage on Kelvin. This file holds all the data used for serving as the simulation data.

You can learn more how to upload and manage files in the file storage on Kelvin here.

The column header in the csv file and the data stream name in metrics_map does not need to be the same.

It is only important that the column name exists in the csv file and the data stream name exists in Kelvin.

API Client (Python) Example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
from kelvin.api.client import Client

# Login
client = Client(config={"url": "https://<url.kelvin.ai>", "username": "<your_username>"})
client.login(password="<your_password>")

# Upload file to filestore
response_csvfile = client.filestorage.upload_file(
    file='/home/ubuntu/publisher-example.csv', 
    metadata={"description": "Data required for the My CSV Publisher Connection."}
)

# Create Connection (Bridge)
response = client.bridge.deploy_bridge(data={
    "app_name": "kelvin-csv-publisher-connector",
    "app_version": "1.0.9",
    "cluster_name": "demo-cluster",
    "name": "doc-demo-csv-publisher",
    "title": "Doc  Demo CSV Publisher",
    "payload": {
        "configuration": {
            "csv_url": f"https://<url.kelvin.ai>/api/v4/filestorage/{response_csvfile.file_id}/download",
            "replay": True,
            "timestamp": {
                "type": "playback"
            }
        },
        "language": {
            "python": {
                "entry_point": "kelvin_python_sdk"
            },
            "type": "python"
        },
        "metrics_map": [
            {
                "access": "RO",
                "asset_name": "pcp_50",
                "configuration": {
                    "column": "tubing_pressure"
                },
                "data_type": "number",
                "name": "tubing_pressure"
            }
        ]
    }
})

print(response)

You will get a response similar to this;

API Client (Python) Example Response
1
cluster_name='demo-cluster' created=datetime.datetime(2024, 6, 1, 7, 38, 17, 154694, tzinfo=datetime.timezone.utc) enabled=True name='doc-demo-csv-publisher' node_name=None payload=AppYaml(configuration={'csv_url': "https://<url.kelvin.ai>/api/v4/filestorage/100ccdb3-a587-488c-87d3-a498bf3cdc06/download', 'replay': True, 'timestamp': {'type': 'playback'}}, metrics_map=[{'access': 'RO', 'asset_name': 'pcp_50', 'configuration': {'column': 'tubing_pressure'}, 'data_type': 'number', 'name': 'tubing_pressure'}], language={'python': {'entry_point': 'kelvin_python_sdk'}, 'type': 'python'}) status=WorkloadStatus(last_seen=datetime.datetime(2024, 6, 1, 7, 38, 18, 148142, tzinfo=datetime.timezone.utc), message='Received', state=<WorkloadStatus.received: 'received'>, warnings=None) title='My CSV Publisher' updated=datetime.datetime(2024, 6, 1, 7, 38, 17, 154694, tzinfo=datetime.timezone.utc) workload_name='doc-demo-csv-publisher' app_name='kelvin-csv-publisher-connector' app_version='1.0.9'