Create a Cumul.io API Plugin with Air Quality Data

Published in

Cumul.io

11 min readOct 7, 2020

Alright, so as I am in the process of learning about Cumul.io myself, what better way to learn about plugins than building one. So, in this post I’ll walk through how I built a plugin with an open API. I’ll go up to how I integrated it in a simple web page. I wanted to build something that I’d actually care about and be interested in looking at so I spent quite some time looking through open APIs. I wanted one that would provide some pollution or climate change data. I’ve ended up using the AirVisual API (if you have any other suggestions for future projects please tell me in the comments!). As I’m using the community version I limited myself to a few major cities around the world and am displaying daily air quality data for them. This is what I ended up with:

Like my previous post on multi-tenancy, I am accompanying this one with open repositories so you can follow the steps exactly if you so wish. First, cumulio-pollution for the plugin itself. Second, cumulio-pollution-landingpage, a simple webpage to integrate the dashboard into. If you don’t use these repos, each of the steps should still be relevant to anyone trying to build a basic plugin from an API for their Cumul.io dashboard.

Why would you build a plugin?

Cumul.io provides great visualizations for your data, but there’s the question of connecting such data to it. One way to accomplish this is by writing your own plugin, for example for an existing API. Plugins work as a bridge, or a connector from an external source of data to Cumul.io.

Without further ado, let me jump into how I got this thing working:

Building the Plugin

First step is to build the plugin itself. I’ve built mine on cumulio-pollution which you can clone if you’d like to follow the next steps exactly. To begin, I followed the steps in a previous tutorial on how to build plugins for Cumul.io dashboards which I found very useful, would recommend.

Here I will be building a basic plugin with no added authorization, since I’m using an open data source. I will connect to the open API from AirVisual as an example. This contains real time and historical weather and air quality data for requested cities, as well as forecast.

Once the plugin is written and added to Cumul.io, it will appear as a new data source when you want to add data to your dashboard in your account.

To start with, you will need the following:

A Cumul.io account (there are free trials available)
An API Key from AirVisual (I used the community edition)
An App Secret from Cumul.io which you get when you create the plugin on your profile

This plugin only needs two endpoints. The /datasets endpoint and the /query endpoint:

The skeleton for our plugin

The /datasets Endpoint

This endpoint gives information about the structure of the data to Cumul.io. In my example, I’m only going to be providing one dataset:

Define the dataset

The code defines 5 columns which the AirVisual API will fill with city names, update time, latitude, longitude and the two air quality indexes (US and China have different measures for air quality).

Note: If you run your code at this point you can have a look on postman to test GET localhost:3030/datasets

The /query Endpoint

At this point, we’ve defined what the dataset will be but there’s no actual data coming in. This is where we use the /query endpoint. In a regular plugin the result of the /query call returns an array of arrays with values that follow the exact same structure as defined in the /datasets endpoint. For example, we could return something like this:

A simple /query endpoint

Now, what the following code does is create this array of arrays returned by the /query endpoint with responses it gets from the AirVisual API. I have picked a few major cities so as to limit the number of requests I make to the API. Feel free to change them as you like. To receive air quality and weather data, the API expects the name of the city, its state and country in the request:

Since there are limits to the number of requests I can make to the API within a given time period, I’ve implemented a cache and am checking whether the fields are over 24 hours old. If they are, I am adding to my cache:

If you’ve cloned and are using the cumulio-pollution repository, you can do the following to run the plugin:

npm install
Create a file called ‘.env’ in the root directory. Here, fill the API_KEY with the key you create with AirVisual API, and CUMULIO_SECRET with the App Secret that is created when you create the plugin on Cumul.io:
API_KEY=XXX CUMULIO_SECRET=XXX
PORT=3030
npm run start or node index.js

To test that the /query endpoint works correctly I used postman (If you do, don’t forget to add the Cumul.io secret to the header):

Now your plugin is running on localhost:3030, but we have to add it to your Cumul.io account and make it accessible to it. For this step, I used ngrok. Running this will give you a https URL that you can use to make your plugin accessible from outside your PC. If you’re using Visual Studio Code you can install the ngrok extension, run it with port number 3030 and copy link (you can also run ngrok on a separate terminal). Once you’ve done that you can now;

Create the plugin on Cumul.io

Now let’s add your plugin to Cumul.io. Go to your profile and Create Plugin on the Plugins tab. Add the URL created in the previous step to Base URL & authentication:

Once it’s been created you will see the plugin on your profile and you can copy the App Secret to the CUMULIO_SECRET field in your .env file:

Building the Dashboard

Now that you have a plugin, when it’s running, you can create a dashboard on Cumul.io and add your plugin as a data source to it. Create a new dashboard and go to add data. You should now see a new plugin in the ‘Your plugins’ tab:

Since we told the plugin to return only one dataset in the /datasets endpoint, you will see the following which you can import:

Now you will see the Air Quality for Select Cities dataset listed on the right as one of your datasets. If you ‘Open in Editor’, you should see the following:

You can now use this dataset to build pretty dashboards to your heart’s content 🙂 I, for example, decided that I would like to display a map, and that I would like the symbol colours to represent the official air quality colouring system. In the rest of this section, I’ll walk through what I did and how I achieved it;

Adding Coordinates and Custom Colours

For those of you who are interested, here is the colour scheme for the US air quality index (which is the one I used):

0–50 Healthy: green
51–100 Moderate: yellow
101–150 Unhealthy for Sensitive Groups: orange
151–200 Unhealthy: red
201–300 Very Unhealthy: purple
301 =< Hazardous: maroon

So I added two new fields to the dataset: Health Status and Coordinates;

Health Status:

On the Dataset Editor, select Add Formula and you can now define a chain of if statements that checks the Air Quality Index (US) column and classifies the health status:

To add colours, in the settings for the Health Status column you’ve just created, you can Edit Hierarchy and define custom colours:

Coordinates:

For this one, I already had latitude and longitude columns in the dataset so I simply had to select one of the two and add coordinates:

Once I had these two columns ready, I added a few charts to my dashboard:

The Map

For this, I’ve used a symbol map and have added the Coordinates column to Geography, the Health Status column to Color and the Air Quality Index (US) column to Measure. For more info on how to add data to your dashboard charts you can visit this webinar on adding charts.

I also want this map and the following bar chart to display today’s figures. So I’ve added a filter that checks the Update time is within the last 1 day:

The Bar Chart

The bar chart is much the same as the map, it displays the days air quality data. So again, I’ve added Air Quality Index (US) as Measure, but this time I’ve added the City to Category and Health Status to Group By. The same filter is also used here to display only the last 1 day. In the Item Settings, you can select Stacked Mode to achieve the same visuals.

The Grouped Line Chart

The line chart aims to provide a timeline for air quality data, so it will build up as you keep your plugin running. In the image above you see mine with 2 days of data. The X-axis is Update time, the Measure again is Air Quality Index (US) and the Group By is City.

The Speedometers

This last one I couldn’t resist. The speedometer was recently introduced to the dashboard editor. So, I selected two cities, Delhi and New York, and am displaying their air quality indexes here as well. In addition to the filter selecting the last 1 day, I’ve also added to each a filter on City and selected the relevant city name:

I do not select a Target for these charts, instead, I set the ranges in the settings:

There is a useful academy article on speedometers that details how to add and customize them.

Integrating the Dashboard

Once the dashboard exists, great, you have a dashboard that you can see when you login to your Cumul.io account. But you may want to have it integrated into another webpage for others to see too. Which is what I wanted to do; for this step, I’ve created cumulio-pollution-landingpage which you can clone and use. To run it:

npm install
Create a file called ‘.env’ in the root directory. Here, fill the CUMULIO_API_KEY and CUMULIO_API_TOKEN fields with the ones from your Cumul.io profile. If you don’t have any yet, you can create them in the API Tokens tab in your profile:
CUMULIO_API_KEY=XXX
CUMULIO_API_TOKEN=XXX
Replace dashboardId in public/js/app.js and server.js with your own dashboard ID which you can find on the dashboard editor in Cumul.io.
npm run start or node index.js

You can visit your webpage on localhost:3000

For this section you can have a look at the Integration API Documentation. As well as a previous blog post that walks through integrating a dashboard. For the full code you can visit the repo but here are some of the key points:

As an initial step, I load the Integration API in the header of my webpage:

Skeleton for index.html

Next up I decided to add a side bar and a main content element where I would display my dashboard, and load the js/app.js script:

The actual dashboard embedding happens in js/app.js. Here I call the addDashboard function. This function loads and renders the dashboard with the specified id in the specified HTML div, which for me is the ‘dashboard-container’:

Adding the dashboard to the relevant container

I wrap the above addDashboard in a loadInsightsPage function that is called when the window loads:

And finally, one last thing to note is server.js where the /authorization endpoint is defined. It uses the Cumul.io Node SDK to request an authorization key & token for the dashboard. The Cumul.io API key and token are first read from the .env file:

Finally, I decided that I’d like to add some extras so as to experiment with filtering from an integrated dashboard. The next section will walk through how I achieved some simple filtering by interacting with the webpage.

Filtering the Dashboard (Extra)

For this final step, I just approached it experimentally so there’s nothing fancy going on. I simply added the list of cities on the sidebar, and filtered the dashboard on that city when clicked:

To achieve this, I first added a list of cities to the sidebar that calls a reloadDashboard function on click:

Then I had to modify the getDashboardAuthorizationToken function to accept an input (the city name) which we can add to /authorization and then refreshData with the filtering added to our authorization options:

And that’s it for coding! The only thing remained to do is to add ‘city’ as a parameter to the dashboard itself on Cumul.io. Now when the reloadDashboard is called on click, we send meta_data to the dashboard with ‘city’ name, and Cumul.io uses the value I send over as its parameter value. To add a parameter, you can go to the FILTERS tab and ‘Create Parameter’ with type Hierarchy[].

I wanted to be able to see all cities on the dashboard editor so added all my cities as default values here. Then I need to use this parameter on the dashboard so on Dashboard Filters ‘Create filter’:

And that’s it 🙂 I now have my own plugin that uses an API, used on a dashboard that I integrate into a webpage and filter from there. Voila ladies and gentlemen: