HomeLearnArticle

A Free REST API for Johns Hopkins University COVID-19 dataset

Published: Jul 28, 2020

  • Atlas
  • Realm
  • JavaScript
  • ...

By Maxime Beugnet

Share

#TL;DR

Here is the REST API Documentation in Postman.

#News

#September 10th, 2020

#Introduction

Recently, we built the MongoDB COVID-19 Open Data project using the dataset from Johns Hopkins University (JHU).

There are two big advantages to using this cluster, rather than directly using JHU's CSV files:

  • It's updated automatically every hour so any update in JHU's repo will be there after a maximum of one hour.
  • You don't need to clean, parse and transform the CSV files, our script does this for you!

The MongoDB Atlas cluster is freely accessible using the user readonly and the password readonly using the connection string:

1
mongodb+srv://readonly:readonly@covid-19.hip2i.mongodb.net/covid19

You can use this cluster to build your application, but what about having a nice and free REST API to access this curated dataset?!

lemur opening big eyes gif

#COVID-19 REST API

You can use the button in the top right corner Run in Postman to directly import these examples in Postman and give them a spin.

Run in Postman button in the Postman documentation website

One important detail to note: I'm logging each IP address calling this REST API and I'm counting the numbers of queries per IP in order to detect abuses. This will help me to take actions against abusive behaviours.

Also, remember that if you are trying to build an application that helps to detect, understand or stop the spread of the COVID-19 virus, we have a FREE MongoDB Atlas credit program that can help you scale and hopefully solve this global pandemic.

#But how did I build this?

Simple and easy, I used the MongoDB Realm 3rd party HTTP service to build my HTTP webhooks.

Each time you call an API, a serverless JavaScript function is executed to fetch your documents. Let's look at the three parts of this function separately, for the Global & US webhook (the most detailed cllection!):

  • First, I log the IP address each time a webhook is called. I'm using the IP address for my _id field which permits me to use an upsert operation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
function log_ip(payload) { const log = context.services.get("pre-prod").db("logs").collection("ip"); let ip = "IP missing"; try { ip = payload.headers["X-Envoy-External-Address"][0]; } catch (error) { console.log("Can't retrieve IP address.") } console.log(ip); log.updateOne({"_id": ip}, {"$inc": {"queries": 1}}, {"upsert": true}) .then( result => { console.log("IP + 1: " + ip); }); }
  • Then I retrieve the query parameters and I build the query that I'm sending to the MongoDB cluster along with the projection and sort options.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
function isPositiveInteger(str) { return ((parseInt(str, 10).toString() == str) && str.indexOf('-') === -1); } exports = function(payload, response) { log_ip(payload); const {uid, country, state, country_iso3, min_date, max_date, hide_fields} = payload.query; const coll = context.services.get("mongodb-atlas").db("covid19").collection("global_and_us"); var query = {}; var project = {}; const sort = {'date': 1}; if (uid !== undefined && isPositiveInteger(uid)) { query.uid = parseInt(uid, 10); } if (country !== undefined) { query.country = country; } if (state !== undefined) { query.state = state; } if (country_iso3 !== undefined) { query.country_iso3 = country_iso3; } if (min_date !== undefined && max_date === undefined) { query.date = {'$gte': new Date(min_date)}; } if (max_date !== undefined && min_date === undefined) { query.date = {'$lte': new Date(max_date)}; } if (min_date !== undefined && max_date !== undefined) { query.date = {'$gte': new Date(min_date), '$lte': new Date(max_date)}; } if (hide_fields !== undefined) { const fields = hide_fields.split(','); for (let i = 0; i < fields.length; i++) { project[fields[i].trim()] = 0 } } console.log('Query: ' + JSON.stringify(query)); console.log('Projection: ' + JSON.stringify(project)); // [...] };
  • Finally, I build the answer with the documents from the cluster and I'm adding a Contact header so you can send us an email if you want to reach out.
1
2
3
4
5
6
7
8
exports = function(payload, response) { // [...] coll.find(query, project).sort(sort).toArray() .then( docs => { response.setBody(JSON.stringify(docs)); response.setHeader("Contact","devrel@mongodb.com"); }); };

Here is the entire JavaScript function if you want to copy & paste it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
function isPositiveInteger(str) { return ((parseInt(str, 10).toString() == str) && str.indexOf('-') === -1); } function log_ip(payload) { const log = context.services.get("pre-prod").db("logs").collection("ip"); let ip = "IP missing"; try { ip = payload.headers["X-Envoy-External-Address"][0]; } catch (error) { console.log("Can't retrieve IP address.") } console.log(ip); log.updateOne({"_id": ip}, {"$inc": {"queries": 1}}, {"upsert": true}) .then( result => { console.log("IP + 1: " + ip); }); } exports = function(payload, response) { log_ip(payload); const {uid, country, state, country_iso3, min_date, max_date, hide_fields} = payload.query; const coll = context.services.get("mongodb-atlas").db("covid19").collection("global_and_us"); var query = {}; var project = {}; const sort = {'date': 1}; if (uid !== undefined && isPositiveInteger(uid)) { query.uid = parseInt(uid, 10); } if (country !== undefined) { query.country = country; } if (state !== undefined) { query.state = state; } if (country_iso3 !== undefined) { query.country_iso3 = country_iso3; } if (min_date !== undefined && max_date === undefined) { query.date = {'$gte': new Date(min_date)}; } if (max_date !== undefined && min_date === undefined) { query.date = {'$lte': new Date(max_date)}; } if (min_date !== undefined && max_date !== undefined) { query.date = {'$gte': new Date(min_date), '$lte': new Date(max_date)}; } if (hide_fields !== undefined) { const fields = hide_fields.split(','); for (let i = 0; i < fields.length; i++) { project[fields[i].trim()] = 0 } } console.log('Query: ' + JSON.stringify(query)); console.log('Projection: ' + JSON.stringify(project)); coll.find(query, project).sort(sort).toArray() .then( docs => { response.setBody(JSON.stringify(docs)); response.setHeader("Contact","devrel@mongodb.com"); }); };

One detail to note: the payload is limited to 1MB per query. If you want to consume more data, I would recommend using the MongoDB cluster directly, as mentioned earlier, or I would filter the output to only the return the fields you really need using the hide_fields parameter. See the documentation for more details.

#Examples

Here are a couple of example of how to run a query.

  • With this one you can retrieve all the metadata which will help you populate the query parameters in your other queries:
1
curl --location --request GET 'https://webhooks.mongodb-stitch.com/api/client/v2.0/app/covid-19-qppza/service/REST-API/incoming_webhook/metadata'
  • The covid19.global_and_us collection is probably the most complete database in this system as it combines all the data from JHU's time series into a single collection. With the following query, you can filter down what you need from this collection:
1
curl --location --request GET 'https://webhooks.mongodb-stitch.com/api/client/v2.0/app/covid-19-qppza/service/REST-API/incoming_webhook/global_and_us?country=Canada&state=Alberta&min_date=2020-04-22T00:00:00.000Z&max_date=2020-04-27T00:00:00.000Z&hide_fields=_id,%20country,%20country_code,%20country_iso2,%20country_iso3,%20loc,%20state'

Again, the REST API documentation in Postman is the place to go to review all the options that are offered to you.

#Wrap Up

I truly hope you will be able to build something amazing with this REST API. Even if it won't save the world from this COVID-19 pandemic, I hope it will be a great source of motivation and training for your next pet project.

Send me a tweet with your project, I will definitely check it out!

If you have questions, please head to our developer community website where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

More from this series

COVID-19
  • Coronavirus Map and Live Data Tracker with MongoDB Charts
  • How to work with Johns Hopkins University COVID-19 Data in MongoDB Atlas
  • A Free REST API for Johns Hopkins University COVID-19 dataset
MongoDB Icon
  • Developer Hub
  • Documentation
  • University
  • Community Forums

© MongoDB, Inc.