Using External Data

A weakness of the Bar Chart webpage we made in the previous section, is that the data values are hard-coded in the data array. In real-life use, we often need the data come from some external source (a downloaded file, or a webservice, or even a sensor device).
To use external data in a web browser, we need to download the data from a web server and then parse it, which converts the data contents into usable JavaScript objects. Fortunately, D3 has several built-in methods that do both of these things in a single function, for several types of data. Here we will use the d3.csv() function. This is used for CSV files, which are plain text files containing comma-separated values, tab-separated values or other arbitrary delimiter-separated values. These tabular formats are popular because they are small and simple, and can be created easily with spreadsheet programs such as Microsoft Excel, or by simple web services. Each line represents a table row, and each row consists of multiple columns separated by tabs. The first line is usually the header row and specifies the column names.

In this exercise we will use a CSV file with data from the municipalities of Overijssel, the province in which the ITC is located. code is a unique identifier, name the name of the municipality and population the number of inhabitants:

data/overijssel_population.csv

code	name	population
GM0141	Almelo	72730
GM0147	Borne	21770
GM0148	Dalfsen	27570
GM0150	Deventer	98580
GM0153	Enschede	158625
...etc...

Do not try to load such files in D3 code from your file--system (e.g. as C:/myfile.csv or similar), but always from an actual web site, e.g. https://gisedu.itc.utwente.nl/... or http://localhost/.... This is the only way to properly test your code!

The reason is that browsers are strict on so-called JavaScript cross-domain security, meaning it won't run JavaScript that is not coming from the same server as the html. And from the browser viewpoint, a file coming from file:/// is from another domain than one from https://gisedu.itc.utwente.nl/, even if these point to the same file...! Also, the security mechanism usually does not allow loading from harddisk (such as C:) anyway. So to do the next parts of the exercises, you need to serve the files through a webserver.
An alternative to using a 'real' server is a local server on your machine, e.g the Apache server installed as http://localhost in most Linux or Mac OSX systems, or a simple Python server (see this webpage for more info).

Using Promises to create asynchronous functions

Loading data introduces a new complexity: downloads are asynchronous. After you call a function,( in this case the d3.csv function), it returns immediately, but the file is still downloading in the background, so there will be no results yet. Therefore we use special functions that contain a so-called promise. After a Javascript promise has been fulfilled, it invokes the then() function, thus anything in the body of that function won't be run until the promise has been fulfilled (or an error is encountered).

Create a copy of the HTML webpage BarChartAxis.html you made before, and call it BarChartOverijssel.html. Replace the line let data = [4, 8, 15, 16, 23, 50]; with the highlighted lines below:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <script src="lib/d3.js"></script>
  <!--CSS styling for bar chart elements:-->
  <style>
    .chart {
      background-color: rgb(245, 245, 245);
      padding: 5px;
    }
    .chart rect {
      fill: steelblue;
    }
    .chart text {
      fill: white;
      font: 9px sans-serif;
      text-anchor: end;
    }
    .axis text {
      fill: black;
      font: 9px sans-serif;
    }
    .axis path, .axis line {
      fill: none;
      stroke: black;
    }
  </style>
</head>
<body>
<svg class="chart"></svg>
<script type="text/javascript">
d3.csv("data/overijssel_population.csv").then(function (data) {
    // in here is the callback function, inside it you have access to the loaded data object
  const svgwidth = 500;
  const barHeight = 20;
  const axisHeight = 30;
  const svgheight = (barHeight * data.length) + axisHeight;
  const xScale = d3.scaleLinear()
          .domain([0, d3.max(data, function (d) { return +d.population })])
          .range([0, svgwidth])
  ;
  const theAxis = d3.axisBottom(xScale);
  let chart = d3.select("svg")
          .attr("width", svgwidth)
          .attr("height", svgheight)
  ;
  let bar = chart.selectAll("g")
          .data(data).enter().append("g")
          .attr("transform", function (d, i) { return "translate(0," + i * barHeight + ")"; })
  ;
  bar.append("rect")
          .attr("width", function (d) { return xScale(+d.population) })
          .attr("height", barHeight - 1)
  ;
  chart.append("g")
          .attr("class", "axis")
          .attr("transform", "translate(0," + (svgheight-barHeight) + ")")
          .call(theAxis);
    } //end of callback function .then()
); //end of d3.csv
</script>
</body>
</html>

d3.csv("data/overijssel_population.csv").then(function (data) {

// in here is the callback function, inside it you have access to the loaded data object

const svgwidth = 500;

const barHeight = 20;

And after the last line of the code (just before the </script> statement), add the proper closing of the callback function and d3.csv method:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <script src="lib/d3.js"></script>
  <!--CSS styling for bar chart elements:-->
  <style>
    .chart {
      background-color: rgb(245, 245, 245);
      padding: 5px;
    }
    .chart rect {
      fill: steelblue;
    }
    .chart text {
      fill: white;
      font: 9px sans-serif;
      text-anchor: end;
    }
    .axis text {
      fill: black;
      font: 9px sans-serif;
    }
    .axis path, .axis line {
      fill: none;
      stroke: black;
    }
  </style>
</head>
<body>
<svg class="chart"></svg>
<script type="text/javascript">
d3.csv("data/overijssel_population.csv").then(function (data) {
    // in here is the callback function, inside it you have access to the loaded data object
  const svgwidth = 500;
  const barHeight = 20;
  const axisHeight = 30;
  const svgheight = (barHeight * data.length) + axisHeight;
  const xScale = d3.scaleLinear()
          .domain([0, d3.max(data, function (d) { return +d.population })])
          .range([0, svgwidth])
  ;
  const theAxis = d3.axisBottom(xScale);
  let chart = d3.select("svg")
          .attr("width", svgwidth)
          .attr("height", svgheight)
  ;
  let bar = chart.selectAll("g")
          .data(data).enter().append("g")
          .attr("transform", function (d, i) { return "translate(0," + i * barHeight + ")"; })
  ;
  bar.append("rect")
          .attr("width", function (d) { return xScale(+d.population) })
          .attr("height", barHeight - 1)
  ;
  chart.append("g")
          .attr("class", "axis")
          .attr("transform", "translate(0," + (svgheight-barHeight) + ")")
          .call(theAxis);
    } //end of callback function .then()
); //end of d3.csv
</script>
</body>
</html>

.call(theAxis);

} //end of callback function .then()

); //end of d3.csv

</script>

If you now run the code in your browser, you will get no result, and several error messages in the Javascript console, such as: Invalid value for <rect> attribute width = 'NaN'.
So, what is wrong...? Now that our dataset contains more than just a simple array of numbers, each data instance (or 'row') is no longer just a simple value, but instead it is a complex object.

To actually see what the data object is, add the line console.log(data) as the first line of your callback function, and load the webpage with the Javascript console opened.

The data object in the console (in this case in the Chrome browser).

If you study the output, you can see the data object is an array (delimited by []) of several objects (each delimited by {}); in each of these objects various key-value pairs represent the objects' fields: code is the id for the municipality (gemeente), name is its name and population is the number of inhabitants.
Thus, if we want to read a data value for number of inhabitants, we must refer to the field as d.population, meaning "the object identified by key population within the data instance d".

Another complication is caused by the fact that Javascript has no strict typing. d3.csv assumes all the field values to be strings. You can convert fields to a number by applying the unary plus operator (+). So, whereas before we could simply pass the scale xScale(d) to compute the width of the bar, we must now specify the correct data value to the scale as such: xScale(+d.population).

And likewise, computing the maximum value from our dataset in our xScale function is no longer simple. Now we must pass a second parameter to d3.max(), an anonymous function that tells it which data variable to use to determine the maximum: d3.max(data, function (d) { return +d.population })

Repair the use of the d value to match the data-structure, as explained above. Test again to see if you now get the correct bar chart for all Overijssel municipalities, as shown in the figure below.
Note that as an optional challenge, this figure also shows we have added "text" elements to each bar that show the name of each municipality...

The solution for the optional challenge will appear below at a later time.

Barchart using loaded CSV data.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <script src="lib/d3.js"></script>
  <!--CSS styling for bar chart elements:-->
  <style>
    .chart {
      background-color: rgb(245, 245, 245);
      padding: 5px;
    }
    .chart rect {
      fill: steelblue;
    }
    .chart text {
      fill: white;
      font: 9px sans-serif;
      text-anchor: end;
    }
    .axis text {
      fill: black;
      font: 9px sans-serif;
    }
    .axis path, .axis line {
      fill: none;
      stroke: black;
    }
  </style>
</head>
<body>
<svg class="chart"></svg>
<script type="text/javascript">
d3.csv("data/overijssel_population.csv").then(function (data) {
      // in here is the callback function, inside it you have access to the loaded data object
      const svgwidth = 500;
      const barHeight = 20;
      const axisHeight = 30;
      const svgheight = (barHeight * data.length) + axisHeight;
      const xScale = d3.scaleLinear()
              .domain([0, d3.max(data, function (d) { return +d.population })])
              .range([0, svgwidth])
      ;
      const theAxis = d3.axisBottom(xScale);
      let chart = d3.select("svg")
              .attr("width", svgwidth)
              .attr("height", svgheight)
      ;
      let bar = chart.selectAll("g")
              .data(data).enter().append("g")
              .attr("transform", function (d, i) { return "translate(0," + i * barHeight + ")"; })
      ;
      bar.append("rect")
              .attr("width", function (d) { return xScale(+d.population) })
              .attr("height", barHeight - 1)
      ;
      chart.append("g")
              .attr("class", "axis")
              .attr("transform", "translate(0," + (svgheight-barHeight) + ")")
              .call(theAxis);
    // this is the OPTIONAL bit with the label texts:
      bar.append("text")
              .attr("x", function(d) { return xScale(+d.population) - 3; })
              .attr("y", barHeight / 2)
              .attr("dy", ".35em")
              .text(function(d) { return d.name; })
      ;
    } //end of callback function .then()
); //end of d3.csv
</script>
</body>
</html>

<!DOCTYPE html>

<head>

<style>

.chart {

background-color: rgb(245, 245, 245);

padding: 5px;

}

.chart rect {

fill: steelblue;

}

.chart text {

fill: white;

font: 9px sans-serif;

text-anchor: end;

}

.axis text {

fill: black;

font: 9px sans-serif;

}

.axis path, .axis line {

fill: none;

stroke: black;

}

</style>

</head>

<body>

d3.csv("data/overijssel_population.csv").then(function (data) {

// in here is the callback function, inside it you have access to the loaded data object

const svgwidth = 500;

const barHeight = 20;

const axisHeight = 30;

const svgheight = (barHeight * data.length) + axisHeight;

const xScale = d3.scaleLinear()

.domain([0, d3.max(data, function (d) { return +d.population })])

.range([0, svgwidth])

;

const theAxis = d3.axisBottom(xScale);

let chart = d3.select("svg")

.attr("width", svgwidth)

.attr("height", svgheight)

;

let bar = chart.selectAll("g")

.data(data).enter().append("g")

.attr("transform", function (d, i) { return "translate(0," + i * barHeight + ")"; })

;

bar.append("rect")

.attr("width", function (d) { return xScale(+d.population) })

.attr("height", barHeight - 1)

;

chart.append("g")

.attr("class", "axis")

.attr("transform", "translate(0," + (svgheight-barHeight) + ")")

.call(theAxis);

// this is the OPTIONAL bit with the label texts:

bar.append("text")

.attr("x", function(d) { return xScale(+d.population) - 3; })

.attr("y", barHeight / 2)

.attr("dy", ".35em")

.text(function(d) { return d.name; })

;

} //end of callback function .then()

); //end of d3.csv

</script>

</body>

</html>

Using the D3 JavaScript Library
for Interactive Graphs and Maps

Using External Data

data/overijssel_population.csv

Using Promises to create asynchronous functions

Exercise Sections

Page Content