Do you have a database full of JSON objects crying out for visual analysis? In this talk we will describe the different visualization concepts that apply to objects and showcase open source technologies that provide that capability (for free) and give you the basic tools you need to get started. Topics covered will include: - A quick overview of JSON objects (properties, ways to reference other objects, etc.); - Simple examples of working with JSON via JavaScript; - How to turn JSON documents into visual feasts using tools like the GoogleMaps API and D3.js library.
MongoDB .local Paris 2020: Les bonnes pratiques pour travailler avec les donn...
Visualizing MongoDB in Objects in Concept and Practice
1. Washington DC 2013
Visualizing MongoDB Objects
in Concept and Practice
https://github.com/cvitter/ikanow.mongodc2013.presentation
2. Introduction
• Do you have a MongoDB database full of BSON
documents crying out for visual analysis?
3. Agenda
• Visualizing Objects vs. Records
• Getting Your Data from MongoDB
• JSON 101
• Free & Open Source Visualization Libraries
– Google Maps JavaScript API
– D3.js
• Other Visualization Libraries You Should Know
• Questions?
4. Objects vs. Records
• Document oriented data stores support dynamic and
complex schemas vs. the simple, static structures in
RDBMs, e.g.:
5. Is There Really a Difference?
• Yes and No
• Yes
• Obviously, document oriented formats are different from
record oriented formats;
• Few common visualizations tools designed for
traditional record based formats support document
based formats
• No
• The basic visualizations are the same even if the data
format feeding them are different
6. Getting your Data from MongoDB
• mongoexport
Export data from MongoDB as JSON or CSV
> mongoexport --db dbname --collection collectionname
--out file.json
• MongoDB’s Simple REST Interface
• Read only access to data
• Start your server with the --rest option
• Simple queries:
http://127.0.0.1:28017/databaseName/collectionName/
• Other options:
• DrowsyDromedary, MongoDB Rest, etc.
7. JSON 101
• JSON (JavaScript Object Notation) documents
are built using two types of common data structures:
• Name/value pairs (objects) and;
{ “string” : value }
• Ordered lists of values (arrays).
{ “string” : [value, value, value] }
• JSON is a subset of the object literal notation
of JavaScript so:
var jsonObj = {"numbers" : [1, 2, 3] };
var numberOne = jsonObj.numbers[0];
var numberTwo = jsonObj.numbers[1];
• Converting the string representation of JSON is as easy as:
var jsonObj = JSON.parse("{"numbers":[1,2,3]}");
8. A Word About the Sample Code
• All of the code used in this presentation is available online
via GitHub at:
https://github.com/cvitter/ikanow.mongodc2013.presentation
• The project includes code from the following Open Source
Projects:
• Bootstrap: http://twitter.github.com/bootstrap/
• JQuery: http://jquery.com/
• D3.js: http://d3js.org/
• The data samples used are simple JSON files read in using
JQuery’a .ajax method
9. Google Maps JavaScript API
• Requirements:
• Get an API Key (its free*)
• Have an Internet Connection
• Build a simple example that:
1. Creates a place holder DIV for the map object
2. Loads the map script
3. Initializes the map
4. Draws markers on the map
* Up to 25,000 map loads per day for commercial applications.
10. Creating the Map
• Use a DIV to create a placeholder for the map:
<div id="map_canvas" style="height:450px; width:870px;"></div>
• Load the map script:
function loadScript() {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "http://maps.googleapis.com/maps/api/js?
key=YOUR_API_KEY&sensor=TRUE_OR_FALSE&callback=initialize";
document.body.appendChild(script);
}
• Initialize the map:
var mapOptions = {
zoom: 11,
center: new google.maps.LatLng( 38.8964, -77.0262 ),
mapTypeId: google.maps.MapTypeId.ROADMAP
};
map = new google.maps.Map(document.getElementById('map_canvas'),
mapOptions);
11. Drawing Markers on the Map
• Sample documents:
{...},
{
search_field: "adams morgan",
country: "United States",
country_code: "US",
region: "District of Columbia",
latitude: "38.9213889",
longitude: "-77.0425000”}
},
{...},
• Add markers to the map:
for (var i=0; i < locations.length; i++) {
var marker = new google.maps.Marker({
map: map,
position: new google.maps.LatLng(
locations[i].latitude , locations[i].longitude ),
title: convertToTitleCase(locations[i].search_field)
});
}
15. D3.js
• D3.js (d3js.org) is:
“a JavaScript library for manipulating documents based on
data”
• Convert data into visualizations in the following formats:
HTML, CSS, SVG
• Download the library or include it via:
<script src="http://d3js.org/d3.v3.min.js"></script>
16. The (Very) Basics
• Select the DIV to write the SVG image to:
var chart = d3.select("#d3_canvas").append("svg")
.attr("class", "chart")
.attr("width", chartWidth)
.attr("height", chartHeight);
• Draw the bars (rectangles):
chart.selectAll("rect")
.data(inputData)
.enter().append("rect")
.attr("y", function(d, i) {return i * (lineHeight + 3); })
.attr("width", function(d, i)
{ return chartWidth * (d.chance_of_rain * 0.01); })
.attr("height", function(d) return lineHeight; });
17. Adding Rules
• Making things scale on the chart:
var x = d3.scale.linear()
.domain([0, 100])
.range([0, chartWidth]);
• Drawing the rules:
chart.selectAll("line")
.data(x.ticks(10))
.enter().append("line")
.attr("x1", x)
.attr("x2", x)
.attr("y1", 0)
.attr("y2", chartHeight)
.style("stroke", "#ccc");
18. Adding Text
• Labeling the X and Y axes:
chart.selectAll(".rule")
.data(x.ticks(10))
.enter().append("text")
.attr("class", "rule")
.attr("x", x)
.attr("y", 0)
.attr("dy", -3)
.attr("text-anchor", "middle")
.text(String);
chart.selectAll("text")
.data(inputData)
.enter().append("text")
.attr("x", 0)
.attr("y", function(d, i) {
return i * (lineHeight + 3) + (lineHeight / 2); })
.attr("dx", -3) // padding-right
.attr("dy", ".35em") // vertical-align: middle
.attr("text-anchor", "end") // text-align: right
.text(function(d) { return d.date; });
20. Treemaps, Bubble Charts, and More
• D3.js can accept an array of values, objects, or a function
that returns an array of values
• Some of the D3.js visualizations allow you to pass data in a
wide variety of formats, others expect a specific format
• The Treemap and Bubble Chart samples use a really
simple JSON tree structure representing multi-level parent
child relationships
22. Treemaps Layout
• D3.js features different layout packs that help you build
complex charts including .treemap
• In this example the layout pack is creating a properly sized
div for each node in our JSON file as opposed to creating
an SVG image
var treemap = d3.layout.treemap()
.size([width, height])
.sticky(true)
.value(function(d) { return d.size; });
var node = div.datum(data).selectAll(".node")
.data(treemap.nodes)
.enter().append("div")
.attr("class", "node")
.call(position)
.style("background", function(d) {
return d.children ? color(d.name) : null; })
.text(function(d) { return d.children ? null : d.name; });
25. More Cool Visualization Libraries
• NVD3 (nvd3.org)
Re-usable charts and chart components for d3.js
• Raphael JS (raphaeljs.com)
JavaScript library designed to simplify working with vector graphics and
build data visualizations
• TimelineJS (timeline.verite.co)
“Beautifully crafted timelines that are easy, and intuitive to use.”
26. Reminder: Get the Example Code
• All of the sample code used in this presentation is available
online via GitHub at:
https://github.com/cvitter/ikanow.mongod
c2013.presentation
Do you have a MongoDB database full of BSON documents crying out for visual analysis?My name is Craig Vitter and I am engineer with IKANOW the company behind the Infinit.e Open Source document analysis platform built on top of MongoDB.Over the past two plus years I have been working with MongoDB I have fielded quite a few questions about how to create visualizations from JSON objects and the kinds open source tools and libraries that are available for visualizationThe goal of this presentation is to help answer some of those questions and get people pointed in this right direction towards visualizing their data.
In this presentation I hope to cover the following topics:Visualizing Objects vs. Visualizing Records – is there a difference?Getting data from MongoDB so you can visualize itA really brief primer on JSON for anyone new to itAnd an introduction to building visualizations with:Google Maps Javascript APID3.js
The primary difference between objects (like MongoDB documents) and records is that objects can consist of dynamic and highly complex schemas versus the simple, static structures found in databases and delimited file formats like CSV files.
Is there really a difference between objects and documents when it comes to visualizations? The short answer is yes and no.YesObviously the data structures are different and few commonly available visualization tools support document based formats out of the box (fortunately there are free and open source libraries available)NoThe basics of visualization are the same regardless of the structure of your data. People still want to see points plotted on maps, graphs, charts, etc.
In order to visualize your data you need to get it out of MongoDB. This presentation isn’t really about the nuts and bolts of MongoDB but I want to briefly mention some of the mechanism you can use as a starting point for creating visualizations:Themongoexport tools is probably the easiest way to get started as it allows you to query your database and do a simple dump of data to a JSON fileMongoDB has a simple, read only REST interface enabled via the –rest option on the mongod command lineFor more complex access you will want to check out one of the more capable REST interface like DrowsyDromedary
JSON, JavaScript Object Notation, is basically the lingua franca of Open Source document analysis these days.JSON documents are simply collections of name/value pairs and ordered lists of values.
It isn’t really possible to teach you everything you need to know about visualizing the data in your MongoDB instance in 40 minutes all of the code I am showing you here is available online via GitHub.In the sample code I have included Bootstrap from Twitter to simplify page layout, JQuery to help access my JSON data via a simple REST client, and D3.js to create visualizations. My hope is that if this presentation gets you interested in using the tools discussed that the sample code will shrink the learning curve.
The Google Maps JavaScript API is a great example of a free (up to 25000 map loads a day) visualization tool that you can use with either traditional structured data sources or document oriented sources like JSON.To use the Google Maps API you need to get an API key and have an Internet connection in order to access the libraries.The first example I am going to cover demonstrates the basic creation of a map via the Google Maps API and drawing markers on the map using a simple JSON document.Let’s take a quick look at the JSON data we are going to use to create our map. This JSON was retrieved from my local MongoDB server using the Simple HTTP REST Interface mention earlier and stored locally in the Web project to simply the process of getting up and running with the sample code.
In all of the examples we are going to look at I have created a DIV in my web page that will act as a placeholder for the visualization that we are going to create. Once we have imported our data and created our visualization the visualization with get written to the DIV.The next step I am showing is how we are going to access the Google Maps JavaScript API by loading the library and appending it to our web page dynamicallyOnce the library is loaded we can create a map object and setup our basic map options including the type of map, the latitude and longitude the map should center on, and the level of zoom.
Adding markers to a map via the JavaScript API is super easy. At a minimum you need two data points: latitude and longitude. For this example we are iterating over a list of 25 objects and adding a marker for each latitude and longitude pair to our map.Note that we are also setting a title for each marker so a tool tip with the name of the location shows up when you hover over each marker.
Demonstrate functionality of the page and show the source code.
Once you have figured out the basics of working with the JavaScript API it is relatively trivial to start adding features to your maps. For example, if you have more than a few dozen markers to add to a map you might wish to display your data in the form of a heat map. For this example we will create a heat map that helps us visualize 1000 data points on our map. Our data file is identical in format to the first example but covers 1000 locations in Virginia as opposed to 25 in Washington DC.Creating a heat map only requires a few basic changes to the code we saw in the first example:- When loading the library we need to add the Visualization library to our request via the libraries parameter (shown in bold in this slide)The map options are set identically to our previous exampleWe create an array of objects that contain latitude and longitude pairs and optionally a weight value thenWe add a heat map layer on top of our map
After you have the basics of adding markers and working with visualization layers like heat maps moving on to more advanced design elements like custom icons, animations, and more.
On the open source side D3.js is a comprehensive JavaScript library designed to make it possible to manipulate documents based on complex data structures.D3.js makes it possible to create a diversity of visualizations in the following format: HTML, CSS, and SVG and getting started with the library is relatively easy.The first step in using D3.js is to download the library from the d3js.org web site or link to it from within your code as show in this slide.
Much like the Google Maps JavaScript API getting started with D3.js is quite easy for someone with a basic grasp of JavaScript development. For the purposes of this presentation I am going to show two really simple examples of creating visualizations with the library. In the first example we will cover the very basics of the D3.js by creating an old school bar chart using simple weather forecast data.Just like the Google Maps example we are going to use a DIV in our HTML to act as a placeholder. In this case we create an object named chart, select our DIV using D3’s select method and allow D3 to handle the process of drawing a Scalable Vector Graphic to that space using the append method.D3.js makes it really easy to draw shapes based on your data sets. In this case we pass in a JSON data via the .data property and then D3 iterates over each record in the JSON and creates a rectangle starting at the specified point on our y axis, and of a specified width and height.
D3.js does a lot of the heavy lifting required to build data driven visualizations like calculating how to draw rules with the appropriate spacing based on data like:.domain – the range of values being display and.range – the size of the image over which those values need to be displayedIn this example we draw a rule for every 10% scaled across the width of our bar chart.
The final step of creating a basic bar chart is labeling the X and Y axes, the hardest part of which is doing the math to position the text properly. D3.js has functionality to help you calculate where to place the text or you can of course do it yourself as demonstrated in the second block of code where I did some very, very basic math to place the dates to the left of and aligned in the middle vertically to their respective bars on the graph.Note how D3.js iterates over the data passed into to the selectall methods in the data parameter to generate the individual text values allowing a huge amount of flexibility in creating your labels.
We have seen with the first example how D3.js supports simple JSON structures. If that was all it could do it wouldn’t be very impressive. Fortunately D3.js supports complex documents.Some of the D3.js visualizations allow you to pass data in a wide variety of formats, others expect a specific format like the Treemap and Bubble Chart samples that take a simple JSON tree structure as shown in the next two examples.
D3.js makes it easy to create complex chart layout through a series of layout packs like the one that created the treemap in this example. In this case the treemap is a series of properly size and placed divs instead an SVG img.Each node, or div, is created by iterating over the nodes in the JSON file, calculating its size, position, and text value to write when drawing the div in our canvas.
This bubble chart uses the exact same data file we used to create our treemap but the layout is generated using the .pack layout module which creates SVG images as opposed to div based layouts.
The three examples included in this presentation are just a small sampling of out of the box functionality that comes with the D3.js library.
There are quite a few useful open source visualization libraries available on the Internet. A few you might also want to look at include:NVD3 which sits on top of D3.js and is designed to make building some common charts easier than with D3.js aloneRaphael JS is a JavaScript library that is designed to simplify creating vector graphics and can be used to build visualizations much the same way you can use D3.jsTimelineJS is very cool library designed to help you create timelines integrating a variety of data sources including Twitter, Flickr, YouTube, and JSON of course
Just a reminder, if you are interested in trying out the Google Maps API or the D3.js library examples you can download the project from Github.