Nishadh KA

Plot Aerocet sample D3JS

2014-11-16

####Plot aerocet data with D3 JS scrip####

  1. Based on this and its source code.
  2. One index.html file combines the code for the meteor web application. Web map marker codes are in the file JS drawMap.js.
  3. The index.html was edited to remove the CHART in line 14 to 19 contains
<div id="year-chart" class="chart display">
            <div class="title">Year of Impact</div>
        </div>

        <div id="mass-chart" class="chart display">
            <div class="title">Mass (g)</div>
        </div>

  1. IN drawMap.js, following lines were removed

    function renderAll(){
    chart.each(render);

    var filterArray = cartoDbIds.all();
    filterArray.forEach(function(d, i){
        if (d.value != lastFilterArray[i]){
            lastFilterArray[i] = d.value;
            d3.select("#id" + d.key).transition().duration(500)
                    .attr("r", d.value == 1 ? 2*metorScale(metors[i].mass) : 0)
                .transition().delay(550).duration(500)
                    .attr("r", d.value == 1 ? metorScale(metors[i].mass) : 0);

        }
    })

    d3.select("#active").text(all.value());
}

  2. Also removed line from 129 tp 159 related with cahrts

  3. To make python to print \ backlash charecter based on this

  4. Instead of json, the meteors web app is reading a csv file and getting data for marker position such as the lat and long. The drawMap.js and the l function queue() contains the codes for functionalities. Such as referring the csv file location, data parsing and drawing all the related process, such as point-making in d3 (by lat-long parsing) and giving other info in the event of marker click and pop up box.

  5. Made different attempts such as trying different maps, shapefiles for cbe map visualization in d3 canvas

  6. The first attempt is with recently created cbe_webmap.shp with all the wards and village converted into topojson by the following command

ogr2ogr -f GeoJSON cbecity.json cbewebmapcity.shp topojson -o cbecitytopo.json cbe_city.json

However, this was visualized in canvas as haphazard rectangular and scribbled image, for a surprise. 1. So it is doubted the problem with shapefile or conversion process. The letsmake map tutorial shapefile converted into toposjon with the following command

ogr2ogr -f GeoJSON world.json ne10madmin0map_subunits.shp topojson -o worldT.json world.json

Replaced the meteors web app toposjon with just created topojson and edited the drawmap.js care this change. The result was perfect visualization of the added topojson in D3 canvas. Indicating the problem is related to shapefile. 1. Using QGIS, a new shapefile was created from the file ne10madmin0mapsubunits.shp. This file named as testwebmapA1.shp was converted into topjson with command

ogr2ogr -f GeoJSON testA1.json test_webmapA1.shp
topojson -o testtopoA1.json testA1.json

This file was again viewed correctly in the canvas. Reassures that the problem is indeed with shapefile. 1. Suspected that the problem might be of wrong projection. Opened the world file and the cbe_webmap.shp and created a new file tracing it. This was testtopoA3.json made into viewed in the canvas of web app but having the problem of zooming. 1. Though changing the values in .scale of var proj and .scaleExtent of var zoom, the zoom is working for a single time and then gets stuck. 1. Created a new shapefile as of past test for solving the problem test_webmapA2.shp and converted into json by

ogr2ogr -f GeoJSON testwebA2.json test_webmapA2.shp topojson -o testtopoA2.json testwebA2.json

Which gets a view in canvas with zoom and drag functionality. However, the zoom is still stuck to a maximum limit. The link gives more information on D3 zoom functionality.. It is found that the command .scaleExtent([height*.33, 4 * height]) is indicating minimum and maximum zooming functionality. So changing that make more control of map viewing and thus solved the problem of cbe map view in d3 canvas. 1. The problem related to zoom is simple arithmetic. To get the maximum zoom value, it has to be above the .scale(180000), so if the height is 500, the scale extent has to be 180000/500*height or above. 1. With this solution, the digitizing cbe map testtopoA3.json in the web app perfectly gets viewed. Proving once again, one day of reading documentation is better than one week of googling. 1. So the cbe_map overlay with world.shp file was traced with keeping the snapping option to both the shapefiles. Not giving this 100 pixels units for snapping option eiher of it didn’t made snapping option true with turn on enabling topological editing and enable snapping on an intersection. 1. This file was converted into topojson by the following command

ogr2ogr -f GeoJSON cbe_webmap.json cbe_webmap_D3A1.shp
topojson -o cbe_webmapT.json cbe_webmap.json
  1. The file got correctly visualized in d3 canvas by giving .scale(150000), .center(76.9715,11.0142), .rotate([0,0]); and .scaleExtent([height*360, 560 * height]).
  2. Further to this the data source, the file fell.csv file was edited to consider the Aerocet data. Beforehand the Aerocet data was made shorten by removing the unnecessary column by following commands. ``` import os ad=[] for file in os.listdir(‘/home/AerocetDATA/Octsample/‘): if file.endswith(“M.csv”): ad.append(file)

for file in os.listdir(‘/home/AerocetDATA/Octsample/‘): if file.endswith(“M_N.csv”): ad.append(file)

for i in ad: db=pd.readcsv(i) db1=db[[‘Time’,‘long’,‘lat’,‘PM2.5(ug/m3)’,‘PM10(ug/m3)’]] db1.tocsv(‘web_‘+i)

There was need of minor change in the function ready(),
```.attr("r",     function(d){return metorScale(d.mass);})``` where it takes up the mass_g column of fell.csv for scaling the point marker size.

###Adding the functionality of drop down to choose date###

1. This functionality is required to visualize the multiple date Aerocet readings saved in different csv file. For this made a first attempt ```webmap_A1``` based on [this](http://fiddle.jshell.net/7KJC7/2/) and second attempt ``was`webmap_A2``` made with [this](http://fiddle.jshell.net/7KJC7/2/) sample code.
1. The attempted functionality was based on drop down select to change the CSV file quired in queue function (the lines 51-55 of drawMap.js) are as follows

queue() .defer(d3.json, “worldTopo.json”) .defer(d3.csv, csv) .defer(d3.json, “pics.json”) .await(ready);

Getting the csv variable value in attempt ```webmap_A1``` by

var csv = function() { var selectedValue = d3.event.target.value; console.log(selectedValue) } d3.select(“#select-list”).on(“change”, csv); .defer(d3.csv, selectedValue)


Attempt got ended with the error of no CSV file. So made the second attempt ```webmap_A2``` by 

d3.select('#select-list')
  .on('change', function() {
    var csv = eval(d3.select(this).property('value'));
    console.log(csv)
});  
this attempt was ended with getting of file name with out csv extension based on [this](http://stackoverflow.com/questions/24193593/d3-how-to-change-dataset-based-on-drop-down-box-selection), such as ```web_08102014M.csv```

1. Then it is found that changing csv file is way around method for the drop down requirement. There is a native d3 method for this requirement named as object constancy. Examples are [this](http://bl.ocks.org/nsonnad/4175202), [this1](https://github.com/mbostock/bost.ocks.org/blob/gh-pages/mike/constancy/index.html) and [this2](http://www.delimited.io/blog/2013/11/8/object-constancy-in-d3)
1. Based on [this](http://bl.ocks.org/nsonnad/4175202) example the csv data filtering by column/ time is made as follows

rawMetors = csv; var nested = d3.nest() .key(function(d) { return d.Time; }) .map(rawMetors)

    var series = menu.property("value");

    // only retrieve data from the selected series, using the nest we just created
    data = nested[series];
    console.log(JSON.stringify(data));



metors = [];
data.forEach(function(d){
    d.mass = +d.PM25;
    d.year = +d.year;
    d.id = +d.cartodb_id;
    metors.push(d);
});

1. With the bove code it can filter csv file with time, which is referen cein drop down element
1. So now to create data for the map, stacking all the csv fileinto one csv file by below code

import os ad=[] for file in os.listdir(‘/home/swl-sacon-dst/Documents/GISE2013/LAB/AerocetDATA/Oct_sample/‘): if file.startswith(“web”): ad.append(file)

#concatenate pandas dataframe and then convertinginto single csv file for i in ad: db0=pd.DataFrame() db=pd.readcsv(i) db2=pd.concat([db0, ], axis=0) db2.tocsv(‘web_aerocetdata.csv’) #this is not working, lets try based on this

frame = pd.DataFrame() list = [] for files in db: df = pd.readcsv(join(directory,files),indexcol=None, header=0) list.append(df) frame = pd.concat(list) #this also not working #based on this for i in ad d = concat([readcsv(f, indexcol=0, header=None, axis=1) for f in ad], keys=files) for i in ad d = pd.concat([(pd.readcsv(i, indexcol=0, header=0).T) for f in i], keys=i) #this also not working neds in some error

# ad1=[] for i in ad: db=pd.read_csv(i) db2=df23=pd.concat([Series(row[1], row[0].split(‘,’)) for , row in db.iterrows()]).resetindex() ad1.append(db2)
#not working # names = pd.DataFrame() for i in ad: frame = pd.readcsv(i) names = pd.concat(names, frame, ignoreindex=True) #Not working # db = pd.DataFrame() for i in ad: frame = pd.readcsv(i) db = db.append(frame, ignoreindex=True) #thjis is working db2=db[[‘Time’,‘lat’,‘long’,‘PM10(ug/m3)’,‘PM2.5(ug/m3)’]] db3=pd.DataFrame(db2.Time.str.split(” “).tolist()) T1 = pd.merge(db2,db3, on=db2.index, how=‘outer’)1. The

  1. with the above file, a new column was added to give the size of the pointer in the map as a circle. The title of the column also has to be changed to plot correctly the map. Which makes the points to be plotted in the map but with a problem. As the points selected by drop down is not getting removed, and the next selection is overlaid with old selection points.
  2. The change function in BRICS app can used to address the problem. However, the difficulty persists in the json queue.
  3. The json queue also not usable for the current case. The extensive data pipeline is to be the underlying problem with this rendering of whole objects in canvas.
  4. Based on this example there is an alternative using the slider and changing/filtering the csv file.
  5. The solution is based on making the d3.js slider as an object/number generator. In which each slider operation gives a new number, and so the date in the aerocet data gets filtered.
  6. To edit the web aerocet file for changing the date formate

import pandas as pd db=pd.readcsv(‘webaerocst.csv’) db[‘Date2’]=pd.to_datetime(db.Date,format=‘%Y-%m-%d’) db[‘Date3’] = db[‘Date2’].map(lambda x: x.strftime(‘%Y%m%d’)) #to get unique values out of pandas column for dates db22=pd.Series(db.Date3.values.ravel()).unique()

  1. The data filtering in slider can coded as per below

$(“#scale”).slider({ min: 20141008, // there are 6 days of sampling 0 is all events max: 20141023, value: 20141008, //default slider value step: 1, // step is the allow increments the slider can move. 1 = one day slide: function(event, ui) {

var months = [0,20141008, 20141021, 20141022, 20141023, 20141017,20141014, 20141020, 20141011];

// handle case where slider at zero = show all sampling if(ui.value == 0) { // highlight all sampling points d3.selectAll(‘.samplingpoints’) .style(“display”,“block”);

// change the slider text to appropriate date d3.select(“#slidertext”) .text(‘Attacks During 01/2004 - 12/2009’);

}else{

// dim all explosion points d3.selectAll(‘.samplingpoints’) .style(“display”,“none”);

// get and format current slider value $tempvalue = “+months[ui.value]; //look up which month the slider value corresponds to and convert to a string for printing $tempvalueclass = ‘._‘+$tempvalue; //take month and convert to a class

// highlight all samplingpoints in the selected month by class d3.selectAll($tempvalueclass) .style(“display”,“block”);

// change the slider text to appropriate date d3.select(“#slidertext”) .text(‘sampling During ‘+$tempvalue.substring(4,6)+’/‘+$tempvalue.substring(0,4));

}//endelse

} });

  1. Usgae of this example can’t be reproduable with the current web app based on metero web app, it seems it is due to different libraries used and projection system used in these two d3.js based web apps. So this attempt named as meteors-master_A5 is left as such due to this lock.
  2. A new effort by adding slider in the native meteors app and tried with log working. The log would give clues to learn the needed functionality.
  3. Changing of libraries in the native meteors app for slider is not working. So tried with changing native slider app with libraries of meteors and it is working. So copied meteors app content in it.
  4. Below code defines various functions needed to visualize topojson.

var w = 1200; var h = 500;

var xy = d3.geo.mercator() .center([76.9715,11.0100]) .scale(145000) .rotate([0,0]);

var path = d3.geo.path() .projection(xy);

var zoom = d3.behavior.zoom() .translate(xy.translate()) .scale(xy.scale()) .scaleExtent([h*290, 645 * h]) .on(“zoom”, zoom);

var svg = d3.select(“#graph”).append(“svg”) .attr(“width”, w) .attr(“height”, h) .call(zoom);

function zoom() { xy.translate(d3.event.translate).scale(d3.event.scale); svg.selectAll(“path”).attr(“d”, path); circles .attr(“cx”, function(d){return xy([d.long, d.lat])[0];}) .attr(“cy”, function(d){return xy([d.long, d.lat])[1];}); }

var borders = svg.append(“g”); var states = svg.append(“svg:g”) .attr(“id”, “states”);

var circles = svg.append(“svg:g”) .attr(“id”, “circles”);

var labels = svg.append(“svg:g”) .attr(“id”, “labels”);

d3.json(“cbewebmapT.json”, function(error, topology) { borders.selectAll(“path”) .data(topojson.object(topology, topology.objects.cbewebmap) .geometries) .enter() .append(“path”) .attr(“d”, path) .attr(“class”, “border”) });

  1. Moreover, the css also has to be changed. Otherwise, the map gets drawn with black fill color. The path has to be defined in css as it is important in the case of topojson.
  2. Slide change for data change
  3. reflect the slider change in data query of canvas drawing element.

  4. It is complicated to reuse js scripts already written. Basics of what; line what behavior it exhibits has to be known to change a single line in it. So list what are all required in the app. One try with slider app, another with meteros app ###with slider app

    1. plot cbe map(it is done in meteors-master_A7 with slider app)
    2. plot the sampling points
    3. give tool tip info of pollution sampled
    4. change the date with slider or drop down box ###with meteros app
    5. plot the sampling points(done with meteros app)
    6. give tool tip info of pollution sampled (done with metros app)
    7. change the date with drop down box(it is not working)

  5. The meteros app can be edited as of slider app, which is more simple and understandable for data filter with the slider option.

  6. meteors-master_A9 is doing that- first removing usage of queue library in the app. Then giving the functionality to view the map and get the csv data into it plot it, then have the feature of zooming, slider data change and tooltip info view.

    1. With attempt 1 in the drwamap.js of meteors-master_A9, it was achieved to view the map, get the csv data and plot it
    2. with attemp 3, it is achieved zooming functionality
    3. with attempt 4, achieved slider view in the web app, it can’t usable with custom dates, the alternative is to go with drop down box only. The library used for the slider is a custom js named d3.slider.js.
    4. with attempt 5, achieved slider functionality of data filtering by slider movment the description of all the steps carried ouyt is given in the file of drawMap.js of meteors-master_A9.

  7. The last attempt gives learning on the working of d3.js based web app. It consists of three components .

    1. view map
    2. plot the sampling point
    3. refresh the sampling point based on the date filter. The web app contains three components for above functionality, all are in the main js file. This file contains three sections, one global variable declaration section, and two is file processing section which does draw the map(json and csv file), three is the replotting section related with filter tool slider and its associated redraw function.

  8. The filtering carried out in the last attempt has to be improved to include the normal csv.

  9. This attempt will be named as meteors-master_A10 and dealt with drop down functionality and redraw functionality with drop down box using the remaining one and second section as such.