Nishadh KA

Pandas JSON

2014-11-26


####Pandas to JSON#### #####It is for this script /scripts/PandasJSONscript.py#####

  1. To convert pandas dataframe into JSON, converted pandas dataframe into csv and then using this script to convert csv into json. But it was not giving satisfactory json file, the fields where completely shuffled.
  2. Then used pandas in built function of to_json but it was giving nested json file. import pandas import pandas as pd d=pd.read_csv('/home/swl-sacon-dst/Documents/GISE_2013/LAB/Aerocet_DATA/Oct_sample/json/A08102014M.csv') db=d[['Time','long','lat','PM1(ug/m3)','PM2.5(ug/m3)','PM4(ug/m3)','PM7(ug/m3)','PM10(ug/m3)','TSP(ug/m3)']] db['geojson']='{\\"type\\":\\"Point\\",\\"coordinates\\":['+db['long'].astype(str)+','+db['lat'].astype(str)+']}' db2=db[['Time','PM2.5(ug/m3)','PM10(ug/m3)','TSP(ug/m3)','geojson']] db2.to_json('testJS3.json') db2.reset_index().to_json('testJS4.json',orient='index') db2.to_json('testJS4.json',orient='index')
  3. Then found that making dictionary element is bettwer way to do json from pandas. This script does exactly that. Though this script ends in error as commented along with the script.
import pandas
import json
import pandas as pd
data=pd.read_csv('/home/swl-sacon-dst/Documents/GISE_2013/LAB/Aerocet_DATA/Oct_sample/json/A08102014M.csv')
db=data[['Time','long','lat','PM2.5(ug/m3)','PM10(ug/m3)','TSP(ug/m3)']]
db['geojson']='{\"type\":\"Point\",\"coordinates\":['+db['long'].astype(str)+','+db['lat'].astype(str)+']}'
d = [
    dict([
        (colname, row[i])
        for i,colname in enumerate(db.columns)
    ])
    for row in df.values
]

#alternative to above which gives error of IndexError: index out of bounds
d = [{colname : row[i] for i, colname in enumerate(db.columns)} for row in db.iterrows()]
#alternative to above with error of IndexError: tuple index out of range
d=[{k:db.values[i][v] for v,k in enumerate(db.columns)} for i in range(len(db)) ]
#the above only is working and genrated the json as of the meteros example, based on [this](https://gist.github.com/mikedewar/1486027)
jsonf = open('testJS7.json','w')
dd=json.dumps(d)
jsonf.write(dd)
jsonf.close()
  1. The dict creation line given as a comment with the script ends without any error. The line is as follows

d=[{k:db.values[i][v] for v,k in enumerate(db.columns)} for i in range(len(db)) ]