Skip to content

Add lil-csv, update readme with latest benchmarks #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
koresar wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add lil-csv, update readme with latest benchmarks #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
// var csv=require('fast-csv')
// var csv=require('csv')
var csvtojson = require('csvtojson')
var csv = require('csv')
var fastCsv = require('fast-csv')
var csvParser = require('csv-parser')
var Papa = require('papaparse')
var lilCsv = require('lil-csv')

var csvFile = '1.csv'
var outputFile = "output.txt";
Expand All @@ -23,6 +22,10 @@ testFastCsv(function () {
console.time("papaparse")
testPapaParse(function () {
console.timeEnd("papaparse")
console.time("lil-csv")
testLilCsv(function () {
console.timeEnd("lil-csv")
})
})
})
})
Expand Down Expand Up @@ -97,4 +100,12 @@ function testPapaParse(cb) {
.on("end",function(){
cb();
})
}
}

function testLilCsv(cb) {
var fileName = 'lilCsv-' + outputFile;
var fileContents = require('fs').readFileSync(csvFile, "utf-8")
var data = lilCsv.parse(fileContents, { header: false })
require('fs').writeFileSync(fileName, JSON.stringify(data.reduce((acc, row) => acc + "\n" + row[0], "")))
cb()
}
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"csv-parser": "^2.3.1",
"csvtojson": "^1.1.0",
"fast-csv": "^2.3.0",
"lil-csv": "^1.4.1",
"papaparse": "^5.1.0"
}
}
17 changes: 11 additions & 6 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,15 @@ $ node ./index.js

## Procedure

All parsers will simply do following
All parsers (except `lil-csv`) will simply do following

1. stream in 1.csv and start to parse
2. get content from 1st column of each row and write to a file

The `lil-csv` would

1. read the whole 1.csv without streaming
2. get content from 1st column of each row and write to a file

`1.csv` contains 300K lines

Expand All @@ -26,11 +30,12 @@ Here is some result running on a 2019 Macbook Pro:

```
$ node ./index.js
csv: 1844.045ms
fast-csv: 1681.046ms
csv-parser: 722.762ms
csvtojson: 436.225ms
papaparse: 434.335ms
fast-csv: 1438.705ms
csv: 2346.760ms
csvtojson: 435.524ms
csv-parser: 878.652ms
papaparse: 430.366ms
lil-csv: 1135.411ms
$ node --version
v12.4.0
```
Expand Down