Freezing problem on one of github archive files

[oskar-j]

Iterload() freezes on file 2011-06-28-0.json.gz

Script will freeze when this file is encountered :(

[bgruszka]

hmm... what is size of this file?

[oskar-j]

after unpacking: 94,1 MB
packed 19,1 MB
maybe yielding json.loads() is better, like below:
http://stackoverflow.com/questions/6886283/how-i-can-i-lazily-read-multiple-json-objects-from-a-file-stream-in-python

[oskar-j]

Is it a problem that I run 2 scripts at once ? Shouldn't be..
First script, tail logs:

(lot of dots..)
/mnt/data1/wikiteams/GitHubArchive/2011-06-28-0.json.gz


and second log:

/mnt/data1/wikiteams/GitHubArchive/2011-06-28-0.json.gz (lot of dots)

So I managed to add to mongo only:

db.events.count()
649122
db.stargazing.count()
57211


But both scripts stops on this file.. or maybe the next one ? It would be good idea to add more verbose messages..

[bgruszka]

We can try yielding json. I hope this solution can help :) Could you check if yielding is ok on this file?

[oskar-j]

Yep I will play around with the code today..

[oskar-j]

So seems like The GitHub allows for exactly one assignee per issue. Assigning to myself.