How to Parse LARGE JSON files with formating error - ruby-on-rails

Question

I have a bunch of large JSON files (> 500MB) which I would like to parse with the ruby script (I am trying. to parse it with YAJL gem).
I have noticed that JSON files have formatting errors such that all the files composed of "multiple" JSON objects without a proper tree-like structure or array. Below you can find how the JSON file looks like:
testfile.json:
{title: "Don Quixote", author: "Miguel de Cervantes", printyear: 2010}
{title: "Great Gatsby", author: "F. Scott Fitzgerald", printyear: 2014}
{title: "Ulysses", author: "James Joyce", printyear: 2010}
This is the script to parse file:
require 'yajl'
json = File.new('testfile.json', 'r')
hash = Yajl::Parser.parse(json)
Here is the error message I get:
Yajl::ParseError: Found multiple JSON objects in the stream but no block or the on_parse_complete callback was assigned to handle them.
I will appreciate if you can guide me on how to solve this issue.

Answer 1

The error message you got ("Found multiple JSON objects in the stream …") implies that your input contains multiple but valid JSON objects, so I assume your actual file looks more like this:
{"title":"Don Quixote","author":"Miguel de Cervantes","printyear":2010}
{"title":"Great Gatsby","author":"F. Scott Fitzgerald","printyear":2014}
{"title":"Ulysses","author":"James Joyce","printyear":2010}
One of YAJL's feature is to:
Parse and encode multiple JSON objects to and from streams or strings continuously.
So given above input (as a file or string), you can pass a block to parse which will be called for each parsed object:
require 'yajl'
io = File.open('testfile.json')
Yajl::Parser.parse(io) do |book|
puts "“#{book['title']}” by #{book['author']} (#{book['printyear']})"
end
Output:
“Don Quixote” by Miguel de Cervantes (2010)
“Great Gatsby” by F. Scott Fitzgerald (2014)
“Ulysses” by James Joyce (2010)

Answer 2

Don't use JSON.parse, beacuse the file's content is not a JSON. Each line from this file looks like a Ruby hash, so different parsing method could be used.
You should be able to parse each line by using: YAML.load(line).
Also, beacuse the file is big, don't load the whole file into memory. Use File.foreach to load line by line.
require 'yaml'
lines = []
File.foreach('testfile.json') do |line|
lines << YAML.load(line)
end