large_csv_reader

Large CSV Reader GEM

This gem is created to help with a known issue, processing large csv files using ruby, without breaking our RAM. I used lazy enumeration to achieve this, that allows the methods to execute actions on one line at a time instead of loading millions of lines in memory at once.

I consider this gem in Beta state, proper testing suite is still missing and there are some improvements or extensions that could be useful to have in a full version of it.

Installation

gem install large_csv_reader

• In your files: require 'large_csv_reader'

Usage

General Methods

Function	Description
reader = LargeCsvReader.new	creates a new instance of the reader
reader.generate_csv(fileName, columnNames)	creates a new csv file with the name and header names passed as parameters
reader.append_to_csv(filename, rows=1000000,rowStructure)	Add lines to the csv, this lines are generated with the rowStructure array parameter. If rows parameters is not present by default it will load 1 million lines to the file.
reader.massive_read_in_csv_data (file_name)	lazy load of each csv row into a list
reader.massive_csv_builder(filename, column_names,rowMult="1")	create a csv with millions of lines, the value of rowMult represents how many millions lines the file will have
reader.row_generator(structure)	generate rows on demand using enumeration

Specific Methods

The rest of the methods are considerations to solve a test problem with book data "Date", "ISBN", "Price"

• massive_total_value_in_stock(csv_file_name)
• massive_number_of_each_isbn(csv_file_name)
• append_book_to_csv(filename,rows=1000000)
• book_generator