Topographer is a gem that provides functionality to conveniently import from various data sources into ActiveRecord or other ORM systems. This accomplished by defining an input wrapper, a mapping from input data to models, and a strategy for use in persisting data.
Add this line to your application's Gemfile:
gem 'topographer'
And then execute:
$ bundle
Or install it yourself as:
$ gem install topographer
Topographer user a data mapping defined by a Mapper object to transform values from a single input record onto a output record.
The general flow of importing data is as follows:
Topogapher::Importer.import_data is called with an instance of an input wrapper, an import class, a persistance strategy class, and a logger instance.
A call to the .get_mapper method of the import class produces an instance of Topographer::Importer::Mapper, which is used to transform data from the input record to the format needed for output. A mapper is constructed with an associated model class, which is generally the ORM class being imported.
An instance of the persistance strategy class is created using the Mapper instance obtained from the input class.
Then each "row" of input data is yielded sequentially as an instance of Topographer::Importer::Input::SourceData, which is then passed to the #input_record method of the strategy instance.
The strategy instance is responsible for using its Mapper instance to transform the input record into output data and the persist it in some way. Generally, this means that the input strategy uses the result of mapping the data to create a new instance of the Mapper's model class and persist it using the appropriate ORM record.
Each call to #input_record is expected to return an instance of Topographer::Importer::Strategy::ImportStatus which encapsulates information about the status of a particular row of input.
This instance of ImportStatus is then used to construct a log entry for the input record, which is added to the logger instance passed to .import_data.
Topographer provides a convenient installer Thor task which can be used to generate a scaffold to get it up and running in your projects. Invoking
thor topographer:install --models=Model1 Model2 Model3
in your project's root path will add a import scaffolding to the lib folder in the following structure
a user interface import which is meant to drive importing from the user interface. This importer currently makes some assumptions about where uploaded files should be saved, so you may want to customize the .store_upload method to suit your needs
additionally, a view, <project_root>/app/views/shared/_import_log_dispay.html.erb is added which will render the import log in application views. It should be called as follows:
<%= render 'shared/import_log_display', logger: @logger%>
where the logger instance variable is the result of a call to Imports::UserInterfaceImport.import_spreadsheet
Briefly, a mapping is a set of rules for translating from an imported row of data to the fields needed to represent an output record. Topographer uses a Mapper, which is an object that encapsulates a mapping, to map input data into a hash of output values that are available to a strategy to create output records.
Mappings are defined as follows:
class Post < ActiveRecord::Base
belongs_to: author, class_name: 'User'
validates :author, presence: true
validates :title, presence: true
validates :body, presence: true
end
with the following database columns:
author_id: integer
title: string
summary: string
body: string
created: date
updated: date
rating: integer
private: boolean, null: false
and an import spreadsheet with the following columns
Author Name | Title | Summary | Body | Created | Rating | Importable
where data is to be imported only if the importable column's value is 'Yes', each imported post is to be given a private flag of false, each post's title should be titlized, and each post without a created date should be given a created date of April 1, 2014
Assuming that the import class for Posts is defined in lib/imports/mappings/post.rb in such a way that the instance method import_mapping is called to produce a mapping for data importing,
A mapping would be defined as follows:
def import_mapping
Topographer::Importer.build_mapper(::Post) do |mapper|
mapper.required_mapping('Author Name', 'author_id') do |inputs|
author = User.find_by(name: inputs['Author Name'])
raise "Unable to find Author with name `#{inputs['Author Name']}`" unless author
author.id
end
mapper.required_mapping('Title', 'title') do |inputs|
inputs['title'].titlize
end
mapper.required_mapping('Body', 'body')
mapper.optional_mapping('Summary', 'summary')
mapper.optional_mapping('Created', 'created') do |inputs|
date = inputs['Created'] || Date.parse('1/4/2014')
end
mapper.optional_mapping('Rating', 'rating')
mapper.validation_field('Is Importable?', 'importable') do
raise 'Not importable' unless inputs['importable'] == 'Yes'
end
mapper.default_value('private') do
false
end
end
end
In the above method, the call to Topographer::Importer.build_mapper requires a model class as an agrument. This argument becomes the model class for the resulting Mapper and is available to any Strategy that uses the Mapper. Generally, this class should be the ORM class that one is trying to import.
The block that follows the call to .build_mapper receives an object that can be used to construct a mapping. The mapping methods that are available are as follows:
#required_mapping
Adds a required mapping for one or more input columns to a single output column. Required mappings mean that each input column is required to have a value for the mapping to be considered valid. If there is a value missing for a particular row of input, then an mapping error will be generated for that row.
Arguments
When there is no behavior block defined, the resulting output field value is simply the value from the import sheet.
When a behavior block is defined, the value returned by that block will be the value present for the output field.
If the block raises an execption, it's message will be added to the row in question as a mapping error.
#optional_mapping
Optional mappings behave exactly as required mappings except that an optional mapping that has one or more input column values missing for a particulr import row will not be considered a mapping error
#validation_field
Validation field mappings allow for a column to be used to validate an import record without that column being mapped to an output field.
When the validation block raises an exeption, the input record is considered 'invalid' and a mapping error is generated
#key_field
Key fields define fields that are made available to strategies in order to look up a given output record. They are primarily used during imports which require updating existing database records.
#default_value
Default value mappings are used to define a static default value for an output field across all imported records
#ignored_column
Ignored columns allow a mapping to explicitly ignore a column in the input data
Given the above mapping for a post import, one might import posts from within a rake task as follows:
logger = Imports::CommandLineImport.import_spreadsheet(
'/var/www/apps/my_app/input_file.xlsx',
Imports::Runners::ImportNewRecord,
Imports::Mappings::Post
)
at which point, logger will encapsulate a log of all each row processed in the input file.
Import strategies encapsulate the logic needed to persist the result of mapping a single input row in some way. Every strategy will have access to an instance of Mapper which it can use to transform input data into output data.
In general, a strategy should use its mapper to transform input data, then it should persist that data in some way, either by creating a new record of the Mapper's model class or through some other means, and finally it should return an ImportStatus object that encapsulates information about the result of the mapping, the result of the persistance attempt, and any other errors that may have occured.
Currently Topographer defines basic strategies for importing new records, updating existing records, and either updating a record, or creating one if a matching record cannot be found.
Because import strategies often differ between projects, subclasses of these strategies are created as part of the Topographer install to facilitate defining custom behaviors.
Check back soon for additional documentation concerning adding new strategies.
git checkout -b my-new-feature)git commit -am 'Add some feature')git push origin my-new-feature)FAQs
Unknown package
We found that topographer demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.