elasticsearch-persistence

Elasticsearch::Persistence

Persistence layer for Ruby domain objects in Elasticsearch, using the Repository pattern.

Compatibility

This library is compatible with Ruby 3.1 and higher.

The version numbers follow the Elasticsearch major versions. Currently the main branch is compatible with version 8.x of the Elasticsearch stack.

Rubygem		Elasticsearch
0.1	→	1.x
2.x	→	2.x
5.x	→	5.x
6.x	→	6.x
7.x	→	7.x
8.x	→	8.x
main	→	8.x

Installation

Install the package from Rubygems:

gem install elasticsearch-persistence

To use an unreleased version, either add it to your Gemfile for Bundler:

gem 'elasticsearch-persistence', git: 'git://github.com/elastic/elasticsearch-rails.git', branch: '6.x'

or install it from a source code checkout:

git clone https://github.com/elastic/elasticsearch-rails.git
cd elasticsearch-rails/elasticsearch-persistence
bundle install
rake install

Usage

The library provides the Repository pattern for adding persistence to your Ruby objects.

The Repository Pattern

The Elasticsearch::Persistence::Repository module provides an implementation of the repository pattern and allows you to save, delete, find and search objects stored in Elasticsearch, as well as configure mappings and settings for the index. It's an unobtrusive and decoupled way of adding persistence to your Ruby objects.

Let's have a simple plain old Ruby object (PORO):

class Note
  attr_reader :attributes

  def initialize(attributes={})
    @attributes = attributes
  end

  def to_hash
    @attributes
  end
end

Let's create a default, "dumb" repository, as a first step:

require 'elasticsearch/persistence'
class MyRepository; include Elasticsearch::Persistence::Repository; end
repository = MyRepository.new

We can save a Note instance into the repository...

note = Note.new id: 1, text: 'Test'

repository.save(note)
# PUT http://localhost:9200/repository/_doc/1 [status:201, request:0.210s, query:n/a]
# > {"id":1,"text":"Test"}
# < {"_index":"repository","_id":"1","_version":1,"created":true}

...find it...

n = repository.find(1)
# GET http://localhost:9200/repository/_doc/1 [status:200, request:0.003s, query:n/a]
# < {"_index":"repository","_id":"1","_version":2,"found":true, "_source" : {"id":1,"text":"Test"}}
=> <Note:0x007fcbfc0c4980 @attributes={"id"=>1, "text"=>"Test"}>

...search for it...

repository.search(query: { match: { text: 'test' } }).first
# GET http://localhost:9200/repository/_search [status:200, request:0.005s, query:0.002s]
# > {"query":{"match":{"text":"test"}}}
# < {"took":2, ... "hits":{"total":1, ... "hits":[{ ... "_source" : {"id":1,"text":"Test"}}]}}
=> <Note:0x007fcbfc1c7b70 @attributes={"id"=>1, "text"=>"Test"}>

...or delete it:

repository.delete(note)
# DELETE http://localhost:9200/repository/_doc/1 [status:200, request:0.014s, query:n/a]
# < {"found":true,"_index":"repository","_id":"1","_version":3}
=> {"found"=>true, "_index"=>"repository", "_id"=>"1", "_version"=>2}

The repository module provides a number of features and facilities to configure and customize the behavior:

• Configuring the Elasticsearch client being used
• Setting the index name, and object class for deserialization
• Composing mappings and settings for the index
• Creating, deleting or refreshing the index
• Finding or searching for documents
• Providing access both to domain objects and hits for search results
• Providing access to the Elasticsearch response for search results (aggregations, total, ...)
• Defining the methods for serialization and deserialization

There are two mixins you can include in your Repository class. The first Elasticsearch::Persistence::Repository, provides the basic methods and settings you'll need. The second, Elasticsearch::Persistence::Repository::DSL adds some additional class methods that allow you to set options that instances of the class will share.

Basic Repository mixin

For simple cases, you can just include the Elasticsearch::Persistence::Repository mixin to your class:

class MyRepository
  include Elasticsearch::Persistence::Repository

  # Customize the serialization logic
  def serialize(document)
    super.merge(my_special_key: 'my_special_stuff')
  end

  # Customize the de-serialization logic
  def deserialize(document)
    puts "# ***** CUSTOM DESERIALIZE LOGIC... *****"
    super
  end
end

client = Elasticsearch::Client.new(url: ENV['ELASTICSEARCH_URL'], log: true)
repository = MyRepository.new(client: client, index_name: :my_notes, klass: Note)
repository.settings number_of_shards: 1 do
  mapping do
    indexes :text, analyzer: 'snowball'
  end
end

The custom Elasticsearch client will be used now, with a custom index, as well as the custom serialization and de-serialization logic.

We can create the index with the desired settings and mappings:

repository.create_index! force: true
# PUT http://localhost:9200/my_notes
# > {"settings":{"number_of_shards":1},"mappings":{ ... {"text":{"analyzer":"snowball","type":"string"}}}}}

Save the document with extra properties added by the serialize method:

repository.save(note)
# PUT http://localhost:9200/my_notes/note/1
# > {"id":1,"text":"Test","my_special_key":"my_special_stuff"}
{"_index"=>"my_notes", "_id"=>"1", "_version"=>4, ... }

And deserialize it:

repository.find(1)
# ***** CUSTOM DESERIALIZE LOGIC... *****
<Note:0x007f9bd782b7a0 @attributes={... "my_special_key"=>"my_special_stuff"}>

The DSL mixin

In some cases, you'll want to set some of the repository configurations at the class level. This makes most sense when the instances of the repository will use that same configuration:

require 'base64'

class NoteRepository
  include Elasticsearch::Persistence::Repository
  include Elasticsearch::Persistence::Repository::DSL

  index_name 'notes'
  klass Note

  settings number_of_shards: 1 do
    mapping do
      indexes :text,  analyzer: 'snowball'
      # Do not index images
      indexes :image, index: false
    end
  end

  # Base64 encode the "image" field in the document
  #
  def serialize(document)
    hash = document.to_hash.clone
    hash['image'] = Base64.encode64(hash['image']) if hash['image']
    hash.to_hash
  end

  # Base64 decode the "image" field in the document
  #
  def deserialize(document)
    hash = document['_source']
    hash['image'] = Base64.decode64(hash['image']) if hash['image']
    klass.new hash
  end
end

You can create an instance of this custom class and get each of the configurations.

client = Elasticsearch::Client.new(url: 'http://localhost:9200', log: true)
repository = NoteRepository.new(client: client)
repository.index_name
# => 'notes'

You can also override the default configuration with options passed to the initialize method:

client = Elasticsearch::Client.new(url: 'http://localhost:9250', log: true)
client.transport.logger.formatter = proc { |s, d, p, m| "\e[2m# #{m}\n\e[0m" }
repository = NoteRepository.new(client: client, index_name: 'notes_development')

repository.create_index!(force: true)

note = Note.new('id' => 1, 'text' => 'Document with image', 'image' => '... BINARY DATA ...')

repository.save(note)
# PUT http://localhost:9200/notes_development/_doc/1
# > {"id":1,"text":"Document with image","image":"Li4uIEJJTkFSWSBEQVRBIC4uLg==\n"}
puts repository.find(1).attributes['image']
# GET http://localhost:9200/notes_development/_doc/1
# < {... "_source" : { ... "image":"Li4uIEJJTkFSWSBEQVRBIC4uLg==\n"}}
# => ... BINARY DATA ...

Functionality Provided by the Repository mixin

Each of the following configurations can be set for a repository instance. If you have included the Elasticsearch::Persistence::Repository::DSL mixin, then you can use the class-level DSL methods to set each value. You can still override the configuration for any instance by passing options to the #initialize method. Even if you don't use the DSL mixin, you can set the instance configuration with options passed the #initialize method.

Client

The repository uses the standard Elasticsearch client.

client = Elasticsearch::Client.new(url: 'http://search.server.org')
repository = NoteRepository.new(client: client)
repository.client.transport.logger = Logger.new(STDERR)
repository.client
# => Elasticsearch::Client

or with the DSL mixin:

class NoteRepository
  include Elasticsearch::Persistence::Repository
  include Elasticsearch::Persistence::Repository::DSL

  client Elasticsearch::Client.new url: 'http://search.server.org'
end

repository = NoteRepository.new
repository.client
# => Elasticsearch::Client

Naming

The index_name method specifies the Elasticsearch index to use for storage, lookup and search. The default index name is 'repository'.

repository = NoteRepository.new(index_name: 'notes_development')
repository.index_name
# => 'notes_development'

or with the DSL mixin:

class NoteRepository
  include Elasticsearch::Persistence::Repository
  include Elasticsearch::Persistence::Repository::DSL

  index_name 'notes_development'
end

repository = NoteRepository.new
repository.index_name
# => 'notes_development'

The klass method specifies the Ruby class name to use when initializing objects from documents retrieved from the repository. If this value is not set, a Hash representation of the document will be returned instead.

repository = NoteRepository.new(klass: Note)
repository.klass
# => Note

or with the DSL mixin:

class NoteRepository
  include Elasticsearch::Persistence::Repository
  include Elasticsearch::Persistence::Repository::DSL

  klass Note
end

repository = NoteRepository.new
repository.klass
# => Note

Index Configuration

The settings and mappings methods, provided by the elasticsearch-model gem, allow you to configure the index properties:

repository.settings number_of_shards: 1
repository.settings.to_hash
# => {:number_of_shards=>1}

repository.mappings { indexes :title, analyzer: 'snowball' }
repository.mappings.to_hash
# => { :note => {:properties=> ... }}

or with the DSL mixin:

class NoteRepository
  include Elasticsearch::Persistence::Repository
  include Elasticsearch::Persistence::Repository::DSL

  mappings { indexes :title, analyzer: 'snowball' }
  settings number_of_shards: 1
end

repository = NoteRepository.new

Create a Repository and set its configuration with a block

You can also use the #create method to instantiate and set the mappings and settings on an instance with a block in one call:

repository = NoteRepository.create(index_name: 'notes_development') do
  settings number_of_shards: 1, number_of_replicas: 0 do
    mapping dynamic: 'strict' do
      indexes :foo do
        indexes :bar
      end
      indexes :baz
    end
  end
end

Index Management

The convenience methods create_index!, delete_index! and refresh_index! allow you to manage the index lifecycle. These methods can only be called on repository instances and are not implemented at the class level.

Serialization

The serialize and deserialize methods allow you to customize the serialization of the document when it is persisted to Elasticsearch, and define the initialization procedure when loading it from the storage:

class NoteRepository
  include Elasticsearch::Persistence::Repository

  def serialize(document)
    Hash[document.to_hash.map() { |k,v|  v.upcase! if k == :title; [k,v] }]
  end

  def deserialize(document)
    MyNote.new ActiveSupport::HashWithIndifferentAccess.new(document['_source']).deep_symbolize_keys
  end
end

Storage

The save method allows you to store a domain object in the repository:

note = Note.new id: 1, title: 'Quick Brown Fox'
repository.save(note)
# => {"_index"=>"notes_development", "_id"=>"1", "_version"=>1, "created"=>true}

The update method allows you to perform a partial update of a document in the repository. Use either a partial document:

repository.update id: 1, title: 'UPDATED',  tags: []
# => {"_index"=>"notes_development", "_id"=>"1", "_version"=>2}

Or a script (optionally with parameters):

repository.update 1, script: 'if (!ctx._source.tags.contains(t)) { ctx._source.tags += t }', params: { t: 'foo' }
# => {"_index"=>"notes_development", "_id"=>"1", "_version"=>3}

The delete method allows you to remove objects from the repository (pass either the object itself or its ID):

repository.delete(note)
repository.delete(1)

Finding

The find method allows you to find one or many documents in the storage and returns them as deserialized Ruby objects:

repository.save Note.new(id: 2, title: 'Fast White Dog')

note = repository.find(1)
# => <MyNote ... QUICK BROWN FOX>

notes = repository.find(1, 2)
# => [<MyNote... QUICK BROWN FOX>, <MyNote ... FAST WHITE DOG>]

When the document with a specific ID isn't found, a nil is returned instead of the deserialized object:

notes = repository.find(1, 3, 2)
# => [<MyNote ...>, nil, <MyNote ...>]

Handle the missing objects in the application code, or call compact on the result.

Search

The search method is used to retrieve objects from the repository by a query string or definition in the Elasticsearch DSL:

repository.search('fox or dog').to_a
# GET http://localhost:9200/notes_development/_doc/_search?q=fox
# => [<MyNote ... FOX ...>, <MyNote ... DOG ...>]

repository.search(query: { match: { title: 'fox dog' } }).to_a
# GET http://localhost:9200/notes_development/_doc/_search
# > {"query":{"match":{"title":"fox dog"}}}
# => [<MyNote ... FOX ...>, <MyNote ... DOG ...>]

The returned object is an instance of the Elasticsearch::Persistence::Repository::Response::Results class, which provides access to the results, the full returned response and hits.

results = repository.search(query: { match: { title: 'fox dog' } })

# Iterate over the objects
#
results.each do |note|
  puts "* #{note.attributes[:title]}"
end
# * QUICK BROWN FOX
# * FAST WHITE DOG

# Iterate over the objects and hits
#
results.each_with_hit do |note, hit|
  puts "* #{note.attributes[:title]}, score: #{hit._score}"
end
# * QUICK BROWN FOX, score: 0.29930896
# * FAST WHITE DOG, score: 0.29930896

# Get total results
#
results.total
# => 2

# Access the raw response as a Hashie::Mash instance.
# Note that a Hashie::Mash will only be created if the 'response' method is called on the results. 
results.response._shards.failed
# => 0

# Access the raw response
results.raw_response
# => {...}

Example Application

An example Sinatra application is available in examples/notes/application.rb, and demonstrates a rich set of features:

• How to create and configure a custom repository class
• How to work with a plain Ruby class as the domain object
• How to integrate the repository with a Sinatra application
• How to write complex search definitions, including pagination, highlighting and aggregations
• How to use search results in the application view

The ActiveRecord Pattern

The ActiveRecord pattern has been deprecated as of version 6.0.0 of this gem. Please use the Repository Pattern instead. For more information on migrating 5.x ActiveRecord-based applications to use the Repository Pattern, please see this blog post.

License

This software is licensed under the Apache 2 license, quoted below.

Licensed to Elasticsearch B.V. under one or more contributor
license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright
ownership. Elasticsearch B.V. licenses this file to you under
the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.