Research
Security News
Malicious npm Package Targets Solana Developers and Hijacks Funds
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
guillotina-elasticsearch
Advanced tools
.. contents::
.. image:: https://travis-ci.org/guillotinaweb/guillotina_elasticsearch.svg?branch=master :target: https://travis-ci.org/guillotinaweb/guillotina_elasticsearch
Elasticsearch integration for guillotina. Supports Elastic search 7.x and 8.x
pip install guillotina_elasticsearch
defaults to Elasticsearch 8.x
support.
config.yaml can include elasticsearch section
.. code-block:: yaml
elasticsearch:
index_name_prefix: "guillotina-"
connection_settings:
hosts:
- "http://127.0.0.1:9200"
sniffer_timeout: 0.5
sniff_on_start: true
security_query_builder: "guillotina_elasticsearch.queries.build_security_query"
Example custom security_query_builder
settings:
.. code-block:: python
async def security_query_builder(container, request):
return {
'query': {
'bool': {
'filter': {
'bool': {
'should': [{'match': {'access_roles': "foobar"}}],
'minimum_should_match': 1
}
}
}
}
}
Setup your python virtual environment for version >=3,8. Tested with 3.8, 3.9 and 3.10
.. code-block:: bash
pip install -e ".[test]" pytest tests/
By default the tests run an ES fixture with version 8. If you want to run the tests for ES version 7, change the image version in the conftest.py
POST SITE_URL/@catalog
{}
DELETE SITE_URL/@catalog
{}
guillotina_elasticsearch
comes with a es-migrate
guillotina command::
./bin/g es-migrate
This command will perform a live migration on the index. It does this by performing the reindex on a new index while the other one is still active.
New index and delete requests are performed on both indexes during live migration.
It is also smart about how to migrate, doing a diff on the mapping and only reindexing the fields that changed.
In this version, the library elasticsearch 7 has been upgraded to elasticsearch 8. There are some changes that need to be taken into account in the settings of old elasticsearch config files.
The elasticsearch field of the config.yaml file is directly passed to instantiate AsyncElasticsearch. The class definition is the same of the synchronous one, to know how to configure your ES take a look at: https://elasticsearch-py.readthedocs.io/en/v8.12.0/api/elasticsearch.html#elasticsearch.Elasticsearch
ES 6 does not have doc types support
aioes deprecated
IElasticSearchUtility changes:
IElasticSearchUtility.conn changes:
If container es (elasticsearch) fails to start when running tests, you should increase max_map_count. command
.. code-block:: bash
sudo sysctl -w vm.max_map_count=262144
All mappings are flattened when they are applied to Elastic Search.
Elasticsearch doesn't support different mappings for different types.
If there are conflicting mappings, this plugin will choose the mapping
in the addon last applications
listed which follows the guillotina practice
of allowing the last listed application to override the settings of
the previously list application
.. code-block:: yaml
elasticsearch:
index_name_prefix: "guillotina-"
connection_settings:
hosts:
- "127.0.0.1:9200"
sniffer_timeout: 0.5
sniff_on_start: true
use_ssl: true
http_auth: admin:admin
Fix potential indexing bug when full=True and reindex_security=True [masipcat]
Being able to build mappings properties with analyzers [nilbacardit26]
type_name=Item&__or=id=foo_item%26id=foo_item2
[nilbacardit26]Replace aioelasticsearch with elasticsearchpy [masipcat]
Dropped support for Elasticsearch 6.x [masipcat]
Disable dynamic mapping (as it was in v3.x) [masipcat]
Fix tm is None when migration is cancelled [masipcat]
Allow to search on context object [bloodbare]
Fix tests not passing [masipcat]
Support es6 and es7 [lferran]
Add github actions and remove travis [lferran]
Add setting 'refresh' [masipcat]
Check code format [acatlla]
default_settings()
[masipcat]Support Guillotina 6 [masipcat]
Support elasticsearch 7.0 [jordic]
Make sure to save sub index changes in ES [vangheem]
Fix default index settings [vangheem]
Pinned aioelasticsearch to <0.6.0 [masipcat]
Be able to import types [vangheem]
Retry conflict errors on delete by query
Pay attention to trashed objects in pg
Fix commands using missing attribute self.request
ISecurityInfo can be async
Fix not iterating over all content indexes in elasticsearch [vangheem]
build_security_query(): changed 'query.bool.filter' to use a list instead of a single object [masipcat]
Fix release
Missing pg conn lock with vacuuming [vangheem]
Pass request on the index progress when possible
Fix release
Do not require request object for vacuuming [vangheem]
G5 support [vangheem]
Do not close indexes on create/delete [vangheem]
Handle another index not found error on vacuum [vangheem]
logging [vangheem]
Handle index not found error [vangheem]
Add parser to work with g5 automatic parsing
Use @id
in results results instead of @absolute_url
Support only elasticsearch 7
We may want to pin newest version of aioelasticsearch when that also supports ES7: https://github.com/aio-libs/aioelasticsearch/pull/165
Check supported ES version on utility initialize
Validate index name does not have ':' characters
[lferran]
ModuleNotFoundError
error in vacuum
[vangheem]Be able to customize how you create es connection object [vangheem]
Not loading utility by default and avoid crash when utility is not configured [bloodbare]
Support for opendistro [vangheem]
elasticsearch["security_query_builder"]
setting.
[vangheem]Add 'creation_date', 'modification_date', 'tags' to stored fields [vangheem]
Log indexing errors [vangheem]
vacuum should work with customized pg table names [vangheem]
Use cursor for iterating over data [vangheem]
cache=True
to Migrator
and Reindexer
to choose
overwrite the txn._cache or not [masipcat]upgrade to guillotina >= 4.3.0 [vangheem]
Fix indexing when object does not yet exist in index [vangheem]
Fix tid check in vacuum [vangheem]
The guillotina.async import is a syntax error on python 3.7 [vangheem]
Don't fail on CREATE_INDEX
Do not reprocess if tid is not present in data [vangheem]
retry conflict errors and thread pool exceeded errors [vangheem]
Handle running migration when existing index does not exist [vangheem]
Make sure to refresh object before writing to it. [vangheem]
provide get_index_for util [vangheem]
provide cleanup_es test util [vangheem]
fix storing annotation data on sub index [vangheem]
get_by_path should accept index param [vangheem]
Fix release [vangheem]
add utils.get_all_indexes_identifier [vangheem]
Create index with mappings/indexes instead of updating them after creation [vangheem]
Add es-fields
command to inspect configured fields
[vangheem]
Add support for sub indexes [vangheem]
Raise exception instead of returning ErrorResponse [vangheem]
Add scroll support to query [gitcarbs]
Add more logging for migrations [vangheem]
Added IIndexProgress to hook on reindex process
Added new event on reindex start.
Added context to the IIndexProgress event [jordic]
replace aioes(unsupported) with aioelasticsearch [vangheem]
Elasticsearch 6 compatibility. [vangheem]
@name
in results
[vangheem]change @id
in search results to @uid
[vangheem]
Add support for analysis character filter [gitcarbs]
index_data
configuration
[vangheem]traverse
instead of do_traverse
.
[vangheem]async with
syntax for manual api calls to elasticsearch using
session object.
[vangheem]Fix issue where a bad original index would screw up index diff calculation [vangheem]
Fix migration failure issue before flipping new index on [vangheem]
Use threads when bulk indexing on elasticsearch to make sure to always keep elasticsearch busy. [vangheem]
Forcing the update of mapping [bloodbare]
es-reindex
command
[vangheem]Improve performance of reindexing of bushy content by using asyncio.gather to traverse sub-trees and index at the same time. [vangheem]
Improve options to reindex command [vangheem]
FAQs
elasticsearch catalog support for guillotina
We found that guillotina-elasticsearch demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 4 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
Security News
Research
Socket researchers have discovered malicious npm packages targeting crypto developers, stealing credentials and wallet data using spyware delivered through typosquats of popular cryptographic libraries.
Security News
Socket's package search now displays weekly downloads for npm packages, helping developers quickly assess popularity and make more informed decisions.