copious-registry
Advanced tools
Very common operations for managing and watching directories of entries and keeping entries in memory for searching
This module provides several classes that may occur commonly together.
For use in node.js projects:
npm install -s copious-registry
There is an implicit dependency on an externally specified (configured) hash function. xxhash32-node-cmake works on some of my machines. But, there are some build problems that have to be addressed with node.js 24. So, as a default xxhash-addon will be required.
The xxhash32-node-cmake may be specified by adding the field, xxhash to the configuration object, conf, passed during construction. Otherwise, xxhash-addon will be loaded.
Neither module will be installed with the installation of this module. So, in order to access them, they should be installed gloabally as such:
npm install -g xxhash32-node-cmake
and/or
npm install -g xxhash32-addon
If the build is not in sync with your version of Linux, the other build may be.
These should be installed prior to using the registy class.
Originally, I made xxhash32-node-cmake to use the cmake module build and to use nan. Older versions worked fine. The mos recent is having some problems.
The original purpose of this module was to provide interfaces for handling receipt and management of publication of meta data. The meta data are just JSON objects. So, one may think of this module as a small collection of utilities classes that keep references, JSON object, to other data.
The objects that refer to other data may provide protocol based paths, URLs, to assets which may be fetched by the protocols. The paths may be modules that may be loaded by node.js.
The JSON object may store data themselves. However, the application developer should keep in mind that the classes provided here make some attempt to keep all the objects in memory. Typically, the data refers to some larger object BLOB that can be accessed when it is needed. One use has objects refering to the data with IDs (Intergalactic IDs) that can be used to access BLOB assets over a repository bridge.
The classes basically control one list of objects (stored according to the FileLists implementation).
The registry provides access to sorting of the stored list. And, the application may key the lists for certain sorts. The objects are expected to have, or may be given, a field having to do with dates of creation and update. Two lists providing sort by these dates is provided by default.
Items find their way into the lists by two processes.
This is the name of the identifier field that is used throughout. Some thought has been given to making this field configurable. But, that may be treated as the subject of another set of classes.
There is no loss of generality in insisting that objects have a field _tracking, which may be distinguished from tracking that some applications might use, say if the assets are in a shipping application. This field has a similar purpose.
The main requirement of the field _tracking is that it must be a unique ID, which may be obtained by a number of methods including cryptographic hashing.
It is also important that _tracking be on the top level of an object. So, if one were to inspect a JSON file containing an object used in this application one should see _tracking as a field not too far removed from the initial brace. It's value will likely be some base64 representation of a large number.
It is assumed that the top level of the objects handled by these classes will have a field dates.
The dates field is itself an object with the update date and created date as fields.
Here is an example:
{
"_tracking" : "sd8fwoerw8tuwrje9fu",
"dates" : {
"created" : Date.now(),
"updated" : Date.now()
}
}
When the object is added, the class instance will create a date for the object if it does not have one. There is an assumption that it is better for applications generating the objects to create the dates, as they will have a better handle on the time they perform their operations.
This is one more top level field. It's primary purpose is to sequence data. But, currently a hash of the data is being placed in the _x_entry field by default.
Clients consuming lists returned as searches or as complete DB listings may overwrite this field with their own sequence numbering scheme.
This field is used in serializing queries in dependent searching modules.
This module sets the score field of new objects to 1.0 if the filed is not present. The method sort_by_score uses it. So, if an application whishes to use the method sort_by_score, it may control the field on the top level of the object
This is a field that is added to the object when it is being removed. The object is being taken out of tables maintained by the Registry instance. But, the Registry does not know how to remove it from other data structures. Instead, it calls a method app_specific_file_removal after the object is marked with this field.
The registry keeps lists of objects. The lists may be implemented as custom iterable classes. There is also a map that maps the configured ID field to the objects, and items may be found through this map.
Several methods are provided for sorting lists and keeping those list in memory.
Finally, there are methods for adding and removing objects. And, those methods check on the object to make sure that it is created or updated and that it has a proper ID field, etc.
The default field of the ID is _tracking.
This is the registry, the keeper of objects, most of which will described larger objects elsewhere in most applications.
The registry keeps objects in a single iterable. But, it also may keep a map of types of sorts, lists of the objects sorted according to a particular key or functional result.
Other classes in this module serve to put objects into and remove objects from the list this class keeps. Typically, the classes make a call to add_just_one to add an object and remove_just_one to remove an object. However, if a class is loading from backup, it may make the assumption that the object is well formed and just call the method belonging to the list, 'add' or 'push'.
constructor Sets up hashing for when it is needed. Initializes the global tables. If the configuration has the field, file_list_class, it will attempt to use the application defined file list object for storing its objects. The member variable, global_file_list, will be set to an instance of the supplied class, which must have the properties of an iterable.
parameters
unique_idReturns a hash of data or other value depending on the application. The default behavior is to stringify the object passed and then to make a hash of the string using an xxhash function.
parameters
set_global_file_list_refsThe parameters are the storage objects that will be shared by the classes in this module and by descendants of those classes. Sharing is limited to these two objects. If the objects are to be handled in a custom way, then the configuration of this class should include a specification of the FileLists type of object. When the FileLists object is specified, the parameters of this method will be passed to it.
The construction and allocation of the storage objects passed is expected to occur in the application. The classes in this module use the storage objects but do not delete them or replace them in any way.
These classes put objects into the storage class and remove objects from the storage class. So, the application only needs to understand that the objects should not be referenced as in place objects unless the application takes care to manage the existence of the object on its own.
In fact, there calls for removing objects are not directly executed by methods in these class. But, the methods for removal are provided.
parameters
ref_big_list - an array or iterable that will be used to store all the objects available to applications of this class.ref_big_list_by -- a map of sort types to lists; this class will create the application storage type for these lists and place objects in these according to sorts determined by the application or defaults.sort_by_updatedTakes in an iterable of objects to be sorted.
This method will return and array (Array object) of objects that are sorted according to their date.update field.
If the object is an array, it will sort it, accessing the objects date.update field. If it is another sort of iterable, it will retrieve an array of values from the iterable and then recursively call this method.
parameters
sort_by_createdTakes in an iterable of objects to be sorted.
This method will return and array (Array object) of objects that are sorted according to their date.created field.
If the object is an array, it will sort it, accessing the objects date.created field. If it is another sort of iterable, it will retrieve an array of values from the iterable and then recursively call this method.
parameters
sort_by_scoreTakes in an iterable of objects to be sorted.
This method assumes the application has supplied the objects in its storage object a score field. The score field should be a number determined by the application.
This method will return and array (Array object) of objects that are sorted according to their score field.
If the object is an array, it will sort it, accessing the objects score field. If it is another sort of iterable, it will retrieve an array of values from the iterable and then recursively call this method.
parameters
sort_by_fieldTakes in an iterable of objects to be sorted.
This method assumes the application has supplied the objects in its storage object a [field] field. The [field] field should be a number determined by the application.
This method will return and array (Array object) of objects that are sorted according to their [field] field.
If the object is an array, it will sort it, accessing the objects [field] field. If it is another sort of iterable, it will retrieve an array of values from the iterable and then recursively call this method.
Here [field] means the field accessed by the value of [field] as in val = obj[field].
parameters
sort_by_fnTakes in an iterable of objects to be sorted. Also, it takes in a functional that will have access to two objects at a time and will return -1,0,1 according to the requirements of the Array object sort method.
This method will return and array (Array object) of objects that are sorted according to their score field.
If the object is an array, it will sort it, accessing the objects score field. If it is another sort of iterable, it will retrieve an array of values from the iterable and then recursively call this method.
parameters
update_tracking_mapExamines every object in the global file list and makes sure that it is accounted for in ther tracking map.
no parameters
update_global_file_list_quotes_byThis method has a default behavior for when the application does not use a FileList class. If the application just uses arrays, then this method calls on the sort methods and touches the score fields of the objects.
This method calls on the sorting methods for soring by dates. It sets all scores to 1.0 if the score field is abscent from the object. Also, it looks for the lists mapped by the [field] parameters needed for sort_by_field.
If the application is using a FileList class this method defers to the update_global_file_list_quotes_by of that class.
no parameters
from_all_filesTakes the parameter that maps a field name to a list of sorted objects. Also, "create_date" and "update_date" may be the parameter passed.
This returns the list which is assumed to have been sorted at some previous time.
parameters
add_just_oneAdds and object into the global iterable. Updates list membership in other lists as well. If the application knows that the object is a new object, the object will be added without trying to look it up. If not, this method first searches for the object and overwrites the one it finds.
In the case that the application has supplied its own iterable objects for the global iterable, this application will simply overwrite the data of a object retrieved by reference from the global iterable.
parameters
remove_just_oneRemoves an object from the global iterable and all lists that it may be found in in this module's classes. It then gives the application to remove the object by calling app_specific_file_removal.
parameters
fetch_singleRefers to the global tracking map to find the object and returns it.
parameters
fetch_multipleGiven a list of tracking IDs, this returns a submap of the global tracking map. The object that is returned also includes a field count, the number of objects actually found and returned.
parameters
app_specific_file_removalparameters
attempt_join_searchesAllows the application to work with an object when it is being added to the global iterable.
The method name is related to its original use, whereby a new object is added to the list of results of certain formal queries.
parameters
This class handles the arrivale of new data. In particular, this class takes in files that arrive in a directory that it watches.
Subclasses of this class may take in objects via other means of communication and make use of the method for adding and removing objects.
While wathing a disk directory is not assumed to be super fast, the planned placement for efficiency these classes provde is in accessing the objects once they are in memory. However, some ram-disk setups may provide abundant throughput.
The default field of the ID is _tracking.
constructorSets up the basic parameters and file operations.
parameters
startThis method starts watching a directory. It obtains a watcher stream from a node.js fsPromise.watch call by using the FileOperations intermediary call.
File watching is ongoing and ends when the application program shuts down.
This looks or 'change' and 'rename' events. In both cases, a file that passes any filters will be passed on to read_and_injest. If the file is being renamed, the previous version of the object will be removed.
no parameters
add_just_one_new_assetGiven that the parameter is an object, this method will call upon the registry to add it, calling add_just_one. If it is an array, it will assume that the first array element is an ID for the group. It will track the group by the ID, adding all the elements in the remaining of the array, and it will know of this group of elements when it is asked to be removed by maintaining a list of their IDs.
parameters
remove_just_one_assetIf the id passed is not a group tracking id, this will call upon the element manager to remove the element by its ID, calling remove_just_one. Otherwise, it will retrieve the group and remove each element.
parameters
read\_and\_injestThis loads the JSON file (using file operations) and then calls add_just_one_new_asset on the resulting object. Ignores failed load attempts.
parameters
pass_filterThis checks that the file name passed to it is a form indicating an actual file rather some systems intermediate file which may appear when watching a directory. It also checks that the extension of the file is the one being processed by the application; in this case, that is '.json'.
parameters
track_list_idThis is an ad hoc check to see if a symbol being used for an id mentions in particular that the object represented is a list. In particular, it looks for the prefix "TRACKLIST::".
parameters
This class deals with backing up data and then reloading it when the application restarts.
It is the intention that this class only deals with data that has already been added previously (not new data) and that has been stored by this class's instances themselves.
constructor Creates an instance of a DirectoryCache from extra-file-classes. This class acts as a proxy to the DirectoryCache instance, but provides a method for injesting the objects found in the directory that is being loaded.
parameters
item_injectorThe item injector inspects its one parameter in order to determine if it is an array to be added or just one object to be added to the registery. It then checks to see if its own list is an iterable or an Array object so that it may 'add' or 'push' accordingly.
Applications wishing for complex behavior may make sure that the applicatino wide iterable is carefully implemented with all the features it needs. This method will only ever call 'add' on that iterable.
parameters
load_directoryProvides the DirectoryCache loading call with parameters set up in the constructor of this class. Passes this class's item injector method as the method to be called by the DirectoryCache instance.
no parameters
backup_to_directoryProvides the DirectoryCache backup call with a file name constructor: ${obj._tracking}.json. Calles the DirectoryCache instance backup_to_directory with the ObjFileDirLoader directory from the constuctor on the application wide iterable also provided in the constuctor.
no parameters
saveSaves just one object using the naming conventions established by this class.
parameters
removeRemoves an object from from disk.
parameters
This is the default store that keeps track of application iterable implementations. Applications wishing to make use of it should override it and call it out in the conf of the Registry class that controls it. Use the file_list_class field in the configuration object.
constructorSets up basic member variables. These will be resets by set_lists
parameters
Assumes the application will supply the actual global iterable that this class proxies for.
parameters
Any id producing function that may use the objects passed if it whishes.
parameters
Returns the values of the internal map of this class. Similar to values of the Map object.
no parameters
Sorts the global iterable
parameters
Gets an object from the global iterable, similar to Map get
parameters
Add an object to the globa iterable as per the application implementation of this class. Should operate similarly to a Map add.
parameters
Deletes an object from the global iterable as per the application implementation of this class. Should operate similarly to a Set delete.
parameters
ordering_tableReturns the object that maps field names and created_date and update_date to lists sorted by those field keys.
no parameters
update_global_file_list_quotes_byWhen a FileList class is in use, this method will be called for sorting lists or iterables controlled by the class. It will be up to the implementation of this class to touch the score field of the objects.
no parameters
FAQs
Very common operations for managing and watching directories of entries and keeping entries in memory for searching
The npm package copious-registry receives a total of 3 weekly downloads. As such, copious-registry popularity was classified as not popular.
We found that copious-registry demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Multiple high-impact npm maintainers confirm they have been targeted in the same social engineering campaign that compromised Axios.
Security News
Axios compromise traced to social engineering, showing how attacks on maintainers can bypass controls and expose the broader software supply chain.
Security News
Node.js has paused its bug bounty program after funding ended, removing payouts for vulnerability reports but keeping its security process unchanged.