mayan-importer

=========== Description

Mayan EDMS app to migrate files from external sources.

======= License

This project is open sourced under the Apache 2.0 License_.

.. _Apache 2.0 License: https://gitlab.com/mayan-edms/importer/raw/master/LICENSE

============ Installation

#. Install from PyPI in the same virtualenv where you installed Mayan EDMS. Or if using the Docker image, pass mayan-importer to the MAYAN_PIP_INSTALLS environment variable.

.. code-block:: console

   pip install mayan-importer

#. Add importer to the COMMON_EXTRA_APPS setting, either as an environment variable, from a Python settings modules, or from the UI via the config.yaml configuration file.

Python settings module example:

.. code-block:: console

   INSTALLED_APPS += (
       'importer',
   )

#. Run the migrations for the app (not required for the Docker image):

.. code-block:: console

   mayan-edms.py migrate

============ Requirements

• dropbox - Dropbox Python SDK (https://www.dropbox.com/developers/documentation/python#install)
• Mayan EDMS version 3.4

===== Usage

• Create an app in Dropbox's App Console (https://www.dropbox.com/developers/apps/).
• Generate an Access Token with no expiration.
• Create a credential instance in the Mayan EDMS Importer app and enter the Access Token.
• Create an Import Setup that will filter the files to fetch from Dropbox.
• Click the "Populate" button and check that the item count is correct.
• Click the "Process" button to start the import process.

3.3 (2024-05-27)

• Support OAuth2 refresh tokens.
• Re-add support for computer team administrator API ID.
• Split views.
• Update method names.

3.2 (2024-05-04)

• Use Dropzone widget.
• Add migration reversing functions.
• Show unknown state labels.

3.1.1 (2024-05-04)

• Fix filer load.

3.1 (2024-05-03)

• Improve navigation.
• Improve processing condition logic.
• Split modules.
• Move common state code into a mixin.
• Add fieldsets.

3.0 (2024-04-01)

• Refactor app.
• Remove import setup action.
• Use source metadata to store import metadata.
• Don't deserialize import item source column.
• Use upstream backend, dynamic forms, credentials mixins, model mixins, error log.
• Use fieldsets.
• Bring code standards up to series 4.5.
• Support processing and deleting individual setup item entries.
• Improve import setup and import setup item state handling.
• Improve navigation, add return links.

2.5.2 (2024-02-22)

• Fix credential get_backend usage.
• Code styles updates.
• Add view icons.

2.5.1 (2024-02-20)

• Fix OAuthAccessToken credential use to CredentialBackendAccessToken.

2.5 (2024-02-20)

• Update import for Mayan EDMS version 4.5.

2.4 (2024-01-24)

• Update import for Mayan EDMS version 4.4.

2.3.1 (2023-06-12)

• Require credentials version 1.4.1.
• Support new event manager module location.

2.3 (2022-09-08)

• Split models module.
• Support item time buffer.

2.2 (2022-09-03)

• Add support for editing import setup items.
• Add new events.
• Expand tests.
• Sort import setup items by identifier.

2.1 (2022-08-19)

• Improve test rig. Add test project using upstream Mayan. Add make file targets and development settings.
• Fix API view action method to accept the new serializer argument.
• Fix ModelFiler deleting all instance. The queryset of instances to delete now uses filter.

2.0 (2022-08-08)

• Remove the view ImportSetupItemSaveDownloadView.
• Reorganize menus and navigation.
• Rename load, prepare, and save links.
• Remove success URL redirection from views.
• Update save to use the Download file and messaging systems.

1.20 (2022-08-06)

• Update SharedUploadedFile model loading. App changed from common to storage.

1.19 (2022-01-25)

• Update for Mayan EDMS 4.2.
• Require credentials 1.4.

1.18 (2021-06-17)

• Use a dedicated worker.

1.17 (2021-06-15)

• Use document label as filename.

1.16 (2021-06-12)

• Use the new document file task to speed up imports.
• Rename the "Populate" strings to "Prepare" for clarity.

1.15 (2021-06-08)

• Remove document creation transaction.

1.14 (2021-06-07)

• Update new document creation interface.

1.13 (2021-06-03)

• Update for version 4.0.

1.12.1 (2021-02-23)

• Require credentials 1.2 version.
• Fix backend SourceColumn label.

1.12 (2020-12-27)

• Add database backed logging to import setups.
• Add events for tracking start and end of the import setup population and processing.
• Improve import setup state tracking.
• Improve tests and add event testing.

1.11 (2020-12-17)

• Add API.

1.10.2 (2020-10-05)

• Fix context variable typo.

1.10.1 (2020-10-05)

• Revert usage of task_upload_new_version. Version processing is now done as part of the same code context as the import.

1.10.0 (2020-10-05)

• Support disabling import setup state update to workaround overloaded databases.
• Keep track of the documents created from an import setup item.
• Use document task_upload_new_version to process the document version as a separate code context.
• Use a queryset iterator when launching the processing tasks of the import setup items to save memory.

1.9.0 (2020-10-04)

• Add cabinet action. This action creates a cabinet structure from a path like value.
• Execute only enabled actions.
• Rename the modules of the test importer and import setup actions.
• Fix grammatical errors.

1.8.0 (2020-10-04)

• Index the import setup item state field.
• Check the state of the import setup item before processing.

1.7.0 (2020-10-03)

• Remove metadata mapping field.
• Add import setup actions. These are execute after the document is imported. Add an import setup action to assign a metadata value from a template.
• Backport the templating_tags from version 3.5.

1.6.0 (2020-09-30)

• Fix "off-by-one" process size issue.
• Delete shared uploaded file after creating document to keep the shared_files folder size small.
• Update import setup clear, populating, and process views to work on single or multiple items.

1.5.0 (2020-09-25)

• Add model filer to load and save models from and to CSV.

1.4.0 (2020-09-24)

• Add import setup item completion event.
• Commit the import setup executed event when the execute method is called instead of the get get_backend_intsance.
• Add more tests.
• Rename fields and models for clarity. Item metadata field renamed to 'data' to avoid confusion with document metadata.
• Event, permission, and action named "Execute" is now "Process".
• Filter items by regular expressions during population and also during processing.
• Add team_admin_id field to the Dropbox backend to avoid an extra API call for each item to be imported.
• Multiple values are now cached for higher performance.
• Process and clear links are disabled for empty import setups.
• Smarter backend import error exclusion.
• Automatic backend keyword argument setup from dynamic fields.
• Support import item fields as attributes or dictionary keys.

1.3.0 (2020-09-23)

• Add support to process individual items.
• Add background task support for individual items. Each item is now processed independently and in parallel.
• Add thousand comma separator to the progress summary column.

1.2.0 (2020-09-22)

• Support Dropbox Team admin access.
• Add import setup state field.
• Add import setup item list view.
• Add import setup item delete view.

1.1.0 (2020-09-08)

• Convert app into a general import app. Dropbox code moved into a separate importers module.

1.0.2 (2020-09-07)

• Update absolute imports to self.

1.0.1 (2020-09-07)

• Update absolute imports to the Credentials app.

1.0.0 (2020-09-01)

• Initial release