logyca-azure-storage-blob

LOGYCA public libraries

---

About us

• LOGYCA Company
• LOGYCA Youtube Channel
• 
• 
• 

---

LOGYCA public libraries: To interact with the blob container files in your Azure Storage account.

Source code | Package (PyPI) | Samples

---

To install the libraries

# Check SQLAlchemy dependency that is installed
pip install logyca_azure_storage_blob

How azcopy performs file consistency validation

Like azcopy, Microsoft's client for uploading large files to Blob, the md5 hash must be computed locally and uploaded to the blob properties along with the file, the blob service will commit this at the end of the process, returning an error if presents inconsistency in value.

Calculates the MD5 hash of the file content and saves it as the Content-MD5 property of the blob:

azcopy --put-md5 #

References:

- https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy

This library uses the concepts of the azcopy client for its implementation.

---

Test upload

bandwith movistar: upload/download: 300 mbps

Test 1, function: container_blob_upload_staging_blocks_commit with console output

file: 20240126084440-backup.dump
size: 2.03 GB
parameters: 
    - self.__chunk_size_bytes=4*1024*1024 # Defaults 4 MiB
Elapsed time: 00:05:22
Output:
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[42/522], Percentage[8.05%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[43/522], Percentage[8.24%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[44/522], Percentage[8.43%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[45/522], Percentage[8.62%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[46/522], Percentage[8.81%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[47/522], Percentage[9.00%]
    ...
    ...
    ...
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[517/522], Percentage[99.04%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[518/522], Percentage[99.23%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[519/522], Percentage[99.43%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[520/522], Percentage[99.62%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[521/522], Percentage[99.81%]
    Upload process: file_to_upload[20240126084440-backup.dump], Chunks[522/522], Percentage[100.00%]

Test 2, function: container_blob_upload_data_transfer_options no console output

file: 20240126084440-backup.dump
size: 2.03 GB        
parameters:
    - self.__chunk_size_bytes=4*1024*1024 # 4 MiB
    - self.__max_single_put_size=8*1024*1024 # 8 MiB.
    - max_concurrency=1
Elapsed time: 00:11:19
Unsupported formats: ---
Output: Not available

Test 3, function: container_blob_upload_data_transfer_options no console output

file: 20240126084440-backup.dump
size: 2.03 GB        
parameters:
    - self.__chunk_size_bytes=4*1024*1024 # 4 MiB
    - self.__max_single_put_size=8*1024*1024 # 8 MiB.
    - max_concurrency=2
Elapsed time: 00:03:50
Unsupported formats: ---
Output: Not available

Test 4, function: container_blob_upload_data_transfer_options no console output

file: 20240126084440-backup.dump
size: 2.03 GB        
parameters:
    - self.__chunk_size_bytes=4*1024*1024 # 4 MiB
    - self.__max_single_put_size=8*1024*1024 # 8 MiB.
    - max_concurrency=3
Elapsed time: 00:03:38
Unsupported formats: ---
Output: Not available

---

Semantic Versioning

logyca_azure_storage_blob < MAJOR >.< MINOR >.< PATCH >

• MAJOR: version when you make incompatible API changes
• MINOR: version when you add functionality in a backwards compatible manner
• PATCH: version when you make backwards compatible bug fixes

Definitions for releasing versions

• https://peps.python.org/pep-0440/

  • X.YaN (Alpha release): Identify and fix early-stage bugs. Not suitable for production use.
  • X.YbN (Beta release): Stabilize and refine features. Address reported bugs. Prepare for official release.
  • X.YrcN (Release candidate): Final version before official release. Assumes all major features are complete and stable. Recommended for testing in non-critical environments.
  • X.Y (Final release/Stable/Production): Completed, stable version ready for use in production. Full release for public use.

---

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Types of changes

• Added for new features.
• Changed for changes in existing functionality.
• Deprecated for soon-to-be removed features.
• Removed for now removed features.
• Fixed for any bug fixes.
• Security in case of vulnerabilities.

[0.0.1rc1] - 2024-05-24

Added

• First tests using pypi.org in develop environment.

[0.1.0] - 2024-05-24

Added

• Completion of testing and launch into production.

[0.1.2] - 2024-07-18

Changed

• Two filter sa container blob list are added. modified hours ago to return modified blobs after x hours and include_subfolders to return only blobs from the received folder list.