dlt-utils-lib - npm Package Compare versions

Comparing version 1.0.7 to 1.0.8

dlt_utils_lib.egg-info/PKG-INFO

 Metadata-Version: 2.1
 Name: dlt-utils-lib
-Version: 1.0.7
+Version: 1.0.8
 Summary: UNKNOWN
@@ -5,0 +5,0 @@ Home-page: UNKNOWN

dlt_utils/main_cdc_utils.py

@@ -20,2 +20,3 @@ from pyspark.sql.functions import col, expr, coalesce, lit, when, current_timestamp
         partition_col = table['partition_col']
+        enable_truncate = table['enable_truncate']
         s3_files_path = f"{source_schema}/{source_table_name}/"
@@ -46,3 +47,4 @@ bronze_table_path = f"{bucket_name}/{bronze_directory}/{s3_files_path}"
                                 partitions=partition_col,
-                                exclude_columns=exclude_columns)
+                                exclude_columns=exclude_columns,
+                                enable_truncate=enable_truncate)
         
@@ -79,3 +81,3 @@
 # Create silver streaming data
-def silver_streaming_process(dlt, table_name: str, keys: list, partitions: dict, exclude_columns: list):
+def silver_streaming_process(dlt, table_name: str, keys: list, partitions: dict, exclude_columns: list, enable_truncate: bool = False):
     dlt.create_streaming_table(
@@ -97,5 +99,5 @@ name=table_name,
         apply_as_deletes=expr("Op = 'D'"),
-        apply_as_truncates=expr("Op = 'T'"),
+        apply_as_truncates=expr("Op = 'T'") if enable_truncate else None,
         except_column_list=["Op", "_rescued_data"] + exclude_columns,
         stored_as_scd_type=1
     )

PKG-INFO

 Metadata-Version: 2.1
 Name: dlt_utils_lib
-Version: 1.0.7
+Version: 1.0.8
 Summary: UNKNOWN
@@ -5,0 +5,0 @@ Home-page: UNKNOWN

Alert delta unavailable

Currently unable to show alert delta for PyPI packages.

Improved metrics

Total package byte prevSize

12971

1.38%

Lines of code

247

0.82%