@leapfrogtechnology/sync-db - npm Package Compare versions

Comparing version 1.0.0-alpha.8 to 1.0.0-alpha.9

CHANGELOG.md

 # Change Log
 
+## [v1.0.0-experimental.8-6](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-6) (2020-02-03)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-experimental.8-5...v1.0.0-experimental.8-6)
+
+**Changes**
+
+- Avoid vendor specific defaults in the config [\#41](https://github.com/leapfrogtechnology/sync-db/pull/41) ([kabirbaidhya](https://github.com/kabirbaidhya))
+- Generate connection with port as number when using --generate-connections CLI arg [\#40](https://github.com/leapfrogtechnology/sync-db/pull/40) ([silwalanish](https://github.com/silwalanish))
+- Update dependencies - typescript & tslint [\#38](https://github.com/leapfrogtechnology/sync-db/pull/38) [[dependencies](https://github.com/leapfrogtechnology/sync-db/labels/dependencies)] ([kabirbaidhya](https://github.com/kabirbaidhya))
+- Config Injection - ability to inject dynamic configurations such that the running SQL transactions could access them [\#36](https://github.com/leapfrogtechnology/sync-db/pull/36) [[experimental](https://github.com/leapfrogtechnology/sync-db/labels/experimental)] [[feature](https://github.com/leapfrogtechnology/sync-db/labels/feature)] ([kabirbaidhya](https://github.com/kabirbaidhya))
+
+## [v1.0.0-experimental.8-5](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-5) (2020-02-01)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-experimental.8-4...v1.0.0-experimental.8-5)
+
+## [v1.0.0-experimental.8-4](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-4) (2020-01-19)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-experimental.8-3...v1.0.0-experimental.8-4)
+
+## [v1.0.0-experimental.8-3](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-3) (2020-01-19)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-experimental.8-2...v1.0.0-experimental.8-3)
+
+## [v1.0.0-experimental.8-2](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-2) (2020-01-19)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-experimental.8-1...v1.0.0-experimental.8-2)
+
+## [v1.0.0-experimental.8-1](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-experimental.8-1) (2020-01-19)
+[Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-alpha.8...v1.0.0-experimental.8-1)
+
+**Implemented enhancements:**
+
+- CLI arg to generate connections from the environment [\#33](https://github.com/leapfrogtechnology/sync-db/pull/33) [[cli](https://github.com/leapfrogtechnology/sync-db/labels/cli)] [[enhancement](https://github.com/leapfrogtechnology/sync-db/labels/enhancement)] [[feature](https://github.com/leapfrogtechnology/sync-db/labels/feature)] ([p0k8h](https://github.com/p0k8h))
+
+**Closed Issues**
+
+- New CLI arg to generate connections file using the environment variables [\#31](https://github.com/leapfrogtechnology/sync-db/issues/31) [[cli](https://github.com/leapfrogtechnology/sync-db/labels/cli)] [[feature](https://github.com/leapfrogtechnology/sync-db/labels/feature)] [[good first issue](https://github.com/leapfrogtechnology/sync-db/labels/good%20first%20issue)]
+- Add new example project based on postgres [\#25](https://github.com/leapfrogtechnology/sync-db/issues/25) [[documentation](https://github.com/leapfrogtechnology/sync-db/labels/documentation)] [[example](https://github.com/leapfrogtechnology/sync-db/labels/example)] [[good first issue](https://github.com/leapfrogtechnology/sync-db/labels/good%20first%20issue)] [[postgres](https://github.com/leapfrogtechnology/sync-db/labels/postgres)]
+
+**Changes**
+
+- Add example project for postgres [\#35](https://github.com/leapfrogtechnology/sync-db/pull/35) [[example](https://github.com/leapfrogtechnology/sync-db/labels/example)] [[postgres](https://github.com/leapfrogtechnology/sync-db/labels/postgres)] ([Prabeshpd](https://github.com/Prabeshpd))
+
 ## [v1.0.0-alpha.8](https://github.com/leapfrogtechnology/sync-db/tree/v1.0.0-alpha.8) (2019-12-18)
@@ -16,3 +54,2 @@ [Full Changelog](https://github.com/leapfrogtechnology/sync-db/compare/v1.0.0-alpha.7...v1.0.0-alpha.8)
 - Setup docker and docker-compose configuration for the node-app-mssql example [\#23](https://github.com/leapfrogtechnology/sync-db/pull/23) [[enhancement](https://github.com/leapfrogtechnology/sync-db/labels/enhancement)] ([silwalanish](https://github.com/silwalanish))
-- Enable passing connection instance as well as connection config for synchronization [\#10](https://github.com/leapfrogtechnology/sync-db/pull/10) [[documentation](https://github.com/leapfrogtechnology/sync-db/labels/documentation)] [[enhancement](https://github.com/leapfrogtechnology/sync-db/labels/enhancement)] [[feature](https://github.com/leapfrogtechnology/sync-db/labels/feature)] ([SafalPandey](https://github.com/SafalPandey))
 
@@ -65,2 +102,6 @@ **Bug Fixes**
 
+**Implemented enhancements:**
+
+- Enable passing connection instance as well as connection config for synchronization [\#10](https://github.com/leapfrogtechnology/sync-db/pull/10) [[documentation](https://github.com/leapfrogtechnology/sync-db/labels/documentation)] [[enhancement](https://github.com/leapfrogtechnology/sync-db/labels/enhancement)] [[feature](https://github.com/leapfrogtechnology/sync-db/labels/feature)] ([SafalPandey](https://github.com/SafalPandey))
+
 **Changes**
@@ -67,0 +108,0 @@

lib/config.d.ts

@@ -10,2 +10,8 @@ import SyncConfig from './domain/SyncConfig';
 /**
+ * Validate the loaded configuration.
+ *
+ * @param {SyncConfig} config
+ */
+export declare function validate(config: SyncConfig): void;
+/**
  * Resolve database connections.
@@ -16,1 +22,7 @@ *
 export declare function resolveConnections(): Promise<ConnectionConfig[]>;
+/**
+ * Resolve database connections from Env.
+ *
+ * @returns {ConnectionConfig[]}
+ */
+export declare function resolveConnectionsFromEnv(): ConnectionConfig[];

lib/config.js

@@ -8,2 +8,4 @@ "use strict";
 const fs = require("./util/fs");
+const types_1 = require("./util/types");
+const configInjection_1 = require("./services/configInjection");
 const constants_1 = require("./constants");
@@ -18,10 +20,25 @@ /**
     const filename = path.resolve(process.cwd(), constants_1.CONFIG_FILENAME);
-    const migrations = await yaml.load(filename);
+    const loadedConfig = (await yaml.load(filename));
     logger_1.log('Resolved sync config file.');
-    const loaded = ramda_1.mergeDeepRight(constants_1.DEFAULT_CONFIG, migrations);
-    // TODO: Validate the loaded config.
-    return loaded;
+    const loaded = ramda_1.mergeDeepRight(constants_1.DEFAULT_CONFIG, loadedConfig);
+    validate(loaded);
+    return Object.assign(Object.assign({}, loaded), { injectedConfig: Object.assign(Object.assign({}, loaded.injectedConfig), { vars: configInjection_1.prepareInjectionConfigVars(loaded.injectedConfig.vars) }) });
 }
 exports.loadConfig = loadConfig;
 /**
+ * Validate the loaded configuration.
+ *
+ * @param {SyncConfig} config
+ */
+function validate(config) {
+    const { injectedConfig } = config;
+    // Shouldn't reach under here unless the user has mismatched the value.
+    if (!injectedConfig.vars || !types_1.isObject(injectedConfig.vars)) {
+        throw new Error('Invalid configuration value for `injectedConfig.vars`.');
+    }
+    // TODO: Validate the remaining loaded config.
+    // Throw error if validation fails.
+}
+exports.validate = validate;
+/**
  * Resolve database connections.
@@ -43,1 +60,34 @@ *
 exports.resolveConnections = resolveConnections;
+/**
+ * Validate connection keys.
+ *
+ * @param {string[]} keys
+ * @returns {void}
+ */
+function validateConnections(keys) {
+    const missingVars = keys.filter(key => !process.env[key]);
+    if (missingVars.length) {
+        throw new Error('Following environment variables were not set: ' + missingVars.join(', '));
+    }
+}
+/**
+ * Resolve database connections from Env.
+ *
+ * @returns {ConnectionConfig[]}
+ */
+function resolveConnectionsFromEnv() {
+    validateConnections(constants_1.ENV_KEYS);
+    const connection = {
+        client: process.env.DB_CLIENT,
+        host: process.env.DB_HOST,
+        port: process.env.DB_PORT ? +(process.env.DB_PORT) : null,
+        user: process.env.DB_USERNAME,
+        password: process.env.DB_PASSWORD,
+        database: process.env.DB_NAME,
+        options: {
+            encrypt: process.env.DB_ENCRYPTION === 'true'
+        }
+    };
+    return [connection];
+}
+exports.resolveConnectionsFromEnv = resolveConnectionsFromEnv;

lib/Connection.d.ts

@@ -56,2 +56,14 @@ import * as Knex from 'knex';
     /**
+     * Returns the underlying Knex instance.
+     *
+     * @returns {Knex}
+     */
+    getInstance(): Knex;
+    /**
+     * Gets access to the schema builder.
+     *
+     * @returns {Knex.SchemaBuilder}
+     */
+    schema(): Knex.SchemaBuilder;
+    /**
      * Returns connection config.
@@ -58,0 +70,0 @@ *

lib/Connection.js

@@ -70,2 +70,18 @@ "use strict";
     /**
+     * Returns the underlying Knex instance.
+     *
+     * @returns {Knex}
+     */
+    getInstance() {
+        return this.instance;
+    }
+    /**
+     * Gets access to the schema builder.
+     *
+     * @returns {Knex.SchemaBuilder}
+     */
+    schema() {
+        return this.instance.schema;
+    }
+    /**
      * Returns connection config.
@@ -72,0 +88,0 @@ *

lib/constants.d.ts

 import SyncConfig from './domain/SyncConfig';
 export declare const CONFIG_FILENAME = "sync-db.yml";
 export declare const CONNECTIONS_FILENAME = "connections.sync-db.json";
+export declare const INJECTED_CONFIG_TABLE = "__sync_db_injected_config";
 export declare const DEFAULT_CONFIG: SyncConfig;
+export declare const ENV_KEYS: string[];

lib/constants.js

@@ -7,2 +7,3 @@ "use strict";
 exports.CONNECTIONS_FILENAME = 'connections.sync-db.json';
+exports.INJECTED_CONFIG_TABLE = '__sync_db_injected_config';
 exports.DEFAULT_CONFIG = {
@@ -14,3 +15,7 @@ basePath: path.resolve(process.cwd(), 'src/sql'),
         post_sync: []
+    },
+    injectedConfig: {
+        vars: {}
     }
 };
+exports.ENV_KEYS = ['DB_HOST', 'DB_PASSWORD', 'DB_NAME', 'DB_USERNAME', 'DB_PORT', 'DB_CLIENT', 'DB_ENCRYPTION'];

lib/domain/SyncConfig.d.ts

@@ -0,1 +1,2 @@
+import Mapping from './Mapping';
 /**
@@ -11,3 +12,6 @@ * Interface for synchronization configuration sycn-db.yml.
     };
+    injectedConfig: {
+        vars: Mapping<string>;
+    };
 }
 export default SyncConfig;

lib/migrator.js

@@ -6,2 +6,3 @@ "use strict";
 const sqlRunner = require("./sqlRunner");
+const configInjection = require("./services/configInjection");
 /**
@@ -21,2 +22,5 @@ * Migrate SQL on a database.
         await connection.transaction(async (t) => {
+            // Config Injection: Setup
+            // This will setup a config table (temporary and accessible only to this transaction).
+            await configInjection.setup(t, config);
             if (preMigrationScripts.length > 0) {
@@ -38,2 +42,5 @@ const preHookScripts = await sqlRunner.resolveFiles(basePath, preMigrationScripts);
             }
+            // Config Injection: Cleanup
+            // Cleans up the injected config and the table.
+            await configInjection.cleanup(t, config);
             logDb('Finished setup');
@@ -73,3 +80,16 @@ });
     const connArr = Array.isArray(conn) ? conn : [conn];
-    const connections = connArr.map(con => {
+    const connections = mapToConnectionInstances(connArr);
+    const promises = connections.map(connection => syncDatabase(connection, config));
+    await Promise.all(promises);
+    logger_1.log('All synchronized');
+}
+exports.synchronize = synchronize;
+/**
+ * Map connection configuration list to the connection instances.
+ *
+ * @param {((ConnectionConfig | Knex)[])} connectionList
+ * @returns {Connection[]}
+ */
+function mapToConnectionInstances(connectionList) {
+    return connectionList.map(con => {
         if (Connection_1.default.isKnexInstance(con)) {
@@ -82,7 +102,3 @@ logger_1.log(`Received connection instance to database: ${con.client.config.connection.database}`);
     });
-    const promises = connections.map(connection => syncDatabase(connection, config));
-    await Promise.all(promises);
-    logger_1.log('All synchronized');
 }
-exports.synchronize = synchronize;
 /**
@@ -89,0 +105,0 @@ * Synchronize a specific database.

lib/SyncDb.d.ts

@@ -14,2 +14,3 @@ import { Command } from '@oclif/command';
         force: import("@oclif/parser/lib/flags").IBooleanFlag<boolean>;
+        'generate-connections': import("@oclif/parser/lib/flags").IBooleanFlag<boolean>;
     };
@@ -16,0 +17,0 @@ /**

lib/SyncDb.js

@@ -5,2 +5,3 @@ "use strict";
 const logger_1 = require("./logger");
+const syncDb_1 = require("./services/syncDb");
 const config_1 = require("./config");
@@ -22,2 +23,6 @@ /**
         try {
+            await syncDb_1.handleFlags(parsedFlags, {
+                info: this.log,
+                error: this.error
+            });
             const config = await config_1.loadConfig();
@@ -43,4 +48,5 @@ const connections = await config_1.resolveConnections();
     help: command_1.flags.help({ char: 'h', description: 'Print help information' }),
-    force: command_1.flags.boolean({ char: 'f', description: 'Force synchronization' })
+    force: command_1.flags.boolean({ char: 'f', description: 'Force synchronization' }),
+    'generate-connections': command_1.flags.boolean({ char: 'c', description: 'Generate connections' })
 };
 exports.default = SyncDb;

lib/util/fs.d.ts

@@ -22,1 +22,16 @@ /**
 export declare function glob(pathName: string): Promise<string[]>;
+/**
+ * Write contents to file.
+ *
+ * @param {string} filepath
+ * @param {any} data
+ * @returns {Promise<void>}
+ */
+export declare function write(filepath: string, data: any): Promise<void>;
+/**
+ * Remove the file.
+ *
+ * @param {string} filepath
+ * @returns {Promise<void>}
+ */
+export declare function remove(filepath: string): Promise<void>;

lib/util/fs.js

@@ -50,1 +50,36 @@ "use strict";
 exports.glob = glob;
+/**
+ * Write contents to file.
+ *
+ * @param {string} filepath
+ * @param {any} data
+ * @returns {Promise<void>}
+ */
+function write(filepath, data) {
+    return new Promise((resolve, reject) => {
+        fs.writeFile(filepath, data, err => {
+            if (err) {
+                return reject(err);
+            }
+            return resolve();
+        });
+    });
+}
+exports.write = write;
+/**
+ * Remove the file.
+ *
+ * @param {string} filepath
+ * @returns {Promise<void>}
+ */
+function remove(filepath) {
+    return new Promise((resolve, reject) => {
+        fs.unlink(filepath, err => {
+            if (err) {
+                return reject(err);
+            }
+            return resolve();
+        });
+    });
+}
+exports.remove = remove;

package.json

 {
   "name": "@leapfrogtechnology/sync-db",
   "description": "Command line utility to synchronize and version control relational database objects across databases",
-  "version": "1.0.0-alpha.8",
+  "version": "1.0.0-alpha.9",
   "license": "MIT",
@@ -39,2 +39,4 @@ "main": "lib/index.js",
     "prepack": "yarn build",
+    "lint": "tslint -c tslint.json 'src/**/*.{ts,js}' 'test/**/*.{ts,js}'",
+    "lint:fix": "tslint --fix -c tslint.json 'src/**/*.{ts,js}' 'test/**/*.{ts,js}'",
     "test": "nyc --extension .ts mocha --forbid-only \"test/**/*.test.ts\"",
@@ -60,3 +62,4 @@ "test:coverage": "nyc mocha --forbid-only \"test/**/*.test.ts\"",
     "@oclif/tslint": "^3",
-    "@types/chai": "^4",
+    "@types/chai": "^4.2.7",
+    "@types/chai-as-promised": "^7.1.2",
     "@types/debug": "^4.1.4",
@@ -67,10 +70,11 @@ "@types/mocha": "^5",
     "@types/yamljs": "^0.2.30",
-    "chai": "^4",
-    "codecov": "^3.6.1",
+    "chai": "^4.2.0",
+    "chai-as-promised": "^7.1.1",
+    "codecov": "^3.5.0",
     "mocha": "^5",
     "nyc": "^14.1.1",
     "ts-node": "^8",
-    "tslint": "^5",
+    "tslint": "^6.0.0",
     "tslint-config-leapfrog": "^1.0.3",
-    "typescript": "^3.7.2"
+    "typescript": "^3.7.5"
   },
@@ -77,0 +81,0 @@ "engines": {

README.md

@@ -59,5 +59,6 @@ # sync-db
 OPTIONS
-  -f, --force    Force synchronization
-  -h, --help     Print help information
-  -v, --version  Print version
+  -f, --force                 Force synchronization
+  -h, --help                  Print help information
+  -v, --version               Print version
+  -c, --generate-connections  Generate connections.sync-db.json file
 ```
@@ -176,2 +177,3 @@
 2. [Node MSSQL Programmatic Usage Sample](examples/node-mssql-programmatic-use)
+3. [Node PostgreSQL Sample](examples/node-app-pg)
 
@@ -178,0 +180,0 @@ ## Changelog

New alerts

Environment variable access

Supply chain risk

Package accesses environment variables, which may be a sign of credential stuffing or data theft.

Found 8 instances in 1 package

Improved metrics

Total package byte prevSize

62415

increased by43.57%

Number of package files

58

increased by31.82%

Lines of code

1279

increased by58.29%

Number of lines in readme file

189

increased by1.07%

Worsened metrics

Dev dependency count

18

increased by12.5%

Number of low supply chain risk alerts

12

increased by500%

No dependency changes