Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

@backstage/config-loader

Package Overview
Dependencies
Maintainers
4
Versions
894
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

@backstage/config-loader - npm Package Compare versions

Comparing version 0.0.0-nightly-20209921112 to 0.0.0-nightly-20210262290

98

CHANGELOG.md
# @backstage/config-loader
## 0.0.0-nightly-20209921112
## 0.0.0-nightly-20210262290
### Minor Changes
- 4295a24: Added support for new shorthand when defining secrets, where `$env: ENV` can be used instead of `$secret: { env: ENV }` etc.
- 789de4b: Removed support for the deprecated `$data` placeholder.
- 789de4b: Enable further processing of configuration files included using the `$include` placeholder. Meaning that for example for example `$env` includes will be processed as usual in included files.
### Patch Changes
- 789de4b: Added support for environment variable substitutions in string configuration values using a `${VAR}` placeholder. All environment variables must be available, or the entire expression will be evaluated to `undefined`. To escape a substitution, use `${...}`, which will end up as `${...}`.
For example:
```yaml
app:
baseUrl: https://${BASE_HOST}
```
## 0.4.1
### Patch Changes
- ad5c56fd9: Deprecate `$data` and replace it with `$include` which allows for any type of json value to be read from external files. In addition, `$include` can be used without a path, which causes the value at the root of the file to be loaded.
Most usages of `$data` can be directly replaced with `$include`, except if the referenced value is not a string, in which case the value needs to be changed. For example:
```yaml
# app-config.yaml
foo:
$data: foo.yaml#myValue # replacing with $include will turn the value into a number
$data: bar.yaml#myValue # replacing with $include is safe
# foo.yaml
myValue: 0xf00
# bar.yaml
myValue: bar
```
## 0.4.0
### Minor Changes
- 4e7091759: Fix typo of "visibility" in config schema reference
If you have defined a config element named `visiblity`, you
will need to fix the spelling to `visibility`. For more info,
see https://backstage.io/docs/conf/defining#visibility.
### Patch Changes
- b4488ddb0: Added a type alias for PositionError = GeolocationPositionError
## 0.3.0
### Minor Changes
- 1722cb53c: Added support for loading and validating configuration schemas, as well as declaring config visibility through schemas.
The new `loadConfigSchema` function exported by `@backstage/config-loader` allows for the collection and merging of configuration schemas from all nearby dependencies of the project.
A configuration schema is declared using the `https://backstage.io/schema/config-v1` JSON Schema meta schema, which is based on draft07. The only difference to the draft07 schema is the custom `visibility` keyword, which is used to indicate whether the given config value should be visible in the frontend or not. The possible values are `frontend`, `backend`, and `secret`, where `backend` is the default. A visibility of `secret` has the same scope at runtime, but it will be treated with more care in certain contexts, and defining both `frontend` and `secret` for the same value in two different schemas will result in an error during schema merging.
Packages that wish to contribute configuration schema should declare it in a root `"configSchema"` field in `package.json`. The field can either contain an inlined JSON schema, or a relative path to a schema file. Schema files can be in either `.json` or `.d.ts` format.
TypeScript configuration schema files should export a single `Config` type, for example:
```ts
export interface Config {
app: {
/**
* Frontend root URL
* @visibility frontend
*/
baseUrl: string;
};
}
```
## 0.2.0
### Minor Changes
- 8c2b76e45: **BREAKING CHANGE**
The existing loading of additional config files like `app-config.development.yaml` using APP_ENV or NODE_ENV has been removed.
Instead, the CLI and backend process now accept one or more `--config` flags to load config files.
Without passing any flags, `app-config.yaml` and, if it exists, `app-config.local.yaml` will be loaded.
If passing any `--config <path>` flags, only those files will be loaded, **NOT** the default `app-config.yaml` one.
The old behaviour of for example `APP_ENV=development` can be replicated using the following flags:
```bash
--config ../../app-config.yaml --config ../../app-config.development.yaml
```
- ce5512bc0: Added support for new shorthand when defining secrets, where `$env: ENV` can be used instead of `$secret: { env: ENV }` etc.

592

dist/index.cjs.js

@@ -5,102 +5,17 @@ 'use strict';

function _interopDefault (ex) { return (ex && (typeof ex === 'object') && 'default' in ex) ? ex['default'] : ex; }
var yaml2 = require('yaml');
var path = require('path');
var Ajv = require('ajv');
var mergeAllOf = require('json-schema-merge-allof');
var config = require('@backstage/config');
var fs = require('fs-extra');
var fs__default = _interopDefault(fs);
var yaml2 = _interopDefault(require('yaml'));
var yup = require('yup');
var typescriptJsonSchema = require('typescript-json-schema');
async function resolveStaticConfig(options) {
const filePaths = [
`app-config.yaml`,
`app-config.local.yaml`,
`app-config.${options.env}.yaml`,
`app-config.${options.env}.local.yaml`
];
const resolvedPaths = [];
for (const rootPath of options.rootPaths) {
for (const filePath of filePaths) {
const path2 = path.resolve(rootPath, filePath);
if (await fs.pathExists(path2)) {
resolvedPaths.push(path2);
}
}
}
return resolvedPaths;
}
function _interopDefaultLegacy (e) { return e && typeof e === 'object' && 'default' in e ? e : { 'default': e }; }
function isObject(obj) {
if (typeof obj !== "object") {
return false;
} else if (Array.isArray(obj)) {
return false;
}
return obj !== null;
}
var yaml2__default = /*#__PURE__*/_interopDefaultLegacy(yaml2);
var Ajv__default = /*#__PURE__*/_interopDefaultLegacy(Ajv);
var mergeAllOf__default = /*#__PURE__*/_interopDefaultLegacy(mergeAllOf);
var fs__default = /*#__PURE__*/_interopDefaultLegacy(fs);
async function readConfigFile(filePath, ctx) {
const configYaml = await ctx.readFile(filePath);
const config2 = yaml2.parse(configYaml);
const context = path.basename(filePath);
async function transform(obj, path2) {
if (ctx.skip(path2)) {
return void 0;
}
if (typeof obj !== "object") {
return obj;
} else if (obj === null) {
return void 0;
} else if (Array.isArray(obj)) {
const arr = new Array();
for (const [index, value] of obj.entries()) {
const out2 = await transform(value, `${path2}[${index}]`);
if (out2 !== void 0) {
arr.push(out2);
}
}
return arr;
}
if ("$secret" in obj) {
console.warn(`Deprecated secret declaration at '${path2}' in '${context}', use $env, $file, etc. instead`);
if (!isObject(obj.$secret)) {
throw TypeError(`Expected object at secret ${path2}.$secret`);
}
try {
return await ctx.readSecret(path2, obj.$secret);
} catch (error) {
throw new Error(`Invalid secret at ${path2}: ${error.message}`);
}
}
const [secretKey] = Object.keys(obj).filter((key) => key.startsWith("$"));
if (secretKey) {
if (Object.keys(obj).length !== 1) {
throw new Error(`Secret key '${secretKey}' has adjacent keys at ${path2}`);
}
try {
return await ctx.readSecret(path2, {
[secretKey.slice(1)]: obj[secretKey]
});
} catch (error) {
throw new Error(`Invalid secret at ${path2}: ${error.message}`);
}
}
const out = {};
for (const [key, value] of Object.entries(obj)) {
if (value !== void 0) {
const result = await transform(value, `${path2}.${key}`);
if (result !== void 0) {
out[key] = result;
}
}
}
return out;
}
const finalConfig = await transform(config2, "");
if (!isObject(finalConfig)) {
throw new TypeError("Expected object at config root");
}
return {data: finalConfig, context};
}
const ENV_PREFIX = "APP_CONFIG_";

@@ -156,104 +71,434 @@ const CONFIG_KEY_PART_PATTERN = /^[a-z][a-z0-9]*(?:[-_][a-z][a-z0-9]*)*$/i;

const secretLoaderSchemas = {
file: yup.object({
file: yup.string().required()
}),
env: yup.object({
env: yup.string().required()
}),
data: yup.object({
data: yup.string().required()
})
};
const secretSchema = yup.lazy((value) => {
if (typeof value !== "object" || value === null) {
return yup.object().required().label("secret");
function isObject(obj) {
if (typeof obj !== "object") {
return false;
} else if (Array.isArray(obj)) {
return false;
}
const loaderTypes = Object.keys(secretLoaderSchemas);
for (const key of loaderTypes) {
if (key in value) {
return secretLoaderSchemas[key];
return obj !== null;
}
async function applyConfigTransforms(initialDir, input, transforms) {
async function transform(inputObj, path, baseDir) {
var _a;
let obj = inputObj;
let dir = baseDir;
for (const tf of transforms) {
try {
const result = await tf(inputObj, baseDir);
if (result.applied) {
if (result.value === void 0) {
return void 0;
}
obj = result.value;
dir = (_a = result.newBaseDir) != null ? _a : dir;
break;
}
} catch (error) {
throw new Error(`error at ${path}, ${error.message}`);
}
}
if (typeof obj !== "object") {
return obj;
} else if (obj === null) {
return void 0;
} else if (Array.isArray(obj)) {
const arr = new Array();
for (const [index, value] of obj.entries()) {
const out2 = await transform(value, `${path}[${index}]`, dir);
if (out2 !== void 0) {
arr.push(out2);
}
}
return arr;
}
const out = {};
for (const [key, value] of Object.entries(obj)) {
if (value !== void 0) {
const result = await transform(value, `${path}.${key}`, dir);
if (result !== void 0) {
out[key] = result;
}
}
}
return out;
}
throw new yup.ValidationError(`Secret must contain one of '${loaderTypes.join("', '")}'`, value, "$secret");
});
const dataSecretParser = {
const finalData = await transform(input, "", initialDir);
if (!isObject(finalData)) {
throw new TypeError("expected object at config root");
}
return finalData;
}
const includeFileParser = {
".json": async (content) => JSON.parse(content),
".yaml": async (content) => yaml2.parse(content),
".yml": async (content) => yaml2.parse(content)
".yaml": async (content) => yaml2__default['default'].parse(content),
".yml": async (content) => yaml2__default['default'].parse(content)
};
async function readSecret(data, ctx) {
const secret = secretSchema.validateSync(data, {strict: true});
if ("file" in secret) {
return ctx.readFile(secret.file);
}
if ("env" in secret) {
return ctx.env[secret.env];
}
if ("data" in secret) {
const url = "path" in secret ? `${secret.data}#${secret.path}` : secret.data;
const [filePath, dataPath] = url.split(/#(.*)/);
if (!dataPath) {
throw new Error(`Invalid format for data secret value, must be of the form <filepath>#<datapath>, got '${url}'`);
function createIncludeTransform(env, readFile) {
return async (input, baseDir) => {
if (!isObject(input)) {
return {applied: false};
}
const ext = path.extname(filePath);
const parser = dataSecretParser[ext];
if (!parser) {
throw new Error(`No data secret parser available for extension ${ext}`);
const [includeKey] = Object.keys(input).filter((key) => key.startsWith("$"));
if (includeKey) {
if (Object.keys(input).length !== 1) {
throw new Error(`include key ${includeKey} should not have adjacent keys`);
}
} else {
return {applied: false};
}
const content = await ctx.readFile(filePath);
const parts = dataPath.split(".");
let value = await parser(content);
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join(".");
throw new Error(`Value is not an object at ${errPath} in ${filePath}`);
const includeValue = input[includeKey];
if (typeof includeValue !== "string") {
throw new Error(`${includeKey} include value is not a string`);
}
switch (includeKey) {
case "$file":
try {
const value = await readFile(path.resolve(baseDir, includeValue));
return {applied: true, value};
} catch (error) {
throw new Error(`failed to read file ${includeValue}, ${error}`);
}
case "$env":
try {
return {applied: true, value: await env(includeValue)};
} catch (error) {
throw new Error(`failed to read env ${includeValue}, ${error}`);
}
case "$include": {
const [filePath, dataPath] = includeValue.split(/#(.*)/);
const ext = path.extname(filePath);
const parser = includeFileParser[ext];
if (!parser) {
throw new Error(`no configuration parser available for included file ${filePath}`);
}
const path2 = path.resolve(baseDir, filePath);
const content = await readFile(path2);
const newBaseDir = path.dirname(path2);
const parts = dataPath ? dataPath.split(".") : [];
let value;
try {
value = await parser(content);
} catch (error) {
throw new Error(`failed to parse included file ${filePath}, ${error}`);
}
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join(".");
throw new Error(`value at '${errPath}' in included file ${filePath} is not an object`);
}
value = value[part];
}
return {
applied: true,
value,
newBaseDir: newBaseDir !== baseDir ? newBaseDir : void 0
};
}
value = value[part];
default:
throw new Error(`unknown include ${includeKey}`);
}
return String(value);
}
throw new Error("Secret was left unhandled");
};
}
class Context {
constructor(options) {
this.options = options;
function createSubstitutionTransform(env) {
return async (input) => {
if (typeof input !== "string") {
return {applied: false};
}
const parts = input.split(/(?<!\$)\$\{([^{}]+)\}/);
for (let i = 1; i < parts.length; i += 2) {
parts[i] = await env(parts[i].trim());
}
if (parts.some((part) => part === void 0)) {
return {applied: true, value: void 0};
}
return {applied: true, value: parts.join("")};
};
}
const CONFIG_VISIBILITIES = ["frontend", "backend", "secret"];
const DEFAULT_CONFIG_VISIBILITY = "backend";
function compileConfigSchemas(schemas) {
const visibilityByPath = new Map();
const ajv2 = new Ajv__default['default']({
allErrors: true,
schemas: {
"https://backstage.io/schema/config-v1": true
}
}).addKeyword("visibility", {
metaSchema: {
type: "string",
enum: CONFIG_VISIBILITIES
},
compile(visibility) {
return (_data, dataPath) => {
if (!dataPath) {
return false;
}
if (visibility && visibility !== "backend") {
const normalizedPath = dataPath.replace(/\['?(.*?)'?\]/g, (_, segment) => `.${segment}`);
visibilityByPath.set(normalizedPath, visibility);
}
return true;
};
}
});
const merged = mergeAllOf__default['default']({allOf: schemas.map((_) => _.value)}, {
ignoreAdditionalProperties: true,
resolvers: {
visibility(values, path) {
const hasFrontend = values.some((_) => _ === "frontend");
const hasSecret = values.some((_) => _ === "secret");
if (hasFrontend && hasSecret) {
throw new Error(`Config schema visibility is both 'frontend' and 'secret' for ${path.join("/")}`);
} else if (hasFrontend) {
return "frontend";
} else if (hasSecret) {
return "secret";
}
return "backend";
}
}
});
const validate = ajv2.compile(merged);
return (configs) => {
var _a;
const config2 = config.ConfigReader.fromConfigs(configs).get();
visibilityByPath.clear();
const valid = validate(config2);
if (!valid) {
const errors = (_a = validate.errors) != null ? _a : [];
return {
errors: errors.map(({dataPath, message, params}) => {
const paramStr = Object.entries(params).map(([name, value]) => `${name}=${value}`).join(" ");
return `Config ${message || ""} { ${paramStr} } at ${dataPath}`;
}),
visibilityByPath: new Map()
};
}
return {
visibilityByPath: new Map(visibilityByPath)
};
};
}
const req = typeof __non_webpack_require__ === "undefined" ? require : __non_webpack_require__;
async function collectConfigSchemas(packageNames) {
const visitedPackages = new Set();
const schemas = Array();
const tsSchemaPaths = Array();
const currentDir = await fs__default['default'].realpath(process.cwd());
async function processItem({name, parentPath}) {
var _a, _b;
if (visitedPackages.has(name)) {
return;
}
visitedPackages.add(name);
let pkgPath;
try {
pkgPath = req.resolve(`${name}/package.json`, parentPath && {
paths: [parentPath]
});
} catch {
return;
}
const pkg = await fs__default['default'].readJson(pkgPath);
const depNames = [
...Object.keys((_a = pkg.dependencies) != null ? _a : {}),
...Object.keys((_b = pkg.peerDependencies) != null ? _b : {})
];
const hasSchema = "configSchema" in pkg;
const hasBackstageDep = depNames.some((_) => _.startsWith("@backstage/"));
if (!hasSchema && !hasBackstageDep) {
return;
}
if (hasSchema) {
if (typeof pkg.configSchema === "string") {
const isJson = pkg.configSchema.endsWith(".json");
const isDts = pkg.configSchema.endsWith(".d.ts");
if (!isJson && !isDts) {
throw new Error(`Config schema files must be .json or .d.ts, got ${pkg.configSchema}`);
}
if (isDts) {
tsSchemaPaths.push(path.relative(currentDir, path.resolve(path.dirname(pkgPath), pkg.configSchema)));
} else {
const path2 = path.resolve(path.dirname(pkgPath), pkg.configSchema);
const value = await fs__default['default'].readJson(path2);
schemas.push({
value,
path: path.relative(currentDir, path2)
});
}
} else {
schemas.push({
value: pkg.configSchema,
path: path.relative(currentDir, pkgPath)
});
}
}
await Promise.all(depNames.map((name2) => processItem({name: name2, parentPath: pkgPath})));
}
get env() {
return this.options.env;
await Promise.all(packageNames.map((name) => processItem({name})));
const tsSchemas = compileTsSchemas(tsSchemaPaths);
return schemas.concat(tsSchemas);
}
function compileTsSchemas(paths) {
if (paths.length === 0) {
return [];
}
skip(path2) {
if (this.options.shouldReadSecrets) {
return false;
const program = typescriptJsonSchema.getProgramFromFiles(paths, {
incremental: false,
isolatedModules: true,
lib: ["ES5"],
noEmit: true,
noResolve: true,
skipLibCheck: true,
skipDefaultLibCheck: true,
strict: true,
typeRoots: [],
types: []
});
const tsSchemas = paths.map((path2) => {
let value;
try {
value = typescriptJsonSchema.generateSchema(program, "Config", {
required: true,
validationKeywords: ["visibility"]
}, [path2.split(path.sep).join("/")]);
} catch (error) {
if (error.message !== "type Config not found") {
throw error;
}
}
return this.options.secretPaths.has(path2);
}
async readFile(path2) {
return fs__default.readFile(path.resolve(this.options.rootPath, path2), "utf8");
}
async readSecret(path2, desc) {
this.options.secretPaths.add(path2);
if (!this.options.shouldReadSecrets) {
if (!value) {
throw new Error(`Invalid schema in ${path2}, missing Config export`);
}
return {path: path2, value};
});
return tsSchemas;
}
function filterByVisibility(data, includeVisibilities, visibilityByPath, transformFunc) {
var _a;
function transform(jsonVal, path) {
var _a2;
const visibility = (_a2 = visibilityByPath.get(path)) != null ? _a2 : DEFAULT_CONFIG_VISIBILITY;
const isVisible = includeVisibilities.includes(visibility);
if (typeof jsonVal !== "object") {
if (isVisible) {
if (transformFunc) {
return transformFunc(jsonVal, {visibility});
}
return jsonVal;
}
return void 0;
} else if (jsonVal === null) {
return void 0;
} else if (Array.isArray(jsonVal)) {
const arr = new Array();
for (const [index, value] of jsonVal.entries()) {
const out = transform(value, `${path}.${index}`);
if (out !== void 0) {
arr.push(out);
}
}
if (arr.length > 0 || isVisible) {
return arr;
}
return void 0;
}
return readSecret(desc, this);
const outObj = {};
let hasOutput = false;
for (const [key, value] of Object.entries(jsonVal)) {
if (value === void 0) {
continue;
}
const out = transform(value, `${path}.${key}`);
if (out !== void 0) {
outObj[key] = out;
hasOutput = true;
}
}
if (hasOutput || isVisible) {
return outObj;
}
return void 0;
}
return (_a = transform(data, "")) != null ? _a : {};
}
async function loadConfigSchema(options) {
let schemas;
if ("dependencies" in options) {
schemas = await collectConfigSchemas(options.dependencies);
} else {
const {serialized} = options;
if ((serialized == null ? void 0 : serialized.backstageConfigSchemaVersion) !== 1) {
throw new Error("Serialized configuration schema is invalid or has an invalid version number");
}
schemas = serialized.schemas;
}
const validate = compileConfigSchemas(schemas);
return {
process(configs, {visibility, valueTransform} = {}) {
const result = validate(configs);
if (result.errors) {
const error = new Error(`Config validation failed, ${result.errors.join("; ")}`);
error.messages = result.errors;
throw error;
}
let processedConfigs = configs;
if (visibility) {
processedConfigs = processedConfigs.map(({data, context}) => ({
context,
data: filterByVisibility(data, visibility, result.visibilityByPath, valueTransform)
}));
} else if (valueTransform) {
processedConfigs = processedConfigs.map(({data, context}) => ({
context,
data: filterByVisibility(data, Array.from(CONFIG_VISIBILITIES), result.visibilityByPath, valueTransform)
}));
}
return processedConfigs;
},
serialize() {
return {
schemas,
backstageConfigSchemaVersion: 1
};
}
};
}
async function loadConfig(options) {
const configs = [];
const configPaths = await resolveStaticConfig(options);
const {configRoot, experimentalEnvFunc: envFunc} = options;
const configPaths = options.configPaths.slice();
if (configPaths.length === 0) {
configPaths.push(path.resolve(configRoot, "app-config.yaml"));
const localConfig = path.resolve(configRoot, "app-config.local.yaml");
if (await fs__default['default'].pathExists(localConfig)) {
configPaths.push(localConfig);
}
}
const env = envFunc != null ? envFunc : async (name) => process.env[name];
try {
const secretPaths = new Set();
for (const configPath of configPaths) {
const config2 = await readConfigFile(configPath, new Context({
secretPaths,
env: process.env,
rootPath: path.dirname(configPath),
shouldReadSecrets: Boolean(options.shouldReadSecrets)
}));
configs.push(config2);
if (!path.isAbsolute(configPath)) {
throw new Error(`Config load path is not absolute: '${configPath}'`);
}
const dir = path.dirname(configPath);
const readFile = (path2) => fs__default['default'].readFile(path.resolve(dir, path2), "utf8");
const input = yaml2__default['default'].parse(await readFile(configPath));
const data = await applyConfigTransforms(dir, input, [
createIncludeTransform(env, readFile),
createSubstitutionTransform(env)
]);
configs.push({data, context: path.basename(configPath)});
}
} catch (error) {
throw new Error(`Failed to read static configuration file: ${error.message}`);
throw new Error(`Failed to read static configuration file, ${error.message}`);
}

@@ -265,3 +510,4 @@ configs.push(...readEnvConfig(process.env));

exports.loadConfig = loadConfig;
exports.loadConfigSchema = loadConfigSchema;
exports.readEnvConfig = readEnvConfig;
//# sourceMappingURL=index.cjs.js.map

68

dist/index.d.ts

@@ -1,2 +0,2 @@

import { AppConfig } from '@backstage/config';
import { AppConfig, JsonObject } from '@backstage/config';

@@ -25,9 +25,67 @@ /**

declare type EnvFunc = (name: string) => Promise<string | undefined>;
/**
* A list of all possible configuration value visibilities.
*/
declare const CONFIG_VISIBILITIES: readonly ["frontend", "backend", "secret"];
/**
* A type representing the possible configuration value visibilities
*/
declare type ConfigVisibility = typeof CONFIG_VISIBILITIES[number];
/**
* A function used to transform primitive configuration values.
*/
declare type TransformFunc<T extends number | string | boolean> = (value: T, context: {
visibility: ConfigVisibility;
}) => T | undefined;
/**
* Options used to process configuration data with a schema.
*/
declare type ConfigProcessingOptions = {
/**
* The visibilities that should be included in the output data.
* If omitted, the data will not be filtered by visibility.
*/
visibility?: ConfigVisibility[];
/**
* A transform function that can be used to transform primitive configuration values
* during validation. The value returned from the transform function will be used
* instead of the original value. If the transform returns `undefined`, the value
* will be omitted.
*/
valueTransform?: TransformFunc<any>;
};
/**
* A loaded configuration schema that is ready to process configuration data.
*/
declare type ConfigSchema = {
process(appConfigs: AppConfig[], options?: ConfigProcessingOptions): AppConfig[];
serialize(): JsonObject;
};
declare type Options = {
dependencies: string[];
} | {
serialized: JsonObject;
};
/**
* Loads config schema for a Backstage instance.
*/
declare function loadConfigSchema(options: Options): Promise<ConfigSchema>;
declare type LoadConfigOptions = {
rootPaths: string[];
env: string;
shouldReadSecrets?: boolean;
configRoot: string;
configPaths: string[];
/** @deprecated This option has been removed */
env?: string;
/**
* Custom environment variable loading function
*
* @experimental This API is not stable and may change at any point
*/
experimentalEnvFunc?: EnvFunc;
};
declare function loadConfig(options: LoadConfigOptions): Promise<AppConfig[]>;
export { LoadConfigOptions, loadConfig, readEnvConfig };
export { ConfigSchema, ConfigVisibility, LoadConfigOptions, loadConfig, loadConfigSchema, readEnvConfig };

585

dist/index.esm.js

@@ -1,98 +0,9 @@

import { resolve, basename, extname, dirname } from 'path';
import fs, { pathExists } from 'fs-extra';
import yaml2 from 'yaml';
import { object, string, lazy, ValidationError } from 'yup';
import { extname, resolve, dirname, sep, relative, isAbsolute, basename } from 'path';
import Ajv from 'ajv';
import mergeAllOf from 'json-schema-merge-allof';
import { ConfigReader } from '@backstage/config';
import fs from 'fs-extra';
import { getProgramFromFiles, generateSchema } from 'typescript-json-schema';
async function resolveStaticConfig(options) {
const filePaths = [
`app-config.yaml`,
`app-config.local.yaml`,
`app-config.${options.env}.yaml`,
`app-config.${options.env}.local.yaml`
];
const resolvedPaths = [];
for (const rootPath of options.rootPaths) {
for (const filePath of filePaths) {
const path2 = resolve(rootPath, filePath);
if (await pathExists(path2)) {
resolvedPaths.push(path2);
}
}
}
return resolvedPaths;
}
function isObject(obj) {
if (typeof obj !== "object") {
return false;
} else if (Array.isArray(obj)) {
return false;
}
return obj !== null;
}
async function readConfigFile(filePath, ctx) {
const configYaml = await ctx.readFile(filePath);
const config2 = yaml2.parse(configYaml);
const context = basename(filePath);
async function transform(obj, path2) {
if (ctx.skip(path2)) {
return void 0;
}
if (typeof obj !== "object") {
return obj;
} else if (obj === null) {
return void 0;
} else if (Array.isArray(obj)) {
const arr = new Array();
for (const [index, value] of obj.entries()) {
const out2 = await transform(value, `${path2}[${index}]`);
if (out2 !== void 0) {
arr.push(out2);
}
}
return arr;
}
if ("$secret" in obj) {
console.warn(`Deprecated secret declaration at '${path2}' in '${context}', use $env, $file, etc. instead`);
if (!isObject(obj.$secret)) {
throw TypeError(`Expected object at secret ${path2}.$secret`);
}
try {
return await ctx.readSecret(path2, obj.$secret);
} catch (error) {
throw new Error(`Invalid secret at ${path2}: ${error.message}`);
}
}
const [secretKey] = Object.keys(obj).filter((key) => key.startsWith("$"));
if (secretKey) {
if (Object.keys(obj).length !== 1) {
throw new Error(`Secret key '${secretKey}' has adjacent keys at ${path2}`);
}
try {
return await ctx.readSecret(path2, {
[secretKey.slice(1)]: obj[secretKey]
});
} catch (error) {
throw new Error(`Invalid secret at ${path2}: ${error.message}`);
}
}
const out = {};
for (const [key, value] of Object.entries(obj)) {
if (value !== void 0) {
const result = await transform(value, `${path2}.${key}`);
if (result !== void 0) {
out[key] = result;
}
}
}
return out;
}
const finalConfig = await transform(config2, "");
if (!isObject(finalConfig)) {
throw new TypeError("Expected object at config root");
}
return {data: finalConfig, context};
}
const ENV_PREFIX = "APP_CONFIG_";

@@ -148,26 +59,64 @@ const CONFIG_KEY_PART_PATTERN = /^[a-z][a-z0-9]*(?:[-_][a-z][a-z0-9]*)*$/i;

const secretLoaderSchemas = {
file: object({
file: string().required()
}),
env: object({
env: string().required()
}),
data: object({
data: string().required()
})
};
const secretSchema = lazy((value) => {
if (typeof value !== "object" || value === null) {
return object().required().label("secret");
function isObject(obj) {
if (typeof obj !== "object") {
return false;
} else if (Array.isArray(obj)) {
return false;
}
const loaderTypes = Object.keys(secretLoaderSchemas);
for (const key of loaderTypes) {
if (key in value) {
return secretLoaderSchemas[key];
return obj !== null;
}
async function applyConfigTransforms(initialDir, input, transforms) {
async function transform(inputObj, path, baseDir) {
var _a;
let obj = inputObj;
let dir = baseDir;
for (const tf of transforms) {
try {
const result = await tf(inputObj, baseDir);
if (result.applied) {
if (result.value === void 0) {
return void 0;
}
obj = result.value;
dir = (_a = result.newBaseDir) != null ? _a : dir;
break;
}
} catch (error) {
throw new Error(`error at ${path}, ${error.message}`);
}
}
if (typeof obj !== "object") {
return obj;
} else if (obj === null) {
return void 0;
} else if (Array.isArray(obj)) {
const arr = new Array();
for (const [index, value] of obj.entries()) {
const out2 = await transform(value, `${path}[${index}]`, dir);
if (out2 !== void 0) {
arr.push(out2);
}
}
return arr;
}
const out = {};
for (const [key, value] of Object.entries(obj)) {
if (value !== void 0) {
const result = await transform(value, `${path}.${key}`, dir);
if (result !== void 0) {
out[key] = result;
}
}
}
return out;
}
throw new ValidationError(`Secret must contain one of '${loaderTypes.join("', '")}'`, value, "$secret");
});
const dataSecretParser = {
const finalData = await transform(input, "", initialDir);
if (!isObject(finalData)) {
throw new TypeError("expected object at config root");
}
return finalData;
}
const includeFileParser = {
".json": async (content) => JSON.parse(content),

@@ -177,76 +126,368 @@ ".yaml": async (content) => yaml2.parse(content),

};
async function readSecret(data, ctx) {
const secret = secretSchema.validateSync(data, {strict: true});
if ("file" in secret) {
return ctx.readFile(secret.file);
}
if ("env" in secret) {
return ctx.env[secret.env];
}
if ("data" in secret) {
const url = "path" in secret ? `${secret.data}#${secret.path}` : secret.data;
const [filePath, dataPath] = url.split(/#(.*)/);
if (!dataPath) {
throw new Error(`Invalid format for data secret value, must be of the form <filepath>#<datapath>, got '${url}'`);
function createIncludeTransform(env, readFile) {
return async (input, baseDir) => {
if (!isObject(input)) {
return {applied: false};
}
const ext = extname(filePath);
const parser = dataSecretParser[ext];
if (!parser) {
throw new Error(`No data secret parser available for extension ${ext}`);
const [includeKey] = Object.keys(input).filter((key) => key.startsWith("$"));
if (includeKey) {
if (Object.keys(input).length !== 1) {
throw new Error(`include key ${includeKey} should not have adjacent keys`);
}
} else {
return {applied: false};
}
const content = await ctx.readFile(filePath);
const parts = dataPath.split(".");
let value = await parser(content);
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join(".");
throw new Error(`Value is not an object at ${errPath} in ${filePath}`);
const includeValue = input[includeKey];
if (typeof includeValue !== "string") {
throw new Error(`${includeKey} include value is not a string`);
}
switch (includeKey) {
case "$file":
try {
const value = await readFile(resolve(baseDir, includeValue));
return {applied: true, value};
} catch (error) {
throw new Error(`failed to read file ${includeValue}, ${error}`);
}
case "$env":
try {
return {applied: true, value: await env(includeValue)};
} catch (error) {
throw new Error(`failed to read env ${includeValue}, ${error}`);
}
case "$include": {
const [filePath, dataPath] = includeValue.split(/#(.*)/);
const ext = extname(filePath);
const parser = includeFileParser[ext];
if (!parser) {
throw new Error(`no configuration parser available for included file ${filePath}`);
}
const path2 = resolve(baseDir, filePath);
const content = await readFile(path2);
const newBaseDir = dirname(path2);
const parts = dataPath ? dataPath.split(".") : [];
let value;
try {
value = await parser(content);
} catch (error) {
throw new Error(`failed to parse included file ${filePath}, ${error}`);
}
for (const [index, part] of parts.entries()) {
if (!isObject(value)) {
const errPath = parts.slice(0, index).join(".");
throw new Error(`value at '${errPath}' in included file ${filePath} is not an object`);
}
value = value[part];
}
return {
applied: true,
value,
newBaseDir: newBaseDir !== baseDir ? newBaseDir : void 0
};
}
value = value[part];
default:
throw new Error(`unknown include ${includeKey}`);
}
return String(value);
}
throw new Error("Secret was left unhandled");
};
}
class Context {
constructor(options) {
this.options = options;
function createSubstitutionTransform(env) {
return async (input) => {
if (typeof input !== "string") {
return {applied: false};
}
const parts = input.split(/(?<!\$)\$\{([^{}]+)\}/);
for (let i = 1; i < parts.length; i += 2) {
parts[i] = await env(parts[i].trim());
}
if (parts.some((part) => part === void 0)) {
return {applied: true, value: void 0};
}
return {applied: true, value: parts.join("")};
};
}
const CONFIG_VISIBILITIES = ["frontend", "backend", "secret"];
const DEFAULT_CONFIG_VISIBILITY = "backend";
function compileConfigSchemas(schemas) {
const visibilityByPath = new Map();
const ajv2 = new Ajv({
allErrors: true,
schemas: {
"https://backstage.io/schema/config-v1": true
}
}).addKeyword("visibility", {
metaSchema: {
type: "string",
enum: CONFIG_VISIBILITIES
},
compile(visibility) {
return (_data, dataPath) => {
if (!dataPath) {
return false;
}
if (visibility && visibility !== "backend") {
const normalizedPath = dataPath.replace(/\['?(.*?)'?\]/g, (_, segment) => `.${segment}`);
visibilityByPath.set(normalizedPath, visibility);
}
return true;
};
}
});
const merged = mergeAllOf({allOf: schemas.map((_) => _.value)}, {
ignoreAdditionalProperties: true,
resolvers: {
visibility(values, path) {
const hasFrontend = values.some((_) => _ === "frontend");
const hasSecret = values.some((_) => _ === "secret");
if (hasFrontend && hasSecret) {
throw new Error(`Config schema visibility is both 'frontend' and 'secret' for ${path.join("/")}`);
} else if (hasFrontend) {
return "frontend";
} else if (hasSecret) {
return "secret";
}
return "backend";
}
}
});
const validate = ajv2.compile(merged);
return (configs) => {
var _a;
const config2 = ConfigReader.fromConfigs(configs).get();
visibilityByPath.clear();
const valid = validate(config2);
if (!valid) {
const errors = (_a = validate.errors) != null ? _a : [];
return {
errors: errors.map(({dataPath, message, params}) => {
const paramStr = Object.entries(params).map(([name, value]) => `${name}=${value}`).join(" ");
return `Config ${message || ""} { ${paramStr} } at ${dataPath}`;
}),
visibilityByPath: new Map()
};
}
return {
visibilityByPath: new Map(visibilityByPath)
};
};
}
const req = typeof __non_webpack_require__ === "undefined" ? require : __non_webpack_require__;
async function collectConfigSchemas(packageNames) {
const visitedPackages = new Set();
const schemas = Array();
const tsSchemaPaths = Array();
const currentDir = await fs.realpath(process.cwd());
async function processItem({name, parentPath}) {
var _a, _b;
if (visitedPackages.has(name)) {
return;
}
visitedPackages.add(name);
let pkgPath;
try {
pkgPath = req.resolve(`${name}/package.json`, parentPath && {
paths: [parentPath]
});
} catch {
return;
}
const pkg = await fs.readJson(pkgPath);
const depNames = [
...Object.keys((_a = pkg.dependencies) != null ? _a : {}),
...Object.keys((_b = pkg.peerDependencies) != null ? _b : {})
];
const hasSchema = "configSchema" in pkg;
const hasBackstageDep = depNames.some((_) => _.startsWith("@backstage/"));
if (!hasSchema && !hasBackstageDep) {
return;
}
if (hasSchema) {
if (typeof pkg.configSchema === "string") {
const isJson = pkg.configSchema.endsWith(".json");
const isDts = pkg.configSchema.endsWith(".d.ts");
if (!isJson && !isDts) {
throw new Error(`Config schema files must be .json or .d.ts, got ${pkg.configSchema}`);
}
if (isDts) {
tsSchemaPaths.push(relative(currentDir, resolve(dirname(pkgPath), pkg.configSchema)));
} else {
const path2 = resolve(dirname(pkgPath), pkg.configSchema);
const value = await fs.readJson(path2);
schemas.push({
value,
path: relative(currentDir, path2)
});
}
} else {
schemas.push({
value: pkg.configSchema,
path: relative(currentDir, pkgPath)
});
}
}
await Promise.all(depNames.map((name2) => processItem({name: name2, parentPath: pkgPath})));
}
get env() {
return this.options.env;
await Promise.all(packageNames.map((name) => processItem({name})));
const tsSchemas = compileTsSchemas(tsSchemaPaths);
return schemas.concat(tsSchemas);
}
function compileTsSchemas(paths) {
if (paths.length === 0) {
return [];
}
skip(path2) {
if (this.options.shouldReadSecrets) {
return false;
const program = getProgramFromFiles(paths, {
incremental: false,
isolatedModules: true,
lib: ["ES5"],
noEmit: true,
noResolve: true,
skipLibCheck: true,
skipDefaultLibCheck: true,
strict: true,
typeRoots: [],
types: []
});
const tsSchemas = paths.map((path2) => {
let value;
try {
value = generateSchema(program, "Config", {
required: true,
validationKeywords: ["visibility"]
}, [path2.split(sep).join("/")]);
} catch (error) {
if (error.message !== "type Config not found") {
throw error;
}
}
return this.options.secretPaths.has(path2);
}
async readFile(path2) {
return fs.readFile(resolve(this.options.rootPath, path2), "utf8");
}
async readSecret(path2, desc) {
this.options.secretPaths.add(path2);
if (!this.options.shouldReadSecrets) {
if (!value) {
throw new Error(`Invalid schema in ${path2}, missing Config export`);
}
return {path: path2, value};
});
return tsSchemas;
}
function filterByVisibility(data, includeVisibilities, visibilityByPath, transformFunc) {
var _a;
function transform(jsonVal, path) {
var _a2;
const visibility = (_a2 = visibilityByPath.get(path)) != null ? _a2 : DEFAULT_CONFIG_VISIBILITY;
const isVisible = includeVisibilities.includes(visibility);
if (typeof jsonVal !== "object") {
if (isVisible) {
if (transformFunc) {
return transformFunc(jsonVal, {visibility});
}
return jsonVal;
}
return void 0;
} else if (jsonVal === null) {
return void 0;
} else if (Array.isArray(jsonVal)) {
const arr = new Array();
for (const [index, value] of jsonVal.entries()) {
const out = transform(value, `${path}.${index}`);
if (out !== void 0) {
arr.push(out);
}
}
if (arr.length > 0 || isVisible) {
return arr;
}
return void 0;
}
return readSecret(desc, this);
const outObj = {};
let hasOutput = false;
for (const [key, value] of Object.entries(jsonVal)) {
if (value === void 0) {
continue;
}
const out = transform(value, `${path}.${key}`);
if (out !== void 0) {
outObj[key] = out;
hasOutput = true;
}
}
if (hasOutput || isVisible) {
return outObj;
}
return void 0;
}
return (_a = transform(data, "")) != null ? _a : {};
}
async function loadConfigSchema(options) {
let schemas;
if ("dependencies" in options) {
schemas = await collectConfigSchemas(options.dependencies);
} else {
const {serialized} = options;
if ((serialized == null ? void 0 : serialized.backstageConfigSchemaVersion) !== 1) {
throw new Error("Serialized configuration schema is invalid or has an invalid version number");
}
schemas = serialized.schemas;
}
const validate = compileConfigSchemas(schemas);
return {
process(configs, {visibility, valueTransform} = {}) {
const result = validate(configs);
if (result.errors) {
const error = new Error(`Config validation failed, ${result.errors.join("; ")}`);
error.messages = result.errors;
throw error;
}
let processedConfigs = configs;
if (visibility) {
processedConfigs = processedConfigs.map(({data, context}) => ({
context,
data: filterByVisibility(data, visibility, result.visibilityByPath, valueTransform)
}));
} else if (valueTransform) {
processedConfigs = processedConfigs.map(({data, context}) => ({
context,
data: filterByVisibility(data, Array.from(CONFIG_VISIBILITIES), result.visibilityByPath, valueTransform)
}));
}
return processedConfigs;
},
serialize() {
return {
schemas,
backstageConfigSchemaVersion: 1
};
}
};
}
async function loadConfig(options) {
const configs = [];
const configPaths = await resolveStaticConfig(options);
const {configRoot, experimentalEnvFunc: envFunc} = options;
const configPaths = options.configPaths.slice();
if (configPaths.length === 0) {
configPaths.push(resolve(configRoot, "app-config.yaml"));
const localConfig = resolve(configRoot, "app-config.local.yaml");
if (await fs.pathExists(localConfig)) {
configPaths.push(localConfig);
}
}
const env = envFunc != null ? envFunc : async (name) => process.env[name];
try {
const secretPaths = new Set();
for (const configPath of configPaths) {
const config2 = await readConfigFile(configPath, new Context({
secretPaths,
env: process.env,
rootPath: dirname(configPath),
shouldReadSecrets: Boolean(options.shouldReadSecrets)
}));
configs.push(config2);
if (!isAbsolute(configPath)) {
throw new Error(`Config load path is not absolute: '${configPath}'`);
}
const dir = dirname(configPath);
const readFile = (path2) => fs.readFile(resolve(dir, path2), "utf8");
const input = yaml2.parse(await readFile(configPath));
const data = await applyConfigTransforms(dir, input, [
createIncludeTransform(env, readFile),
createSubstitutionTransform(env)
]);
configs.push({data, context: basename(configPath)});
}
} catch (error) {
throw new Error(`Failed to read static configuration file: ${error.message}`);
throw new Error(`Failed to read static configuration file, ${error.message}`);
}

@@ -257,3 +498,3 @@ configs.push(...readEnvConfig(process.env));

export { loadConfig, readEnvConfig };
export { loadConfig, loadConfigSchema, readEnvConfig };
//# sourceMappingURL=index.esm.js.map

17

package.json
{
"name": "@backstage/config-loader",
"description": "Config loading functionality used by Backstage backend, and CLI",
"version": "0.0.0-nightly-20209921112",
"version": "0.0.0-nightly-20210262290",
"private": false,

@@ -15,3 +15,3 @@ "publishConfig": {

"type": "git",
"url": "https://github.com/spotify/backstage",
"url": "https://github.com/backstage/backstage",
"directory": "packages/config-loader"

@@ -34,12 +34,19 @@ },

"dependencies": {
"@backstage/config": "^0.1.1-alpha.24",
"@backstage/cli-common": "^0.1.1",
"@backstage/config": "^0.1.1",
"ajv": "^6.12.5",
"fs-extra": "^9.0.0",
"json-schema": "^0.2.5",
"json-schema-merge-allof": "^0.7.0",
"typescript-json-schema": "^0.47.0",
"yaml": "^1.9.2",
"yup": "^0.29.1"
"yup": "^0.29.3"
},
"devDependencies": {
"@types/jest": "^26.0.7",
"@types/json-schema": "^7.0.6",
"@types/json-schema-merge-allof": "^0.6.0",
"@types/mock-fs": "^4.10.0",
"@types/node": "^12.0.0",
"@types/yup": "^0.28.2",
"@types/yup": "^0.29.8",
"mock-fs": "^4.13.0"

@@ -46,0 +53,0 @@ },

4

README.md

@@ -11,3 +11,3 @@ # @backstage/config-loader

- [Backstage Readme](https://github.com/spotify/backstage/blob/master/README.md)
- [Backstage Documentation](https://github.com/spotify/backstage/blob/master/docs/README.md)
- [Backstage Readme](https://github.com/backstage/backstage/blob/master/README.md)
- [Backstage Documentation](https://github.com/backstage/backstage/blob/master/docs/README.md)
dist/index.cjs.js.map

Sorry, the diff of this file is not supported yet

dist/index.esm.js.map

Sorry, the diff of this file is not supported yet

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc