@azure/storage-file-datalake

Azure Storage File Data Lake client library for JavaScript

Azure Data Lake Storage (ADLS) includes all the capabilities required to make it easy for developers, data scientists, and analysts to store data of any size, shape, and speed, and do all types of processing and analytics across platforms and languages. It removes the complexities of ingesting and storing all of your data while making it faster to get up and running with batch, streaming, and interactive analytics.

This project provides a client library in JavaScript that makes it easy to consume Microsoft Azure Storage Data Lake service.

Use the client libraries in this package to:

• Create/List/Delete File Systems
• Create/Read/List/Update/Delete Paths, Directories and Files

Source code | Package (npm) | API Reference Documentation | Product documentation | Samples | Azure Storage Data Lake REST APIs

Getting started

Prerequisites: You must have an Azure subscription and a Storage Account to use this package. If you are using this package in a Node.js application, then Node.js version 8.0.0 or higher is required.

Install the package

The preferred way to install the Azure Storage Data Lake client library for JavaScript is to use the npm package manager. Type the following into a terminal window:

npm install @azure/storage-file-datalake

Authenticate the client

Azure Storage supports several ways to authenticate. In order to interact with the Azure Data Lake Storage service you'll need to create an instance of a Storage client - DataLakeServiceClient, DataLakeFileSystemClient, or DataLakePathClient for example. See samples for creating the DataLakeServiceClient to learn more about authentication.

• Azure Active Directory
• Shared Key
• Shared access signatures

Azure Active Directory

The Azure Data Lake Storage service supports the use of Azure Active Directory to authenticate requests to its APIs. The @azure/identity package provides a variety of credential types that your application can use to do this. Please see the README for @azure/identity for more details and samples to get you started.

Compatibility

This library is compatible with Node.js and browsers, and validated against LTS Node.js versions (>=8.16.0) and latest versions of Chrome, Firefox and Edge.

Compatible with IE11

You need polyfills to make this library work with IE11. The easiest way is to use @babel/polyfill, or polyfill service.

You can also load separate polyfills for missed ES feature(s). This library depends on following ES features which need external polyfills loaded.

• Promise
• String.prototype.startsWith
• String.prototype.endsWith
• String.prototype.repeat
• String.prototype.includes
• Array.prototype.includes
• Object.assign
• Object.keys (Overrides the IE11's Object.keys with a polyfill to enable the ES6 behavior)
• Symbol
• Symbol.iterator

Differences between Node.js and browsers

There are differences between Node.js and browsers runtime. When getting started with this library, pay attention to APIs or classes marked with "ONLY AVAILABLE IN NODE.JS RUNTIME" or "ONLY AVAILABLE IN BROWSERS".

• If a file holds compressed data in gzip or deflate format and its content encoding is set accordingly, downloading behavior is different between Node.js and browsers. In Node.js storage clients will download the file in its compressed format, while in browsers the data will be downloaded in de-compressed format.

Features, interfaces, classes or functions only available in Node.js

• Shared Key Authorization based on account name and account key
  • StorageSharedKeyCredential
• Shared Access Signature(SAS) generation
  • generateAccountSASQueryParameters()
  • generateDataLakeSASQueryParameters()
• Parallel uploading and downloading. Note that DataLakeFileClient.upload() is available in both Node.js and browsers.
  • DataLakeFileClient.uploadFile()
  • DataLakeFileClient.uploadStream()
  • DataLakeFileClient.readToBuffer()
  • DataLakeFileClient.readToFile()

Features, interfaces, classes or functions only available in browsers

• N/A

JavaScript Bundle

To use this client library in the browser, first you need to use a bundler. For details on how to do this, please refer to our bundling documentation.

CORS

You need to set up Cross-Origin Resource Sharing (CORS) rules for your storage account if you need to develop for browsers. Go to Azure portal and Azure Storage Explorer, find your storage account, create new CORS rules for blob/queue/file/table service(s).

For example, you can create following CORS settings for debugging. But please customize the settings carefully according to your requirements in production environment.

• Allowed origins: *
• Allowed verbs: DELETE,GET,HEAD,MERGE,POST,OPTIONS,PUT
• Allowed headers: *
• Exposed headers: *
• Maximum age (seconds): 86400

Notice: Data Lake currently shares CORS settings for blob service.

Key concepts

Azure Data Lake Storage Gen2 was designed to:

• Serve multiple petabytes of information while sustaining hundreds of gigabits of throughput
• Allow you to easily manage massive amounts of data

Key Features of DataLake Storage Gen2 include:

• Hadoop compatible access
• A super set of POSIX permissions
• Cost effective in terms of low-cost storage capacity and transactions
• Optimized driver for big data analytics

A fundamental part of Data Lake Storage Gen2 is the addition of a hierarchical namespace to Blob storage. The hierarchical namespace organizes objects/files into a hierarchy of directories for efficient data access.

In the past, cloud-based analytics had to compromise in areas of performance, management, and security. Data Lake Storage Gen2 addresses each of these aspects in the following ways:

• Performance is optimized because you do not need to copy or transform data as a prerequisite for analysis. The hierarchical namespace greatly improves the performance of directory management operations, which improves overall job performance.
• Management is easier because you can organize and manipulate files through directories and subdirectories.
• Security is enforceable because you can define POSIX permissions on directories or individual files.
• Cost effectiveness is made possible as Data Lake Storage Gen2 is built on top of the low-cost Azure Blob storage. The additional features further lower the total cost of ownership for running big data analytics on Azure.

Data Lake storage offers three types of resources:

• The storage account used via DataLakeServiceClient
• A file system in the storage account used via DataLakeFileSystemClient
• A path in a file system used via DataLakeDirectoryClient or DataLakeFileClient

Azure DataLake Gen2	Blob
Filesystem	Container
Path (File or Directory)	Blob

Note: This client library only supports storage accounts with hierarchical namespace (HNS) enabled.

Examples

• Import the package
• Create the data lake service client
• Create a new file system
• List the file systems
• Create and delete a directory
• Create a file
• List paths inside a file system
• Download a file and convert it to a string (Node.js)
• Download a file and convert it to a string (Browsers)

Import the package

To use the clients, import the package into your file:

const AzureStorageDataLake = require("@azure/storage-file-datalake");

Alternatively, selectively import only the types you need:

const {
  DataLakeServiceClient,
  StorageSharedKeyCredential
} = require("@azure/storage-file-datalake");

Create the data lake service client

The DataLakeServiceClient requires an URL to the data lake service and an access credential. It also optionally accepts some settings in the options parameter.

with DefaultAzureCredential from @azure/identity package

Recommended way to instantiate a DataLakeServiceClient

Notice. Azure Data Lake currently reuses blob related roles like "Storage Blob Data Owner" during following AAD OAuth authentication.

Setup : Reference - Authorize access to blobs (data lake) and queues with Azure Active Directory from a client application - https://docs.microsoft.com/azure/storage/common/storage-auth-aad-app

• Register a new AAD application and give permissions to access Azure Storage on behalf of the signed-in user.

  • Register a new application in the Azure Active Directory(in the azure-portal) - https://docs.microsoft.com/azure/active-directory/develop/quickstart-register-app
  • In the API permissions section, select Add a permission and choose Microsoft APIs.
  • Pick Azure Storage and select the checkbox next to user_impersonation and then click Add permissions. This would allow the application to access Azure Storage on behalf of the signed-in user.

• Grant access to Azure Data Lake data with RBAC in the Azure Portal

  • RBAC roles for blobs (data lake) and queues - https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-portal.
  • In the azure portal, go to your storage-account and assign Storage Blob Data Contributor role to the registered AAD application from Access control (IAM) tab (in the left-side-navbar of your storage account in the azure-portal).

• Environment setup for the sample

  • From the overview page of your AAD Application, note down the CLIENT ID and TENANT ID. In the "Certificates & Secrets" tab, create a secret and note that down.
  • Make sure you have AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET as environment variables to successfully execute the sample(Can leverage process.env).

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

// Enter your storage account name
const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

See the Azure AD Auth sample for a complete example using this method.

[Note - Above steps are only for Node.js]

using connection string

Alternatively, you can instantiate a DataLakeServiceClient using the fromConnectionString() static method with the full connection string as the argument. (The connection string can be obtained from the azure portal.) [ONLY AVAILABLE IN NODE.JS RUNTIME]

const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const connStr = "<connection string>";

const DataLakeServiceClient = DataLakeServiceClient.fromConnectionString(connStr);

with StorageSharedKeyCredential

Alternatively, you instantiate a DataLakeServiceClient with a StorageSharedKeyCredential by passing account-name and account-key as arguments. (The account-name and account-key can be obtained from the azure portal.) [ONLY AVAILABLE IN NODE.JS RUNTIME]

const {
  DataLakeServiceClient,
  StorageSharedKeyCredential
} = require("@azure/storage-file-datalake");

// Enter your storage account name and shared key
const account = "<account>";
const accountKey = "<accountkey>";

// Use StorageSharedKeyCredential with storage account and account key
// StorageSharedKeyCredential is only available in Node.js runtime, not in browsers
const sharedKeyCredential = new StorageSharedKeyCredential(account, accountKey);
const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  sharedKeyCredential
);

with SAS Token

Also, You can instantiate a DataLakeServiceClient with a shared access signatures (SAS). You can get the SAS token from the Azure Portal or generate one using generateAccountSASQueryParameters().

const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account name>";
const sas = "<service Shared Access Signature Token>";
const serviceClientWithSAS = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net${sas}`
);

Create a new file system

Use DataLakeServiceClient.getFileSystemClient() to get a file system client instance then create a new file system resource.

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

async function main() {
  // Create a file system
  const fileSystemName = `newfilesystem${new Date().getTime()}`;
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);
  const createResponse = await fileSystemClient.create();
  console.log(`Create file system ${fileSystemName} successfully`, createResponse.requestId);
}

main();

List the file systems

Use DataLakeServiceClient.listFileSystems() function to iterate the file systems, with the new for-await-of syntax:

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

async function main() {
  let i = 1;
  let fileSystems = datalakeServiceClient.listFileSystems();
  for await (const fileSystem of fileSystems) {
    console.log(`File system ${i++}: ${fileSystem.name}`);
  }
}

main();

Alternatively without using for-await-of:

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

async function main() {
  let i = 1;
  let iter = datalakeServiceClient.listFileSystems();
  let fileSystemItem = await iter.next();
  while (!fileSystemItem.done) {
    console.log(`File System ${i++}: ${fileSystemItem.value.name}`);
    fileSystemItem = await iter.next();
  }
}

main();

In addition, pagination is supported for listing too via byPage():

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

async function main() {
  let i = 1;
  for await (const response of datalakeServiceClient
    .listFileSystems()
    .byPage({ maxPageSize: 20 })) {
    if (response.fileSystemItems) {
      for (const fileSystem of response.fileSystemItems) {
        console.log(`File System ${i++}: ${fileSystem.name}`);
      }
    }
  }
}

main();

Create and delete a directory

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

const fileSystemName = "<file system name>";

async function main() {
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);
  const directoryClient = fileSystemClient.getDirectoryClient("directory");
  await directoryClient.create();
  await directoryClient.delete();
}

main();

Create a file

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

const fileSystemName = "<file system name>";

async function main() {
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);

  const content = "Hello world!";
  const fileName = "newfile" + new Date().getTime();
  const fileClient = fileSystemClient.getFileClient(fileName);
  await fileClient.create();
  await fileClient.append(content, 0, content.length);
  await fileClient.flush(content.length);
  console.log(`Create and upload file ${fileName} successfully`);
}

main();

List paths inside a file system

Similar to listing file systems.

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

const fileSystemName = "<file system name>";

async function main() {
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);

  let i = 1;
  let paths = fileSystemClient.listPaths();
  for await (const path of paths) {
    console.log(`Path ${i++}: ${path.name}, is directory: ${path.isDirectory}`);
  }
}

main();

Download a file and convert it to a string (Node.js)

const { DefaultAzureCredential } = require("@azure/identity");
const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const defaultAzureCredential = new DefaultAzureCredential();

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net`,
  defaultAzureCredential
);

const fileSystemName = "<file system name>";
const fileName = "<file name>";

async function main() {
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);
  const fileClient = fileSystemClient.getFileClient(fileName);

  // Get file content from position 0 to the end
  // In Node.js, get downloaded data by accessing downloadResponse.readableStreamBody
  const downloadResponse = await fileClient.read();
  const downloaded = await streamToBuffer(downloadResponse.readableStreamBody);
  console.log("Downloaded file content:", downloaded.toString());

  // [Node.js only] A helper method used to read a Node.js readable stream into a Buffer.
  async function streamToBuffer(readableStream) {
    return new Promise((resolve, reject) => {
      const chunks = [];
      readableStream.on("data", (data) => {
        chunks.push(data instanceof Buffer ? data : Buffer.from(data));
      });
      readableStream.on("end", () => {
        resolve(Buffer.concat(chunks));
      });
      readableStream.on("error", reject);
    });
  }
}

main();

Download a file and convert it to a string (Browsers)

const { DataLakeServiceClient } = require("@azure/storage-file-datalake");

const account = "<account>";
const sas="<sas token>"

const datalakeServiceClient = new DataLakeServiceClient(
  `https://${account}.dfs.core.windows.net${sas}`
);

const fileSystemName = "<file system name>";
const fileName = "<file name>"

async function main() {
  const fileSystemClient = datalakeServiceClient.getFileSystemClient(fileSystemName);
  const fileClient = fileSystemClient.getFileClient(fileName);

  // Get file content from position 0 to the end
  // In browsers, get downloaded data by accessing downloadResponse.contentAsBlob
  const downloadResponse = await fileClient.read();
  const downloaded = await blobToString(await downloadResponse.contentAsBlob);
  console.log(
    "Downloaded file content",
    downloaded
  );

  // [Browsers only] A helper method used to convert a browser Blob into string.
  async function blobToString(blob: Blob): Promise<string> {
    const fileReader = new FileReader();
    return new Promise<string>((resolve, reject) => {
      fileReader.onloadend = (ev: any) => {
        resolve(ev.target!.result);
      };
      fileReader.onerror = reject;
      fileReader.readAsText(blob);
    });
  }
}

main();

Troubleshooting

Enabling logging may help uncover useful information about failures. In order to see a log of HTTP requests and responses, set the AZURE_LOG_LEVEL environment variable to info. Alternatively, logging can be enabled at runtime by calling setLogLevel in the @azure/logger:

import { setLogLevel } from "@azure/logger";

setLogLevel("info");

Next steps

More code samples:

• DataLake Storage Samples (JavaScript)
• DataLake Storage Samples (TypeScript)
• DataLake Storage Test Cases

Contributing

If you'd like to contribute to this library, please read the contributing guide to learn more about how to build and test the code.