Security News
tea.xyz Spam Plagues npm and RubyGems Package Registries
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
@google-cloud/dataproc
Advanced tools
Readme
Google Cloud Dataproc API client for Node.js
A comprehensive list of changes in each version may be found in the CHANGELOG.
Read more about the client libraries for Cloud APIs, including the older Google APIs Client Libraries, in Client Libraries Explained.
Table of contents:
npm install @google-cloud/dataproc
// This quickstart sample walks a user through creating a Dataproc
// cluster, submitting a PySpark job from Google Cloud Storage to the
// cluster, reading the output of the job and deleting the cluster, all
// using the Node.js client library.
'use strict';
function main(projectId, region, clusterName, jobFilePath) {
const dataproc = require('@google-cloud/dataproc');
const {Storage} = require('@google-cloud/storage');
// Create a cluster client with the endpoint set to the desired cluster region
const clusterClient = new dataproc.v1.ClusterControllerClient({
apiEndpoint: `${region}-dataproc.googleapis.com`,
projectId: projectId,
});
// Create a job client with the endpoint set to the desired cluster region
const jobClient = new dataproc.v1.JobControllerClient({
apiEndpoint: `${region}-dataproc.googleapis.com`,
projectId: projectId,
});
async function quickstart() {
// Create the cluster config
const cluster = {
projectId: projectId,
region: region,
cluster: {
clusterName: clusterName,
config: {
masterConfig: {
numInstances: 1,
machineTypeUri: 'n1-standard-2',
},
workerConfig: {
numInstances: 2,
machineTypeUri: 'n1-standard-2',
},
},
},
};
// Create the cluster
const [operation] = await clusterClient.createCluster(cluster);
const [response] = await operation.promise();
// Output a success message
console.log(`Cluster created successfully: ${response.clusterName}`);
const job = {
projectId: projectId,
region: region,
job: {
placement: {
clusterName: clusterName,
},
pysparkJob: {
mainPythonFileUri: jobFilePath,
},
},
};
const [jobOperation] = await jobClient.submitJobAsOperation(job);
const [jobResponse] = await jobOperation.promise();
const matches =
jobResponse.driverOutputResourceUri.match('gs://(.*?)/(.*)');
const storage = new Storage();
const output = await storage
.bucket(matches[1])
.file(`${matches[2]}.000000000`)
.download();
// Output a success message.
console.log(`Job finished successfully: ${output}`);
// Delete the cluster once the job has terminated.
const deleteClusterReq = {
projectId: projectId,
region: region,
clusterName: clusterName,
};
const [deleteOperation] =
await clusterClient.deleteCluster(deleteClusterReq);
await deleteOperation.promise();
// Output a success message
console.log(`Cluster ${clusterName} successfully deleted.`);
}
quickstart();
}
const args = process.argv.slice(2);
if (args.length !== 4) {
console.log(
'Insufficient number of parameters provided. Please make sure a ' +
'PROJECT_ID, REGION, CLUSTER_NAME and JOB_FILE_PATH are provided, in this order.'
);
}
main(...args);
Samples are in the samples/
directory. Each sample's README.md
has instructions for running its sample.
Sample | Source Code | Try it |
---|---|---|
Autoscaling_policy_service.create_autoscaling_policy | source code | |
Autoscaling_policy_service.delete_autoscaling_policy | source code | |
Autoscaling_policy_service.get_autoscaling_policy | source code | |
Autoscaling_policy_service.list_autoscaling_policies | source code | |
Autoscaling_policy_service.update_autoscaling_policy | source code | |
Batch_controller.create_batch | source code | |
Batch_controller.delete_batch | source code | |
Batch_controller.get_batch | source code | |
Batch_controller.list_batches | source code | |
Cluster_controller.create_cluster | source code | |
Cluster_controller.delete_cluster | source code | |
Cluster_controller.diagnose_cluster | source code | |
Cluster_controller.get_cluster | source code | |
Cluster_controller.list_clusters | source code | |
Cluster_controller.start_cluster | source code | |
Cluster_controller.stop_cluster | source code | |
Cluster_controller.update_cluster | source code | |
Job_controller.cancel_job | source code | |
Job_controller.delete_job | source code | |
Job_controller.get_job | source code | |
Job_controller.list_jobs | source code | |
Job_controller.submit_job | source code | |
Job_controller.submit_job_as_operation | source code | |
Job_controller.update_job | source code | |
Node_group_controller.create_node_group | source code | |
Node_group_controller.get_node_group | source code | |
Node_group_controller.resize_node_group | source code | |
Session_controller.create_session | source code | |
Session_controller.delete_session | source code | |
Session_controller.get_session | source code | |
Session_controller.list_sessions | source code | |
Session_controller.terminate_session | source code | |
Session_template_controller.create_session_template | source code | |
Session_template_controller.delete_session_template | source code | |
Session_template_controller.get_session_template | source code | |
Session_template_controller.list_session_templates | source code | |
Session_template_controller.update_session_template | source code | |
Workflow_template_service.create_workflow_template | source code | |
Workflow_template_service.delete_workflow_template | source code | |
Workflow_template_service.get_workflow_template | source code | |
Workflow_template_service.instantiate_inline_workflow_template | source code | |
Workflow_template_service.instantiate_workflow_template | source code | |
Workflow_template_service.list_workflow_templates | source code | |
Workflow_template_service.update_workflow_template | source code | |
Quickstart | source code |
The Google Cloud Dataproc Node.js Client API Reference documentation also contains samples.
Our client libraries follow the Node.js release schedule. Libraries are compatible with all current active and maintenance versions of Node.js. If you are using an end-of-life version of Node.js, we recommend that you update as soon as possible to an actively supported LTS version.
Google's client libraries support legacy versions of Node.js runtimes on a best-efforts basis with the following warnings:
Client libraries targeting some end-of-life versions of Node.js are available, and
can be installed through npm dist-tags.
The dist-tags follow the naming convention legacy-(version)
.
For example, npm install @google-cloud/dataproc@legacy-8
installs client libraries
for versions compatible with Node.js 8.
This library follows Semantic Versioning.
This library is considered to be stable. The code surface will not change in backwards-incompatible ways unless absolutely necessary (e.g. because of critical security issues) or with an extensive deprecation period. Issues and requests against stable libraries are addressed with the highest priority.
More Information: Google Cloud Platform Launch Stages
Contributions welcome! See the Contributing Guide.
Please note that this README.md
, the samples/README.md
,
and a variety of configuration files in this repository (including .nycrc
and tsconfig.json
)
are generated from a central template. To edit one of these files, make an edit
to its templates in
directory.
Apache Version 2.0
See LICENSE
FAQs
Google Cloud Dataproc API client for Node.js
The npm package @google-cloud/dataproc receives a total of 259 weekly downloads. As such, @google-cloud/dataproc popularity was classified as not popular.
We found that @google-cloud/dataproc demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
Security News
As cyber threats become more autonomous, AI-powered defenses are crucial for businesses to stay ahead of attackers who can exploit software vulnerabilities at scale.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.